October 31, 2014

Levi’s Stadium Wi-Fi update: Usage down from record, but still strong — 2.4 TB for Eagles game

Fans take pictures of opening-day kickoff from southwest concourse. Credit, all photos: Paul Kapustka, MSR

Fans take pictures of opening-day kickoff from southwest concourse. Credit, all photos: Paul Kapustka, MSR

With three NFL games and now one collegiate contest under its belt, the Wi-Fi network at Levi’s Stadium is still handling big loads of data traffic, though not quite at the Super Bowl-beating level of 3.3 Terabytes recorded at the Niners’ home opener.

According to statistics provided by Niners vice president for technology Dan Williams, the Levi’s Wi-Fi network carried 2.4 TB of data during the Sept. 28 game against the Philadelphia Eagles, and another 2 TB during the Oct. 5 contest with the Kansas City Chiefs, both sellouts with reported attendance of 70,799 fans. And at an Oct. 24 college game between Cal and Oregon with 55,575 fans in attendance, the network carried 1.5 TB of traffic.

As you might be able to guess from the bandwidth numbers, the Sept. 28 game also had more users on the network, with 22,942 unique users, compared to 21,133 at the Oct. 5 game. For Cal-Oregon the unique user count was 13,508. At the Niners’ regular-season home opener on Sept. 14, there were more than 30,000 fans using the Wi-Fi network, with a peak of 19,000 simultaneous connections. The peak numbers for the later dates were 15,500 for the Sept. 28 game, 14,500 for the Oct. 5 game, and 8,400 for the college game.

In-seat and express food orders stay strong

Screen grab from Levi's Stadium app showing in-seat food delivery option. Credit all photos: Paul Kapustka, Mobile Sports Report.

Screen grab from Levi’s Stadium app showing in-seat food delivery option.

Food and beverage orders via the Levi’s Stadium app appear to be holding between the 1,500-2,000 order level for both the express pickup option as well as the in-seat delivery, with both features available to every seat in the house. According to Williams, for the Eagles game there were 1,144 express pick-up orders and 1,652 in-seat deliveries, while at the Kansas City game there were 1,162 express and 1,712 in-seat orders. At the Cal-Oregon game, where many fans were probably not as aware of the Levi’s app functionality, there were 551 express orders and 308 in-seat deliveries.

The video-replay feature of the Levi’s app also continues to attract solid interest, with 1,297 unique users watching 5,089 replays at the Eagles game and 1,111 unique users watching 4,986 replays at the Kansas City game. At the college game 234 fans used the replay feature, watching 1,059 replays.

One more interesting stat provided by Williams — the Apple iPhone is the overwhelming favorite device for connecting to the Levi’s Stadium Wi-Fi, with 62 percent share for the Sept. 28 game, and 60 percent share for both the Oct. 5 and Oct. 24 games. Android devices represented 24 percent (Sept. 28), 26 percent (Oct. 5) and 25 percent of all devices, while Apple iPads accounted for 2 percent, 3 percent and 1 percent of devices for the respective Sept. 28, Oct. 5 and Oct. 24 games.

DAS usage also remains strong, but not tops in NFL

On the DAS side of the Levi’s Stadium network we have some stats from AT&T to share (but none from any other carriers). According to AT&T, for the Oct. 5 game AT&T customers used 549 GB of data, which was only the fourth-highest AT&T DAS total for that weekend. Dallas (827 GB), San Diego (716 GB) and New Orleans (598 GB) all had higher AT&T DAS traffic totals for that weekend’s games. (Remember, these are results for stadiums with AT&T DAS networks only, not for all stadiums.) For the Sept. 14 game, the Levi’s Stadium AT&T DAS recorded 673 GB of traffic, according to AT&T.

AT&T: DAS network also set traffic records for 2014 World Series games at AT&T Park

Giants CIO Bill Schlough (left) talks with workers in the park's main DAS head end facility.

Giants CIO Bill Schlough (left) talks with workers in the park’s main DAS head end facility.

In addition to the over-the-top Wi-Fi usage numbers, it turns out that the in-stadium AT&T cellular network also experienced record usage during the recent World Series games at AT&T Park, with Saturday night’s crowd using 477 Gigabytes of data, according to AT&T.

In case you’re not familiar with the DAS acronym, it stands for distributed antenna system, and basically is a network of small antennas that bring cellular service to the tightly packed fans inside stadiums or other large public venues. At AT&T Park, AT&T runs the network as a neutral host, meaning that Verizon Wireless, Sprint and T-Mobile customers can also use the DAS to connect. If you’re in the park and you haven’t enabled your device to connect via Wi-Fi, you’re probably connected via the DAS. However, the stats provided here are only for AT&T customers on the DAS, since AT&T doesn’t have visibility into the other carriers’ metrics. Since the Wi-Fi network is open to all, the Wi-Fi numbers we reported earlier are for all customers, no matter who their provider is.

But even just the AT&T cellular numbers are pretty impressive, perhaps not surprisingly so since a good-weather World Series game is a bucket-list event for most attendees, meaning that texts, tweets, selfies and Vines were likely flowing freely at all three games in San Francisco. Here’s a breakdown from AT&T about how much more data was used during the games last weekend:

– Fans used an average of approximately 447GB of data per game over the weekend on the AT&T cellular network. This is equivalent to more than 1.27M social media post with photos.

– The numbers represent an increase of approximately 29 percent in cellular data usage compared to the average game during the League Championship series vs. St. Louis.

– It’s an increase of approximately 109 percent in cellular data usage compared to the average game during the final home series of the regular season vs. San Diego (9/25-9/28).

– The peak hour of data usage during three home games was on 10/25 was from 5-6pm PT, the hour in which the first pitch occurred. In this hour more than 83GB of data crossed the AT&T venue-specific cellular network.

According to AT&T, the total combined Wi-Fi and AT&T DAS traffic hit a record high of 2.09 Terabytes on Oct. 25, the highest single-game total in AT&T Park network history. What would be even more interesting would be if we could get DAS statistics from Verizon, Sprint and T-Mobile to get the total-total number for wireless data consumed during the biggest games of the year.

Half a stadium using Wi-Fi? 2014 World Series games set Wi-Fi records at AT&T Park

Though this year’s World Series still isn’t over, the three games this past weekend in San Francisco set new local records for fan Wi-Fi use, arguably cementing AT&T Park’s title as baseball’s best-connected stadium.

One of the more stunning numbers provided to us by Bill Schlough, the senior vice president and CIO for the Giants, was that nearly half the crowd in AT&T Park for Sunday’s Game 5 used the Wi-Fi network — a “take rate” we’ve never heard of before at any sporting event of any kind. According to Schlough the Giants’ Wi-Fi network had 20,638 unique users during Game 5, a take rate of 49.7 percent. (According to the official box score, the attendance that night was 43,087, give or take a Panda or four.) The game also set an AT&T Park record for Wi-Fi upload traffic with 747 Gigabytes of data being sent by people at the park out to the network.

(Including one special photo tweeted out by a former Giant you may have heard of.)

A record for downloaded Wi-Fi traffic, of 937 GB, was set during Friday night’s Game 3, and the record for a combination of download and upload traffic was set during Game 4, with a total of 1.62 Terabytes. According to Schlough, the total data transferred during the eight games of the 2014 postseason at AT&T Park was 10.3 TB, which is easily the high-water mark for any baseball stadium network we know of. (The total does not include regular cellular traffic carried by the AT&T Park DAS; we will add those numbers in when they become available.)

And while individual games such as Super Bowls or contests down the San Francisco Peninsula at Levi’s Stadium may set higher overall marks (thanks in part to having more fans in the building), we’d be interested to see if any other stadium networks can match AT&T Park’s season-long data number of 59.3 TB, which according to Schlough represents total traffic over 2014′s 91 games (regular and postseason) at AT&T Park.

One more fun number from Schlough: Since the park turned on its fan-facing Wi-Fi network in 2004, there have been 4,085,518 Wi-Fi connections during the 946 game days. Kansas City’s Kaufmann Stadium, which just got its Wi-Fi network installed recently, has one or two more games this season to start trying to catch the Giants from a networking perspective.

UPDATE: Looks like the Royals are claiming some already-decent usage of their Wi-Fi network:

Confirmed: KC has MLB-provided Wi-Fi, part of plan to bring Wi-Fi and DAS to all MLB stadiums

KC fans at seriesThe rumored Wi-Fi network at Kansas City’s Kauffman Stadium is now a confirmed entity, according to Joe Inzerillo, executive vice president and CTO for MLB.com. In a phone call Thursday, Inzerillo said the Royals’ new network is part of a league-wide effort to bring Wi-Fi to all MLB parks, a task he thinks may take another year or two to complete.

As MLB.com has stated publicly before — but maybe not spelled out in so much detail — it has a program under which Major League Baseball teams can “opt in” to have MLB.com and the nation’s top four wireless carriers participate in the funding and building of both Wi-Fi and DAS networks in MLB stadiums. Though he wouldn’t divulge the specific financial commitments for specific deals, Inzerillo said that under the program “everyone has some skin in the game,” though he did allow that the league and the carriers, not the teams, foot the bulk of the bills.

Still, Inzerillo stressed that individual teams play a huge role in the Wi-Fi deployments, from design to deployment to management on site. “It’s not just like we show up and we’re the Wi-Fi fairies,” Inzerillo said. “This program wouldn’t be possible without the teams and the work they do.”

Screen Shot 2014-10-23 at 1.27.54 PMGiven MLB’s synchronized digital strategy of having the same app — and only the same app — available for fans in each ballpark, Inzerillo said that having high-quality cellular and Wi-Fi networks in each stadium was a key necessity, especially to make MLB app functions like seat upgrades and concession purchases work.

“You need to have the right [network] plumbing in place or none of the other stuff matters,” Inzerillo said.

Kauffman Stadium’s network, Inzerillo said, was just one of about a dozen MLB Wi-Fi projects that got underway this year. That it was finished in time for postseason play was just luck, and not some last-minute installation due to the Royals’ on-field successes. “It was just a fortuitous thing that it was ready,” said Inzerillo, who said that construction of Wi-Fi at Kauffman had been ongoing for the past 5 to 6 months.

Though the network wasn’t promoted on the team’s website or anywhere else on the Internet, Inzerillo said the Royals were promoting it at the stadium. Even without a lot of advertising, fans found the network, he said, claiming “tens of thousands” of Wi-Fi connections during the Royals’ postseason run. However, Inzerillo also said some extra cellular trucks were brought in by some of the carriers for the Royals playoff games because the DAS at Kauffman isn’t quite finished yet.

Inzerillo said that anywhere from 22 to 26 teams will eventually end up using some combination of league-provided Wi-Fi and/or DAS. The league’s goal of having every stadium fully wired should be nearly complete by opening day of 2015, he said, with a more likely “final” goal of complete coverage reached sometime in 2016. In 2014, Mobile Sports Report research showed that 10 of the 30 MLB stadiums didn’t have fan-facing Wi-Fi; some of those teams (like Kansas City) will be getting MLB networks, while some other program participants are upgrading existing systems, Inzerillo said.

Under the MLB network deal, the DAS in each stadium will be a neutral-host deployment hosted by one of the four major U.S. cellular carriers — AT&T, Verizon Wireless, Sprint and T-Mobile — depending on things like regional/historical market share and existing contracts, Inzerillo said. While the carriers will operate the MLB DAS deployments, the Wi-Fi networks will be deployed, run and monitored by MLB, either from its New York City or San Francisco network operation centers, Inzerillo said. Wi-Fi gear will come mainly from Cisco, though Inzerillo said there is also a small percentage of Meru Networks gear based on teams that had previously installed Meru equipment.

Though Inzerillo said MLB may make some overall announcement once the league-wide project is completed, he didn’t necessarily think that having working Wi-Fi and DAS in stadiums was such a big deal.

“It’s kind of a weird thing to think about bragging about,” said Inzerillo, who compared Wi-Fi and DAS to plumbing as a basic stadium necessity, not an amenity. Having high quality networks, he said, “are table stakes for a modern facility.”

UPDATE: The Kansas City Royals are now officially promoting the Wi-Fi, with some usage claims:

Giants: NLCS stadium Wi-Fi usage at AT&T Park quadrupled since 2012

Giants senior VP and CIO Bill Schlough, at the office

Giants senior VP and CIO Bill Schlough, at the office

If you build a better stadium network, they will come… and use it even more. That’s what the San Francisco Giants are finding out, with stadium Wi-Fi data usage more than quadrupling this year compared to the last time the Giants were in the NLCS in 2012.

According to Bill Schlough, senior vice president and CIO for the Giants, fans at AT&T Park for the recent National League championship series with the St. Louis Cardinals used more than four times the Wi-Fi data that fans used in the 2012 series. Schlough said that in 2012 (also facing the Cardinals) fans at the four AT&T Park NLCS games used an average of 302 Gigabytes per game; this year, in the three games in San Francisco, that number was 1,247 GB per game.

Screen Shot 2014-10-20 at 9.26.21 PM

What makes the jump in data even more interesting is the fact that the number of overall users of the Wi-Fi network only increased about 19 percent from 2012 to 2014, with an average of 16,683 users on the network this year compared to 13,509 in 2012, according to figures from Schlough and the Giants. “It definitely shows how data per connection is what is really exploding,” said Schlough in an email.

Looking back another couple years is also instructive; in 2010, during the NLCS the AT&T Park network only saw an average of 6,511 fans using Wi-Fi per game. (The team did not track data usage at that time.) Though the World Series starts today in Kansas City, fans will have to wait until Game 3 in San Francisco Friday to use Wi-Fi since the Royals’ Kaufmann Stadium is one of the 10 MLB parks that don’t have fan-facing Wi-Fi.

White box at bottom is one of the under-the-seat Wi-Fi access points at AT&T Park.

White box at bottom is one of the under-the-seat Wi-Fi access points at AT&T Park.

Stadium Tech Report: Nebraska adds Cisco-powered Wi-Fi, IPTV to Memorial Stadium

Memorial Stadium, University of Nebraska. Credit all photos: University of Nebraska.

Memorial Stadium, University of Nebraska. Credit all photos: University of Nebraska.

The state of Nebraska might not have a professional football team, but University of Nebraska fans now have a pro-style wireless experience at home games thanks to a high-density Wi-Fi network and IPTV features recently installed by the school at the 85,000-plus seat Memorial Stadium.

Combined with a new mobile app that delivers multiple live video streams and replay options, the loyal Cornhusker fans — who’ve come to Lincoln, Neb., and filled Memorial Stadium to the brim every game day since the 1960s — now have a wireless fan in-game experience among the best anywhere, collegiate or pro. Installed for use this season by CDW, the new network features Cisco Wi-Fi gear and Cisco’s StadiumVision and StadiumVision Mobile systems, as well as a new app developed by NeuLion and additional video-streaming capabilities from EVS.

“We have the most loyal fans in the country,” said Kelly Mosier, director of digital communications for the University of Nebraska athletic department. Part of putting in a high-definition wireless network, Mosier said, was to meet growing fan connectivity needs and to “stay ahead of the curve” in stadium experiences. “We wanted to reward the fans for being so loyal, and give them a new experience to brag about,” Mosier said.

A look at video options in the Husker app

A look at video options in the Husker app (click picture for bigger view)

One of the first collegiate programs to install big TV screens in its stadium, Nebraska now has a Wi-Fi network with more than 800 access points, both inside Memorial’s seating bowl as well as outside, at entrance and parking areas. In addition to synchronized IPTV broadcasts on stadium flat-panel screens, Nebraska fans also have access to a wide range of live video and video replays of game-day action, thanks to the new game-day app.

According to Dan Floyd, director of information technology for Nebraska athletics, the new network is already a hit, with a peak of 25,000 simultaneous users during the second game this fall. And even though Floyd and Mosier are aware of some problem areas, as Floyd said there has already been “a lot of positive feedback” from the technology upgrades.

Better sound, better connectivity

The network deployment was part of a recent $12.3 million upgrade to the football facility, which has been the home of the Huskers since 1923. Included in the total spend was also a revamp of the public-address and sound systems, but according to Mosier “the biggest chunk” of the spending went to cover the Wi-Fi deployment.

The need for better connectivity became apparent a few years back, Mosier said, and it set in motion a “very long process” of a couple years in length in which Nebraska officials looked at other existing stadium deployments and technology choices before making their decisions. One of the easier picks was selecting Cisco as the main Wi-Fi gear supplier, since Mosier said that most of the rest of the school’s campus was already wired with Cisco networking gear. And a recent deployment of Cisco stadium Wi-Fi at Pinnacle Bank Arena in downtown Lincoln, Neb., where the Husker basketball team plays, helped further push the football decision toward using Cisco.

“We wanted to provide a seamless experience for fans between multiple venues, and to play well with the rest of the campus,” Mosier said. “It was just a no-brainer to use a Cisco [Wi-Fi] product.”

Also in Cisco’s favor was its StadiumVision digital display system and its StadiumVision Mobile product, which both bring advanced IPTV features to static stadium screens as well as to mobile apps. StadiumVision allows for synchronization of programming across a wide array of networked displays, while StadiumVision Mobile supports several live broadcast channels that can be used to provide live content to mobile apps. The Brooklyn Nets use StadiumVision Mobile to bring live action channels to fans who visit the Barclays Center for games.

“We wanted to not just provide the networking infrastructure, but on top of that provide something extra,” Mosier said. “Our fans are pretty savvy, and they are looking for things beyond what the casual fan might be looking for. StadiumVision and StadiumVision Mobile makes sense for our fan base. The ‘extra screen’ approach really lets them control their own video experience.”

Some additional replay options in the Husker app

Some additional replay options in the Husker app

On the Huskers’ game day app, Mosier said, fans at the start of the season could choose between three StadiumVision Mobile-powered “channels” that showed the big-screen broadcast, an alternate angle view, and an “all-22″ camera that is like what coaches view to see the players across the entire field. A “phase II” of the app live video, which had not yet launched at the time of our interview, will include further user-controlled selections for more camera angles and replays. According to Mosier, the Phase II capabilities are supported by the C-Cast system from EVS. The entire new app, he said, was built by developer NeuLion.

Though Mosier said the live video production for the app was “definitely a growing process” that will require further tuning and learning, he said the system already has impressive performance, with delays between live action and app action at “a second to a second and a half.” Mosier said that Nebraska also plans to bring more live action to the mobile app for basketball season, with multiple camera angles including a “slam cam” based near the rims.

Handrails and fan interference of Wi-Fi signals

While the 800 Wi-Fi APs give Nebraska pretty good coverage throughout the facility, Floyd said the IT team knew it would have connectivity challenges in the north and south end zone stands, since neither of those sides have any structural overhangs.

“Since the north and south sides [of the stadium] don’t have overhangs we knew they would be problematic,” Floyd said. One option used in venues including AT&T Park and Levi’s Stadium, the under-the-seat access point, wasn’t an option at Memorial Stadium, Floyd said, because the seats there aren’t high enough to meet safety requirements for keeping bodies away from the antennas.

One creative way CDW and Nebraska brought Wi-Fi to the north and south stands is via Wi-Fi antennas inside railing enclosures, but those are not without their own challenges. Though the railing antennas get a signal close to fans, the long rows of seats at Memorial Stadium — up to 30 in between rows — means that the “waterbags,” or human bodies, can act as signal-blockers for fans in the middle of the rows.

“The first 10 or 12 people on either side get a pretty good signal,” said Floyd, but he added that the fans in the middle are still a challenge to reach. Both Mosier and Floyd said Nebraska will continue to seek ways to upgrade the Wi-Fi network, including possibly putting APs on top of towers or in other creative deployments.

“We knew it would need tuning, and some tuning is easier than others,” Mosier said. “We know we still have pockets of [connectivity] problems. We knew that when we put in a system like this, it wasn’t going to be perfect on day 1.”

Something to brag about

According to Floyd, Memorial Stadium has had a Verizon DAS in place for several years, which fulfilled most of the fans’ basic cellular connectivity needs since he said that “70 to 75 percent” of Cornhusker fans were Verizon customers. However, the new sound system, with its big speaker arrays, has also given room for AT&T and U.S. Cellular to add some DAS equipment of their own, with antennas mounted right inside the speaker enclosures.

Unlike other schools or teams, the Cornhuskers are not pressed to make money off their wireless network, given the stadium sellout streak that dates back to Nov. 3, 1962. But Mosier said that even the Huskers aren’t immune to the lure of the living room couch, with its comfort, HD screen and close-by food and drink.

“We definitely have a blessed situation [with the sellouts],” Mosier said, while allowing that some fans might still prefer sitting at home. “But you can’t match the experience of being at the venue,” he added. “If we can address the connectivity issues, plus add to the stadium experience [with technology], it’s a win-win for us.”

Using the app at Memorial Stadium

Using the app at Memorial Stadium