Commentary: Will stadium networks ever catch up with demand?

Dallas fan in mobile action at AT&T Stadium. Photo: Phil Harvey, MSR

Dallas fan in mobile action at AT&T Stadium. Photo: Phil Harvey, MSR

If there’s one thing the stadium-networking industry learned — or should have learned — from the stunning stats from Super Bowl 50, it’s that nobody really knows where or when the demand for in-venue bandwidth will stop growing.

Thanks to the smart folks who built the Levi’s Stadium Wi-Fi network — and the smart deployers and carrier partners who doubled down on the DAS this summer — the venue was able to handle an incredible 26 terabytes of wireless data on Super Sunday, with 15.9 TB amongst the four major cellular carriers on the DAS, and 10.1 TB on the stadium’s Wi-Fi network. Both were easily new records for single-day events, far eclipsing the totals seen at Super Bowl XLIX the year before.

The question, of course, for everyone else is: What comes next? And what does that mean for networks built in the past few years?

Keep building for growth

Editor’s note: This column is an excerpt from our latest STADIUM TECHNOLOGY REPORT, which is available for FREE DOWNLOAD from our site. In addition to this analysis and stadium tech deployment profiles we also take an in-depth look at the new trend of deploying Wi-Fi and DAS antennas under seats, and provide a wireless recap from Super Bowl 50. GET YOUR COPY today!

Texas A&M student recording the halftime show. Photo: Paul Kapustka, MSR

Texas A&M student recording the halftime show. Photo: Paul Kapustka, MSR

In some follow-up discussions with folks from Aruba, the Levi’s Stadium networking team, Verizon Wireless and others in the industry, I wondered out loud if what we are seeing is just yearly growth in demand, or whether expanded networks are merely satisfying pent-up demand that was always there.

To be sure, the “big” events like the Super Bowl, the World Series and the Final Four are always going to generate outsize traffic numbers, especially as more and more fans join the selfie craze and post photos and videos from the stadium, showing the rest of the world how great their life is. According to Verizon, the amount of traffic uploaded on their DAS network at Super Bowl 50 was double the upload traffic the carrier saw at Super Bowl XLIX, validating Verizon’s belief that traffic was going to grow significantly.

For mobile traffic in general, and for in-stadium wireless activity specifically, the continued growth over the past few years can likely be traced to a number of factors, including the ever-increasing power of mobile devices; the new number of bandwidth-hungry apps that incorporate images and video; and a steady increase in familiarity with devices and apps, as older folks — those more likely to purchase event tickets — catch up to their kids in wanting to use, and being able to use, the content-sharing features of social media and other communication apps. And, simply, more people trying and successfully connecting to in-stadium networks, reversing historical thinking that told them from experience in past years that connecting at a stadium was futile. In almost all venues these days, no longer is that the case.

So with no end yet in sight to the general doubling of traffic on a year to year basis, the big question out there for stadium technology teams has to be — is what we have now enough to handle growing needs, and if not, what are we going to do about it? The fact that cellular carriers (correctly) assumed they needed to add capacity to the Levi’s Stadium DAS a year after it was deployed should give pause to anyone building a network right now; though your venue may not be hosting the Super Bowl anytime soon, it might not hurt to re-think the traffic projections that have been made and step them up a bit, just in case.

These signs are up all over AT&T Park. Photo: Paul Kapustka, MSR

These signs are up all over AT&T Park. Photo: Paul Kapustka, MSR

On both the Wi-Fi side and the DAS side, we may also be looking soon at a need for new kinds of technology to help build the networks of the near future. In this issue we take an in-depth look at the trend toward putting antennas for both Wi-Fi and DAS under seats, mainly to build networks that are more dense and can handle more traffic.

Verizon, which pioneered DAS under-seat antennas at Levi’s Stadium this year, said it is already seeing stadiums where just adding another sector of traditional DAS produces diminishing returns; if traffic on both types of networks keeps growing at the current rate, when will existing designs become obsolete? And what will replace them?

Stadium network owners shouldn’t ignore policy

As stadium-networking types ponder the future, it would serve the industry well to start thinking together as well, especially in the areas of telecom policy and standards-setting. As the builders of telecom networks worldwide ponder the future of 5G technologies there are some discussions where stadium networking representatives are nowhere to be found, especially in the potentially troubling direction of LTE-U, the idea of carriers using unlicensed bands for LTE traffic.

It’s still early days in the LTE-U discussions, but a recent decision by the FCC to allow carriers to conduct tests of the technology should be a call to action for the stadium networking industry. While nobody knows quite for sure yet what will happen to an in-venue Wi-Fi network if and when LTE-U traffic appears, it seems to make sense for the industry to get together and at least present some kind of unified voice before decisions get made.

At the recent Silicon Flatirons telecom policy conference in Boulder, Colo., there were exactly zero representatives from the venue Wi-Fi ecosystem present, including no visible representation from the Wi-Fi and DAS gear manufacturers who sell into the stadium networking space. With telecom providers hungry for spectrum of any kind, it’s a fool’s bet to assume that the unlicensed bands used in stadiums worldwide today will remain as free and available as they are now. It’s just another factor that’s needed to be considered, as venue tech professionals plan for a future of ever-increasing needs.

New Report: Super Bowl 50’s super wireless, under-seat Wi-Fi feature and more!

STR Q1 THUMBThe record-setting wireless network consumption at Super Bowl 50 is one of the lead topics in our latest STADIUM TECH REPORT, our long-form publication that takes an in-depth look at the most important news of the stadium technology world, alongside some great in-depth profiles of successful stadium technology deployments. Download your free copy today!

With fans consuming 26 terabytes of wireless data — 15.9 TB on the stadium’s distributed antenna system (DAS) and another 10.1 on the Wi-Fi network — the Super Bowl provided the ultimate test for the Levi’s Stadium wireless infrastructure, one that the venue passed with flying colors. One unique factor of the stadium’s wireless deployment, under-seat antennas for both the DAS and the Wi-Fi networks, is covered in-depth in our most recent issue, with a feature story about how under-seat deployments got started, and why they may become the default antenna placement for large public venues going forward.

Also in the issue: A profile of Wi-Fi and associated mobile device strategies at the University of Wisconsin, including geo-fencing for fan marketing at away games; a close-up look at the wireless infrastructure at the Denver Broncos’ Sports Authority Field at Mile High; a profile of the new Wi-Fi network at the Montreal Canadiens’ Bell Centre; and a look at some new social-media strategies deployed by the Miami Dolphins. All this information is available now for FREE DOWNLOAD so get your copy today!

We’d like to thank our Stadium Tech Report sponsors, who make this great content free for readers thanks to their support. For our Q1 issue our sponsors include Mobilitie, Crown Castle, CommScope, Samsung, Corning, JMA Wireless, Aruba, SOLiD, Xirrus and 5 Bars.

Levi’s Stadium crowd sets single-day Wi-Fi record with 10.1 TB used at Super Bowl 50

Broncos fans celebrate during Super Bowl 50 at Levi's Stadium. Photo: LevisStadium.com

Broncos fans celebrate during Super Bowl 50 at Levi’s Stadium. Photo: LevisStadium.com

The 71,088 fans at Levi’s Stadium for Super Bowl 50 helped set a single-day record for Wi-Fi usage, with 10.1 terabytes of traffic on the stadium network, according to the NFL and the San Francisco 49ers network staff.

According to figures provided to us by Roger Hacker, senior manager for corporate communications for the Niners, the Super Bowl 50 crowd broke last year’s previous Wi-Fi record of 6.23 TB by halftime, and ended up with the 10.1 total after recording traffic from 6 a.m. local time until 11 p.m. Of that total, 9.3 TB was used by fans on the free Super Bowl network and another 453 GB was used by media at the game. The remainder of 370 GB was used on dedicated internal operations networks, Hacker said.

When the Wi-Fi number is added to the 15.9 TB of cellular data used at the game, the total of 26.0 TB of wireless traffic is fairly stunning, and perhaps a wake-up call to current network operators at large public venues or those designing new ones, signifying that the usage pattern for mobile data at big events is still growing rapidly, with no top yet in sight.

Levi’s Stadium also set other Super Bowl connectivity records, the first by recording 27,316 unique Wi-Fi users and 20,300 concurrent users (set at 5:55pm PT), topping the previous Super Bowl records from last year of 25,936 uniques and 17,322 concurrent users, respectively. The previous max for concurrent Wi-Fi users at Levi’s Stadium was 18,901 for the stadium’s inaugural regular season game vs. the Chicago Bears on Sept. 14, 2014. At that game, the stadium saw 3.3 TB of Wi-Fi use.

Also new records for sustained connectivity and average use

While we’re still waiting for news about usage of the Super Bowl stadium app, there are some more record-setting stats to note: According to the stadium IT figures, the big-bandwidth day also saw a Levi’s Stadium record for peak Wi-Fi bandwidth used at 3.67 Gbps — this number is the total amount of bandwidth going through the network at a single moment in time, in this case at 3:25 p.m. Pacific Time. The previous record was a mark of 3.55 Gbps set during the Coors Light Stadium Series hockey game on Feb. 21, 2015, a night when not everything went well on the stadium-network side.

Sunday at Super Bowl 50 there were no apparent big glitches, with some Twitter complainers noting that stadium network technicians were quick to respond to any mentions of network downtime. Bandwidth provider Comcast has an interesting infographic of game-day data use, and said the peaks in Wi-Fi network activity happened during the following list of Super Bowl moments:

The 10 moments that generated the most data traffic at the stadium included:

The introduction of the 50 Super Bowl MVPs

Lady Gaga singing the National Anthem and the Blue Angels flyover

The opening kickoff

The first coach’s challenge

Von Miller’s forced fumble and the first touchdown of the game by Malik Jackson

The halftime show with Coldplay, Beyonce and Bruno Mars

Von Miller’s second forced fumble and C.J. Anderson’s game-sealing touchdown

Peyton Manning exiting the field and Gary Kubiak’s Gatorade shower

The Lombardi Trophy presentation

Using apps to get back home and to hotels

For those who are interested, here is our updated list of the top five big-venue single-day Wi-Fi records. If anyone has one to add to this list, please let us know!

1) 10.1 TB — Super Bowl 50, Levi’s Stadium, Santa Clara, Calif., Feb. 7, 2016
2) 6.23 TB — Super Bowl XLIX, University of Phoenix Stadium, Glendale, Ariz., Feb. 1, 2015
3) 5.7 TB — Alabama vs. Texas A&M, Kyle Field, College Station, Texas, Oct. 17, 2015
4) 4.93 TB — College Football Playoff championship game, AT&T Stadium, Arlington, Texas, Jan. 12, 2015
5) 4.9 TB — College Football Playoff championship game, University of Phoenix Stadium, Glendale, Ariz., Jan. 11, 2016

Congrats to the Niners, the NFL, Aruba, Comcast, and Brocade, as well as DAS Group Professionals, DAS gear supplier JMA Wireless and all the major cellular carriers, Verizon Wireless, AT&T, T-Mobile and Sprint, who all made exceptional efforts to ensure great connectivity for fans across the board.

UPDATE: Top 4 carriers combine for 15.9 TB of cellular data use at Super Bowl 50

New Verizon Wireless under-seat DAS antenna placement at Levi's Stadium. Photo: Verizon Wireless

New Verizon Wireless under-seat DAS antenna placement at Levi’s Stadium. Photo: Verizon Wireless

UPDATE, 2/8/16, 1:50 p.m. — We now have data totals in from all four of the major U.S. cellular carriers, and at Sunday’s Super Bowl 50, fans combined to use 15.9 terabytes of data on the networks in and directly around Levi’s Stadium in Santa Clara, Calif.

Leading the way in usage was Verizon Wireless with a claim of 7 TB used by its customers; AT&T was next with 5.2 TB of claimed usage, followed by T-Mobile with a report of 2.1 TB, and Sprint with 1.6 TB. All the carriers’ numbers are well above figures from last year’s Super Bowl, where by our reporting Sprint, AT&T and Verizon had a combined 6.56 TB of cellular data consumed during the big game. (We did not have any T-Mobile reports from last year.)

For all the carriers, the data apparently includes both traffic on the in-stadium distributed antenna system (DAS) network was well as any macro deployments outside the stadium in parking lot areas. The final total was well over double the 6.56 TB of cellular traffic seen at last year’s big game in Glendale, Ariz. We are still waiting for Wi-Fi numbers from the Levi’s Stadium networking crew but it’s a good bet the 6.23 TB number from last year’s game will be eclipsed and we will have a new single-game Wi-Fi record as well so stay tuned.

Though we did hear and see some scattered reports of network connectivity issues during the Denver Broncos’ 24-10 victory over the Carolina Panthers it appears the upgrade of the DAS Group Professionals DAS install at Levi’s Stadium with its gear mainly provided by JMA Wireless stood up to the biggest-ever test of traffic. Congrats to all involved.

Thanks also to the Verizon and AT&T crews who supplied us with tweet reports and emails Sunday night, it made for some entertaining in-game stats. Some tweets embedded below.

Commentary: Super Bowl 50 and the pursuit of stadium network statistics

Dan Williams talks Wi-Fi while the Levi's Stadium new turf grows silently behind him in this 2014 photo. All photos: Paul Kapustka, MSR

Dan Williams talks Wi-Fi while the Levi’s Stadium new turf grows silently behind him in this 2014 photo. All photos: Paul Kapustka, MSR

If there’s one person who didn’t get enough recognition in the run-up to today’s Super Bowl 50 at Levi’s Stadium, it’s Dan Williams, the former director of information technology for the Niners and the man who was largely behind the venue’s celebrated network design.

From our standpoint, Williams deserves credit not just for leading the deployment of a stadium network that set the standard for the future of connected venues, but also for being among the first to openly talk about the network’s performance, sharing statistics both good and bad during his short stay at Levi’s Stadium. His honesty helped open the door wider on a potentially rich stream of information that we think could greatly assist stadium network professionals everywhere, especially if more teams followed his lead and shared stats openly and honestly.

But the pursuit of meaningful network statistics remains a challenging process on many levels, and it’s one that is ironically likely to get more confusing today as multiple parties are set to release live network statistics from Super Bowl 50 at Levi’s Stadium. Both Verizon Wireless and AT&T are planning to release live game-day performance figures for their cellular networks in and around the venue today, and we also expect to get Wi-Fi network statistics from the San Francisco 49ers, if not during the game then shortly thereafter.

Niners VP of technology Dan Williams attempts to fix my Droid 4 Wi-Fi issues (while trying not to laugh at the fact that I actually have and use a Droid 4) during the first preseason game at Levi's Stadium, July 2014

Niners VP of technology Dan Williams attempts to fix my Droid 4 Wi-Fi issues (while trying not to laugh at the fact that I actually have and use a Droid 4) during the first preseason game at Levi’s Stadium, July 2014

While we salute the change of heart (especially from Verizon) to make more information available, we’d also like to make a plea here for more independent access to venues on game day, so there can be an independent, objective voice on hand to counter the “sanitization” or simple non-reporting of less-impressive data that is almost sure to occur when interested parties are talking about themselves. But unless team, league or venue IT representatives band together and craft recognized standard methods of reporting the most-interesting data, we’re going to be left to sift through an ever-increasing mix of figures provided by those with their own agendas on what’s important and what should remain hidden. We’ll do our best on that front, but it’d be better if we had some industry-wide help.

Out of school behavior provides the most interesting info

Like many other stadium network professionals we’ve been fortunate enough to meet and spend time with — a list here that includes folks like the San Francisco Giant’s Bill Schlough, Madison Square Garden’s Katee Panter, the Dallas Cowboys’ John Winborn, and Chip Foley, formerly with the Barclays Center, among many others — Williams from the start showed a passion for making the network better, so it could better serve the fans who used it. Our first face to face meeting came during the first preseason football game at Levi’s Stadium, when I tweeted that I was having problems connecting to the Wi-Fi network. William’s response? An email asking me where I was sitting, followed quickly by an in-person visit. Now that’s customer support.

The outcome of that interaction was a revelation that the Levi’s Stadium Wi-Fi network hadn’t prepared adequately for older devices that were still using the 2.4 GHz unlicensed bands; what was cool to me was not just that I kind-of discovered it by accident, but that Williams and his team admitted it publicly, then went about fixing it. The combination of objective reporting and honest confirmation, I think, worked to the benefit of all. But not everybody sees such things in the same light.

Season opener issues: Picture of app late in the first half.

Season opener issues: Picture of app late in the first half.

For instance, during the rest of Levi’s Stadium’s first year of operation, Williams shared with us not just raw “tonnage” numbers of Wi-Fi use but also interesting stats from the app, like exact numbers of replays watched and in-seat food delivery orders. Since those numbers were kind of low — in the low thousands for food deliveries and a decline in replay use after the season opener — they weren’t necessarily flattering to the owners of the system, even if you took into account the fact that some of this (like delivering to any seat in a 68,500-seat stadium) had never been tried before. In the end, even Niners CEO Jed York admitted that some elements of the stadium hadn’t performed as expected, as the stats we’d been seeing had borne out.

After Williams left the team, the Niners took control of the network stats reporting and hugely trimmed down the list of things they were willing to report, though we would also note that Roger Hacker and the Niners communications team are still at the top of the list when it comes to consistent, repeated reporting of the network stats they feel comfortable talking about. When there are big numbers to report, all goes well because everyone loves to talk about systems that deliver for fans as promised. But I can also tell you that when things go south, so does a lot of the honest communication, an unfortunate if understandable situation.

Who will tell you when things don’t work?

For Levi’s Stadium, the darkest night for the network came on Feb. 21, 2015, at the NHL’s Coors Light Stadium Series outdoor hockey game between the San Jose Sharks and the Los Angeles Kings. In what has easily been our highest-traffic post ever, we described in detail some of the big problems that surfaced that evening, which included breakdowns of the delivery service, the Wi-Fi network, the light rail boarding process, and part of Verizon’s cellular network. What’s funny about our reporting is that I hadn’t planned on “working” that night — I had bought full-price tickets for myself and my brother, and as longtime hockey nuts we proudly wore our Blackhawks jerseys to represent our hometown team.

Screen shot of Levi's Stadium app during hockey game issues

Screen shot of Levi’s Stadium app during hockey game issues

My plan was just to relax, take CalTrain and VTA to the stadium and show off the network by ordering us food and drink to be delivered to our seats, but when some obvious and incomprehensible network breakdowns were evident, I went into reporting mode, and eventually found out mostly what went wrong. My point here is that, without independent, objective reporting live from the scene, it’s doubtful that parties with vested interests — like cellular providers or owners of the stadium network — are going to honestly and quickly talk about breakdowns or failures. Yet it’s always the lessons learned from problem times that help the most; my worry is that with more “sponsored” or in-house statistics flooding the zone, the more valuable objective data will get lost or devalued.

How can this be fixed? For starters, teams and leagues that spend an inordinate amount of time and money promoting technical features of stadiums should at the very least credential some reporters who are interested in following up to see how said deployments perform. While we’re grateful for the few forward-thinking organizations that have given us the same access that sports reporters get, more often than not our requests for media access to events are denied.

Since we can’t be at every stadium, we also rely on teams, venues and network operators to provide stats when possible, and we encourage more participants to join the growing list who do provide us with network performance figures, like the IT teams at Nebraska and Texas A&M, among others. While we have also drawn some industry criticism for our headline focus on such stats, our response is that the more data we get and the more opinions on which data matter, the better we will get over time. We also encourage any and all interested parties to attend the SEAT Conference this summer in Las Vegas, where you can be sure we will have discussions about this topic and how to make stats reporting more useful for those to whom it matters most — the professionals who install and operate these networks so that fans can stay connected at events.

MSR on the scene at Sports Authority Field at Mile High in January

MSR on the scene at Sports Authority Field at Mile High in January

Niners: All (tech) systems go at Levi’s Stadium for Super Bowl 50

Levi's Stadium, ready for the Super Bowl. All stadium photos: Levi's Stadium (click on any photo for a larger image)

Levi’s Stadium, ready for the Super Bowl. All stadium photos: Levi’s Stadium (click on any photo for a larger image)

As far as the technology at Levi’s Stadium is concerned, it’s all systems go for Sunday’s Super Bowl 50, according to San Francisco 49ers chief operating officer Al Guido.

In a phone interview with Mobile Sports Report, Guido said the 2-year-old stadium’s vaunted technology underpinnings — especially the wireless connectivity for fans — is ready to go for the NFL’s biggest yearly event, after a second season spent mainly fine-tuning the different components.

“We couldn’t feel more confident, hosting the game,” said Guido, speaking specifically about the technology infrastructure at Levi’s Stadium. As he stated before the regular season began, the Niners didn’t do anything radical to the stadium’s Wi-Fi network, which uses gear from Aruba, an HP Enterprise company, to bring the main wireless bandwidth to fans.

And while the stadium’s distributed antenna system (DAS) got a complete replacement over the summer, the new capabilities including under-seat DAS antennas for Verizon Wireless should only lead to better reception than the year before. According to Guido, representatives from Aruba as well as from “all the carriers” will be on hand Super Sunday just in case anything needs close attention.

“Everybody’s going to be at a high tech [support] level” on game day, Guido said.

Drink delivery order page on Super Bowl stadium app, including the $13 Bud Light.

Drink delivery order page on Super Bowl stadium app, including the $13 Bud Light.

No food, but in-seat beverage delivery as part of stadium app

The Super Bowl 50 stadium app, designed for the NFL by the Niners’ in-house app development company VenueNext, will have some but not all of the features Niners fans have available during the regular season. The most obvious omission is the lack of food delivery to all seats, something that makes Levi’s Stadium stand apart from any other large public sporting venue. Instead, the stadium app will only allow fans to order beverages for in-seat delivery, with the option to order food, beverage and merchandise that can be claimed at “express pickup” concession windows.

According to Guido, the decision to only have beverage deliveries at Super Bowl 50 was one reached jointly by the NFL, the Niners and VenueNext, and the catering company for the stadium, Centerplate. Guido said that the potential “amount of education” for all the fans new to the stadium and new to the app led the league, the Niners and the caterers toward a path of greater simplicity, namely just having beverages available for in-seat delivery.

“It was a risk-reward decision about the amount of fan education needed,” Guido said. “There’s so much going on at a Super Bowl and so many people new to the stadium that it didn’t seem worth it to us to risk someone not getting an order delivered because of their error, or our error.” Guido added that with all the extra breaks in action for a Super Bowl, and additional concessions stands, “there’s enough time to get around” to get food.

View of the temporary media towers on the Dignity Health concourse

View of the temporary media towers on the Dignity Health concourse

Michelle McKenna-Doyle, senior vice president and chief information officer for the NFL, told Sports Business Journal that the league was also concerned about game-day delivery traffic patterns being disrupted by the new media towers that have been built for the game in the corner plaza areas of the stadium. “We were worried about having to keep up with demand … and we need to keep the aisles clear, which is important to the security team,” McKenna-Doyle said in a story by SBJ’s Don Muret.

The app will, however, include its normal live wayfinding capabilities, which should prove useful to new visitors to Levi’s Stadium since they can watch themselves walk through a map of the facility as a familiar moving blue dot. Like it does for Niners games, the app will also have instant replays from multiple camera angles available, as well as Super Bowl extras like a “celebrity cam” and the ability to watch Super Bowl commercials right after they are broadcast on TV.

Guido said the Levi’s Stadium app performed well all season, with an average of about “2,000 to 2,500” in-seat delivery orders per game. What was especially pleasing to the team was the number of fans who used the app’s ability to support digital ticketing, a feature that makes life somewhat simpler for fans but exponentially better for the team, which can gain valuable marketing insight from digital ticket-use statistics. According to Guido almost 35 percent of fans used digital ticketing during the past season.

Media towers save seats for fans

Niners fans watching Sunday’s game on TV might be surprised by the media towers, which Guido said were built in the Intel and Dignity Health concourse areas, which during regular-season games are simply open spaces. Guido said the decision to build temporary facilities for media means that the regular stadium seats will be saved for fans. At many other pro championship or playoff events, the overflow media are often housed in regular seating areas.

“The NFL made a great decision there” to put the media in the pavilions, Guido said.

If there is one thing that can’t really be controlled, it’s the traffic and transportation issues of bringing fans to the game. On Sunday fans coming to the game will confront Levi’s Stadium’s unique location in the middle of many Silicon Valley corporate headquarters buildings, which presents challenges that stadiums like AT&T Stadium in Dallas or the University of Phoenix Stadium in Glendale, Ariz. — which are surrounded by acres of stadium-controlled parking lots — simply don’t have. To help with Super Bowl traffic the planners are using multiple methods, including using Google employee buses as shuttles as well as signing Uber as a sponsor with its own dedicated pickup and dropoff lot. There is also light rail service which stops right outside the stadium, which intially in the past experienced lengthy delays especially after games, but has improved over time.

“Traffic and transportation is our largest concern,” Guido said.

Bring on the players and fans!

Bring on the players and fans!