Levi’s Stadium sees 5.1 TB of Wi-Fi data used at college football championship

Fans and media members at Monday night’s College Football Playoff championship game used a total of 5.1 terabytes of data on the Wi-Fi network at Levi’s Stadium, according to figures provided by the San Francisco 49ers, who own and run the venue.

With 74,814 in attendance for Clemson’s 44-16 victory over Alabama, 17,440 of those in the stands found their way onto the stadium’s Wi-Fi network. According to the Niners the peak concurrent connection number of 11,674 users was seen at 7:05 p.m. local time, which was probably right around the halftime break. The peak bandwidth rate of 3.81 Gbps, the Niners said, was seen at 5:15 p.m. local time, just after kickoff.

In a nice granular breakout, the Niners said about 4.24 TB of the Wi-Fi data was used by fans, while a bit more than 675 GB was used by the more than 925 media members in attendance. The Wi-Fi data totals were recorded during an 8-1/2 hour period on Monday, from 1 p.m. to 9:30 p.m. local time.

Added to the 3.7 TB of DAS traffic AT&T reported inside Levi’s Stadium Monday night, we’re up to 8.8 TB total wireless traffic so far, with reports from Verizon, Sprint and T-Mobile still not in. The top Wi-Fi number at Levi’s Stadium, for now, remains Super Bowl 50, which saw 10.1 TB of Wi-Fi traffic.

AT&T: Lots of DAS traffic for college football championship

DAS on a cart: DAS Group Professionals deployed mobile DAS stations to help cover the parking lots at Levi’s Stadium for the college football playoff championship. Credit: DGP

This may not be a news flash to any stadium network operations team but the amount of mobile data consumed by fans at college football games continues to hit high levels, according to some new figures released by AT&T.

In a press release blog post where AT&T said it saw 9 terabytes of cellular data used over the college football playoff championship-game weekend in the Bay area, AT&T also crowned a cellular “data champion,” reporting that Texas A&M saw 36.6 TB of data used on the AT&T networks in and around Kyle Field in College Station, Texas.

(Actually, AT&T pointedly does NOT declare Texas A&M the champs — most likely because of some contractural issue, AT&T does not identify actual stadiums or teams in its data reports. Instead, it reports the cities where the data use occurred, but we can figure out the rest for our readers.)

For the College Football Playoff championship, AT&T was able to break down some specific numbers for us, reporting 3.7 TB of that overall total was used inside Levi’s Stadium on game day. Cell traffic from the parking lots and tailgating areas (see photo of DAS cart to left) added another 2.97 TB of traffic on AT&T’s networks, resulting in a game-day area total of 6.67 TB. That total is in Super Bowl range of traffic, so we are excited to see what the Wi-Fi traffic total is from the game (waiting now for the college playoff folks to get the statistics finalized, so stay tuned).

DAS antennas visible at Levi’s Stadium during a Niners game this past season. Credit: Paul Kapustka, MSR

For the additional 2+ TB of traffic, a footnote explains it somewhat more: “Data includes the in-venue DAS, COWs, and surrounding macro network for AT&T customers throughout the weekend.”

Any other carriers who want to add their stats to the total, you know where to find us.

Back to Texas A&M for a moment — in its blog post AT&T also noted that the stadium in College Station (which we will identify as Kyle Field) had the most single-game mobile usage in the U.S. this football season, with nearly 7 TB used on Nov. 24. Aggie fans will remember that as the wild seven-overtime 74-72 win over LSU, an incredible game that not surprisingly resulted in lots of stadium cellular traffic.

Niners, SAP announce stadium-operations management application

A sample screen shot from the new Executive Huddle stadium operations management platform, developed by SAP for the San Francisco 49ers. Credit: San Francisco 49ers (click on any photo for a larger image)

A desire by the San Francisco 49ers to see stadium operations information in real time has become a real product, with today’s announcement of Executive Huddle, a stadium operations management application developed for the Niners by SAP.

In use at the Niners’ Levi’s Stadium since the start of the current football season, Executive Huddle brings transaction information from nine different stadium operations systems, including parking, concessions, retail sales, weather and fan opinions into a visual output that allows team executives to make real-time decisions on how to fix problems or otherwise enhance the game-day experience.

Demonstrated at Sunday’s home game against the Los Angeles Rams, the software not only reports raw data like concession sales or parking lot entries, but also provides a layer of instant feedback to let team executives make immediate changes to operations if necessary. The cloud-based application, developed by SAP and Nimbl, is currently only in use at one upper-level suite at Levi’s Stadium, where the output runs during Niners’ game days on several video screens. SAP, however, plans to make the system available to other teams in the future, according to SAP executives at Sunday’s demonstration.

Fixing issues in real time

Al Guido, president of the 49ers, said Executive Huddle was the end product of a desire of his to be able to fix any game-day experiences on the day of the game, instead of in the days or weeks after. According to Guido, the Niners have been passionate about collecting fan-experience data since Levi’s Stadium opened in 2014. But in the past, the compilation of game-day data usually wasn’t complete until a day or two after each event, meaning any issues exposed were only learned lessons that needed to wait until the next games to be fixed.

Executives huddle: from left, SAP’s Mark Lehew, Niners’ Moon Javaid, SAP’s Mike Flannagan and Niners president Al Guido talk about the Executive Huddle system at a Sunday press event at Levi’s Stadium. Credit: Paul Kapustka, MSR

Things like slower sales at concession stands, or issues with parking-lot directions, Guido said, wouldn’t be known as they were happening, something he wanted to change.

“I really wanted to be able to act on it [the operations data] in real time, instead of waiting until the Wednesday after a Sunday game,” Guido said.

Now, with Executive Huddle, the Niners’ operations team can sit in a single room and watch as operations events take place, and can make in-game moves to fix things, like calling on the radio to a parking lot to tell gate operators of traffic issues.

“It’s like having an air traffic control system,” said Mark Lehew, global vice president for sports and entertainment industry solutions at SAP. Lehew said SAP worked with the Niners’ list of operations vendors, including Ticketmaster, ParkHub, caterer Levy and point-of-sale technology provider Micros to provide back-end application links so that Executive Huddle could draw information from each separate system into the uber-operations view that Executive Huddle provides. According to SAP, Executive Huddle is based on SAP’s Leonardo and Analytics platform.

The manager of managers

Though the system doesn’t currently monitor some other key stadium operations information, like performance of the Levi’s Stadium Wi-Fi network, Michael Pytel, chief innovation officer for Nimbl, said the system could conceivably add “any information we can get from an API.”

The Levi’s Stadium suite where the Niners monitor Executive Huddle information. Credit: San Francisco 49ers

Moon Javaid, the Niners’ vice president of strategy and analytics, said the continued robust performance of the stadium’s wireless networks make them a lower-priority need for the kind of oversight Executive Huddle provides.

Javaid, the quarterback of the program’s development from the Niners’ side of the equation, noted that part of its power comes not just from surfacing the data, but also from providing some instant intuitive markers — like red for declining metrics and green for positive — and the ability to compare current data to those from other events so that data could not just be seen but also understood, within seconds.

And while SAP plans to make Executive Huddle available to other teams, it’s clear that the program — as well as education and training for the decision-making staff who will use it — will need different care and feeding for each stadium that might want to use it. But SAP’s Lehew noted that being able to provide real-time data in an exposed fashion was becoming table stakes for operations providers, who would have to move past old ways of doing things if they wanted to be a part of the next generation of stadium service providers.

Massive MIMO is Sprint’s path to 5G, says CTO Saw

Dr. John Saw, CTO of Sprint, at an IEEE keynote speech. Credit all photos: Sprint

Sprint chief technical officer John Saw has seen the future of cellular wireless, and according to him it was at a sports event.

“I was at the [Winter] Olympics where KT [Korea Telecom] and Intel set up the first 5G network,” said Saw in a recent phone interview. “Stadiums will be a good showplace for the capabilities of 5G. It’s pretty impressive what you can do with 5G that you can’t do today.”

Saw, who was CTO at WiMAX play Clearwire before that company became part of Sprint, will be the first to admit that the network built for the PyeongChang Olympics wasn’t “true” 5G, but said it was a good precursor. He also added that it wasn’t a cost-conscious deployment, something MSR had heard from other sources who said Intel and KT didn’t hold back when it came to spending.

“They spent a lot of money [on the network],” Saw said.

But some of the services the Olympic network was able to support included local viewing of replays using Intel’s True View technology, which gives fans the ability to watch a play or action from a 360-degree angle. While Intel has had limited deployments of the technology at some U.S. sporting events, for the Olympics Saw said they used hundreds of cameras linked over millimeter wave frequencies, which can offer very low latency.

“They needed [to have the images] in real time,” Saw said, and built the millimeter wave network to do just that. While the network “wasn’t fully compliant to the subsequent 5G standards, a lot of what they built is the forerunner to 5G,” Saw said. “It was a pretty cool showcase, and will certainly find a home in stadiums.”

No Millimeter Wave spectrum for Sprint

Editor’s note: This profile is from our latest STADIUM TECH REPORT, an in-depth look at successful deployments of stadium technology. Included with this report is a profile of a new MatSing ball DAS deployment at Amalie Arena, a new DAS for the Chicago Cubs at Wrigley Field, and a look at the networks inside the new Banc of California Stadium in Los Angeles! DOWNLOAD YOUR FREE COPY now!

Millimeter wave networks, however, won’t be part of Sprint’s early push toward 5G, said Saw. Instead, he said Sprint will concentrate on deploying “Massive MIMO” networks in its rich space of spectrum at the 2.5 GHz frequency, where Sprint controls upwards of 150 MHz of spectrum in most major U.S. metro markets.

Without trying too hard here to explain exactly how Massive MIMO works — think splitting up transmissions between mulitple antennas then using lots of compute power to bring the data back together — the key here is Sprint’s spectrum holdings, which Saw said are still only about half used.

“When we launched LTE [on the 2.5 Ghz spectrum] we used less than half the spectrum we had,” Saw said. “With 5G, we will use all the spectrum we have in market. We’ll be one of very few carriers who launch 5G in the same [spectrum] footprint [as LTE].”

With the ability to carry “four to 10 times the capacity of regular LTE,” Saw sees Massive MIMO 5G as something perfect for large public venues like stadiums and shopping malls.

Dr. John Saw

“When you have sports events with 50,000 people in the stadium, you need this kind of capacity,” Saw said. “Were building out the footprint for [5G] this year, and we’ll launch next year.”

Saw said that part of the infrastructure support for 5G networks will be different as well.

“It’s more than just speed, or more capacity. It’s more than tonnage,” Saw said. “We’ll have a different way of deploying the new network, with a more distributed core, one [with more resources] out to the edge of the network.”

Why is such equipment redistribution necessary? According to Saw, a network with more components at the edge can help with content delivery for the new bandwidth-hungry apps like virtual-reality replays.

“Say you want VR at a hockey game, where you want to give real time [replay] viewing to customers, with different camera angles,” Saw said. “You literally have to have the 5G core inside the stadiums, so it can process [the content] without having to go back to the cloud.”

Will DAS trail in the path to 5G?

One type of network Saw doesn’t see leading the way to 5G is the traditional DAS, or distributed antenna system.

“DAS is going to have to migrate to 5G,” Saw said. “It’s not going to lead the pack.”

In fact, Saw said Sprint has been somewhat of a reluctant DAS participant at times, including at the most recent Super Bowls. In the last two of the NFL’s “big game” events, Super Bowl 51 in Houston and Super Bowl 52 in Minneapolis, Saw said Sprint used small cell deployments instead of the neutral DAS systems to augment its coverage.

“We had hundreds of small cells, inside and outside [the venues],” Saw said. “We got the same performance, maybe better, for a lot less money.”

Part of the issue for Sprint and DAS, Saw said, is that the carrier usually has to pay more for its unique spectrum bands, especially the 2.5 GHz frequencies which are not used by any of the other major wireless carriers.

“We always think through before we sign up for DAS fees… there’s more than one way to skin a cat,” Saw said. While in many cases there is no alternative except to participate in a neutral-host configuration, Saw said “we do prefer small cells.”

Will CBRS help?

One of the more hyped platforms being pushed this year is use of the CBRS spectrum at the 3.5 GHz range for not just more carrier networks, but even for “private” LTE networks, like for venues or campuses.

“It’s an interesting concept because it opens things up to more than just four operators,” Saw said. But he also called out the need for an online database to make sure CBRS spectrum use doesn’t interfere with systems run by the U.S. Navy, and added that without any definitive FCC action yet, the rules for future CBRS use are still unclear.

“There’s quite a lot of work to be done, and not a lot of spectrum there,” said Saw. While claiming that Sprint is “watching CBRS with interest,” he added that with its 2.5 GHz holdings, Sprint most likely won’t be at the front of any CBRS deployments.

“At the end of the day, CBRS is not 5G,” Saw said.

How will a merger with T-Mobile help?

Since our conversation took place just a day after Sprint and T-Mobile announced their renewed plans to merge, Saw didn’t have a lot of details to share, beyond his opinion that the two companies’ different spectrum holdings would build a more powerful competitor when put together.

“When you put our 2.5 (GHz) with their 600 MHz it gives you a much larger footprint with higer capacity,” Saw said. “There’s tremendous synergy. Both [companies] are enthusiastic about this deal.”

Editor’s note: This post is part of Mobile Sports Report’s new Voices of the Industry feature, in which industry representatives submit articles, commentary or other information to share with the greater stadium technology marketplace. These are NOT paid advertisements, or infomercials. See our explanation of the feature to understand how it works.

Eagles sign Appetize for new point-of-sale system at Lincoln Financial Field

Self-serve kiosks from Appetize allow fans to order and pay for their own food for nearby pickup. Credit all photos: Appetize

The Philadelphia Eagles have signed a deal with Appetize to bring its technology-centric point of sale system into Lincoln Financial Field, a deal designed in part to help speed up concessions transactions for home fans of the new Super Bowl champions.

According to a press release out today, Appetize will install “more than 500” iOS- and Android-based terminals inside the Linc, including some touch-screen fan-facing checkout displays as well as self-service concession kiosks that are meant to function much like the terminals found at airports for checking in to flights.

Kevin Anderson, co-founder and chief strategy officer for Appetize, said in a phone interview that internal company tests have shown that the self-service kiosks can speed up a concessions transaction by as much as 20 percent, good news for fans who are tired of spending lost minutes standing in line waiting for a cheesesteak. For teams and venue owners, the 10-inch screens being installed at other, regular concession stands in the Linc can help with upsell, as Anderson said that the screen space allows the operator to program in add-on options (like adding a drink or fries to a sandwich order) via a side-of-screen advertisement that makes it easy to add to the order with a click.

In addition to the new customer-facing technology, Appetize is also gaining entree to venues for its cloud-based back-end systems, which Anderson said cuts out the need for teams to have localized infrastructure to buy and manage. Though he won’t name them all yet, in addition to the Eagles win Anderson said Appetize has claimed three other NFL contracts that were out for bid this summer, perhaps proof that the company’s mantra of having “enterprise and modern” facets in their systems is finding receptive ears.

Making sure the infrastructure is set up for kiosks

While MSR clearly needs to schedule a stadium visit sometime to check out kiosk wait times compared to older concessions systems, Anderson did note that teams can’t just plug the kiosks in and expect them to work with an existing infrastructure. “There is a shift in operations” that is necessary, he said, since kiosks can double or triple the number of orders in a given time to an existing kitchen location. However, having kiosks also means that self-service stands can be staffed with workers who simply put orders together, instead of having to train those workers on payment systems and devices.

New tablet-based POS terminals can entice fans into add-on purchases

One area where Appetize doesn’t see a lot of explosive growth is on the in-seat delivery end, a trend that seems to slowing down and finding its way mostly into premium seating areas at most venues. While Appetize can support mobile-device ordering and delivery (it even started its corporate life with an end-user focus on a mobile/delivery app) Anderson said the infrastructure and human engineering necessary to support a full-stadium delivery scheme is usually found to be unworkable. The San Francisco 49ers, who opened Levi’s Stadium in 2014 with mobile-app delivery of concessions to every seat, scrapped that service last season.

“We’re definitely not seeing [customers] asking us to do full-stadium” in-seat delivery, Anderson said. However, having the ability to place an order via a mobile device does have value in premium seating areas, where stadiums may already have systems like the Appetize-based one currently used at Lincoln Financial Field, where servers with wireless devices roam the seating areas offering in-seat ordering as a white-glove service.

“It’s a nice line-item for the season ticket sales sheet” to offer in-seat delivery services in places where it makes sense, Anderson said. “Venues are being smart [now] about where they are putting it.”

State of the Stadium Network, 2018: Smooth sailing right now but rough waters ahead?

Here at Mobile Sports Report we used to have a yearly survey (called “State of the Stadium”) which we used mainly to see if and when wireless networks were being deployed in large sports venues. After just a few years, it quickly became apparent that for almost all the respondents we heard from, the question was no longer “if” networks would be deployed, but just “when.” And for more than most, the “when” was happening already.

Looking back over the past year or so of our stadium profile visits, it’s clear that the still-young market of large-venue wireless connectivity has reached a certain level of maturity, especially when it comes to well-funded deployments of Wi-Fi and cellular distributed antenna system (DAS) networks. Where in the recent past the San Francisco 49ers’ Levi’s Stadium was a groundbreaker with its extensive wireless coverage when it opened in 2014, such networks have now become the standard expectation for new venues like the Sacramento Kings’ Golden 1 Center, U.S. Bank Stadium in Minneapolis, Mercedes-Benz Stadium in Atlanta, T-Mobile Arena in Las Vegas and even in many “Tier 2” stadiums like Colorado State University’s new football stadium.

Similar high-quality networks are also finding their way into older stadiums as those venues get networking for the first time or revamp their initial outlays. Over the past couple years we’ve seen new networks appear in old venues like Notre Dame Stadium, SAP Center in San Jose and more recently, the Alamodome. Other venues that led the initial charge toward wireless networks for fans, like the New England Patriots’ Gillette Stadium, the Bank of America Stadium in Charlotte and Lincoln Financial Field in Philadelphia, all had recent upgrades to their wireless infrastructures as the venues smartly stayed in tune with the ever-increasing demands of fans and their mobile devices. And then there are pioneers like AT&T Park and AT&T Stadium, which have always managed to lead the way in finding new ways to keep their connectivity at state of the art levels.

What really helps point to a certain level of maturity is the different methods and manufacturers who all have figured out their own ways to get things done. Wi-Fi antenna deployments placed under seats, in railing mounts or overhead have all proven themselves in numerous live tests; DAS deployments have shown similar successes in a somewhat corresponding number of techniques and equipment usages; in all, there seems to be well more than one path to a successful wireless infrastructure. But before we start taking networking for granted as a commodity like electricity or plumbing, it’s a good time to remember that unlike those two services, networking doesn’t stand still. As new end-user devices and the apps they run continue to drive growth in demand, the question now is whether current Wi-Fi and DAS networks for venues will be able to keep up, or whether new technology is needed.

The need for more wireless spectrum

Editor’s note: This profile is an excerpt from our latest STADIUM TECH REPORT issue for Spring 2018, which includes a look at Wi-Fi performance during the Final Four, a recap of wireless performance at Super Bowl 52, a profile of new venue construction in Los Angeles and more! DOWNLOAD YOUR FREE COPY right now from our site!

In a previous lifetime as a cellular systems analyst, yours truly wrote a long research paper about the importance of spectrum, predicting that at some point the leading wireless carriers, namely AT&T and Verizon Wireless, were going to need new bands to expand their services. While there have been some technological tweaks to find more capacity than originally thought in the 4G LTE space, on the cellular front the march to so-called “5G” systems is well underway, with the predictable problem of marketing promises being far out ahead of usable reality.

While we’ll save an in-depth look at 5G for another point in time, it’s useful to notice that all the large wireless carriers are already making 5G announcements, of 5G trials, of 5G local networks and other assorted claims of leadership. While nobody really knows exactly what 5G is for sure, what is known is that to get to the faster/better claims being staked there is going to be new spectrum in play for 5G services, and some of it may work better than others for use inside venues.

What’s clearly not known at all is how 5G services will arrive for sports stadiums, as in whether or not they will fit inside the current DAS model. Will carriers be able to share 5G systems like they do now on neutral-host DAS deployments? Right now that’s doubtful given that carriers like Sprint and T-Mobile are already talking about 5G deployments on much different spectrum spaces — and if the proposed merger between the two carriers becomes reality, how does that further change the 5G planning landscape? Perhaps the only thing we can be sure of is a lot of mixed messages in the near future about the best way to move forward from a cellular perspective.

Will carriers take over unlicensed bands?

On the Wi-Fi side of things, a smart friend of ours once claimed that when it came to Wi-Fi network deployments, “real estate is the new spectrum” since building owners could pretty much stake a free claim to the unlicensed spectrum spaces within their walls.

But now, there may be some storm clouds brewing as carriers seek to implement systems that let them use some of the 5 GHz unlicensed channels for LTE networks, an idea with possible consequences for current venue networks.

Aruba’s Chuck Lukaszewski wrote about this issue for Mobile Sports Report last summer, and some of his points bear repeating and remembering, especially these two: One, most Wi-Fi networks in large stadiums are already “spectrum constrained,” meaning that they need all the channels in the unlicensed band to ensure good service across an entire venue; Two, by introducing a system where cellular providers would use a chunk of that spectrum for LTE networks, the effects are as yet unknown — and venue operators would most likely be at the mercy of carriers to both acknowledge and comply with any possible conflicts that might arise.

As we here at Mobile Sports Report are cynics of the first order, our first question in this matter is about whether or not there are any clauses in those contracts venues have signed with carriers that will allow the cellular providers to “share” spectrum in the Wi-Fi space as well. While Verizon, AT&T and other service providers have paid quite a few dollars to support many stadium systems, it’s worth it to wonder if some of those deals may not look so good going forward if they include the legal ability for carriers to poach spectrum currently used only by Wi-Fi.

CBRS to the rescue?

Another technology/spectrum space we’ll be looking at more closely in the near future is the Citizens Broadband Radio Service, which sits at the 3.5 GHz space in the electromagnetic spectrum roster. Though new FCC rules on the use of this spectrum (currently used primarily by the U.S. Navy) haven’t yet been solidified, it seems from all signals that eventually what will emerge is a kind of tiered licensing type of situation with licenses that cover large, small or even local geographic areas, which may allow for building owners to set up private networks that work sort of like Wi-Fi does now.

One attractive option being touted is “private” LTE networks, where venue or building owners could build their own DAS-like LTE network infrastructure for CBRS spectrum, then rent out space to carriers or run their own networks like Wi-Fi but with LTE technology instead.

What’s unknown is exactly how the licensing scheme will shake out and whether or not big carriers will be able to dominate the space; here it’s helpful to remember that big wireless carriers typically spend millions in lobbying fees to influence decisions in places like the FCC, and venue owners spend… nothing. Verizon recently announced it expects to have CBRS-ready devices working before the end of this calendar year, so it’s likely that CBRS systems may be more of an immediate concern (or opportunity) for venues than 5G. And the marketing folks behind CBRS are on full speed ahead hype mode, even crafting a marketing name called “OnGo” as an easier-to-sell label than the geeky “CBRS.” So buyer beware.

Already, Mobile Sports Report has heard chatter from folks who are helping design networks for greenfield operations that the choices simply aren’t as clear as they were recently, when you could pretty much count on Wi-Fi and DAS to meet whatever wireless needs there were. While that duo may still be able to get the job done for the near future, looking farther ahead the direction is much less clear and the sailing no doubt much less smooth. Here at MSR, we’ll do our best to help batten the hatches and give as much clear guidance as we can. At the very least, it should be an interesting trip.