AT&T Super Bowl Sunday stadium wireless traffic jumps 60 percent to 624 GB; Verizon claims 1.9 TB

Inside the AT&T head-end building at MetLife. Cables! Credit: AT&T

Inside the AT&T head-end building at MetLife. Cables! Credit: AT&T

Well, our prediction that this year’s Super Bowl might not set new wireless data records was about as solid as the Denver Broncos’ performance in the big game. According to AT&T, its wireless network in and around MetLife Stadium on Super Sunday saw 624 GB of traffic, a 60.8 percent increase from last year’s 388 GB total at the Superdome in New Orleans.

Part of the gain no doubt came from the fact that MetLife Stadium holds approximately 10,000 more fans during a football game than the Superdome, 82,566 to 72,003. But AT&T’s numerical evidence — including a new peak-time high of 119 GB during the hour before kickoff — shows that the desire to access mobile devices inside stadiums is still growing, especially at big games like the Super Bowl.

Verizon Wireless, which like AT&T spent more than a year getting MetLife’s network ready for Sunday’s game, also said it had experienced record traffic on its networks, claiming a total of 1.9 TB of cellular traffic inside MetLife Stadium. During the game Verizon Twitter accounts said new traffic records were set before halftime; however we still have no figures from Verizon for previous years’ data for comparison.

Verizon also claimed its peak hour usage was “800 percent” greater than last year’s, but again, we don’t have exact numbers so can’t confirm the accuracy of such claims. Still unkown is how much traffic was carried by the stadium Wi-Fi network which Verizon manages (and was free to all MetLife fans), since Verizon did not provide Wi-Fi traffic numbers. Our guess is it saw similar traffic leaps as AT&T’s and Verizon’s cellular nets.

Another interesting note from the AT&T data is a trend toward more mobile data use, and fewer voice calls. Fans at the game Sunday made 55,000 voice calls on AT&T networks, down from 73,000 calls in 2012. Some of that might have to do with the fact that last year’s game had a lengthy power outage, which no doubt prompted many “I’m OK” calls to loved ones. But still — a 20,000 call dropoff in the face of more than doubling data traffic seems to show that people are using their phones more, even if they are talking less.

Why the NFL is blocking streaming at the Super Bowl: Blame the network, not the fans

In case you are wondering why you won’t be able to watch the Super Bowl live on your phone while you’re at the game, Jon Brodkin at Ars Technica has a good story about why the NFL is blocking streaming video inside MetLife Stadium. To quickly recap, NFL CIO Michelle McKenna-Doyle told Brodkin that streaming video takes up too much bandwidth, and that it could hamper overall wireless communications on game day, so the league is proactively blocking live feeds at the game.

While I agree with her assessment of the situation and the solution (blocking live video), I don’t agree with her claim that the “vast majority of our fans want to watch the game on the field, watch the replays on the jumbo board, and participate in the event more than they want to be checking their phone,” and I’m surprised that Brodkin didn’t put up more of a challenge to this claim. Her follow-on claim that the league is doing the “vast majority” a favor by blocking the few video viewers rings hollow and reminds me of the old “data hogs” arguments the carriers used to use against people who were exercising their rights to their unlimited data contracts. My point: don’t blame fans who want to watch live video as being the people ruining the network for everyone else. Put the blame where it deserves to be, namely on the in-stadium networks that can’t yet handle the demands of a large crowd that wants video at the game.

We’ve talked before about why people want to stay connected while at the game. It’s not for everyone, but the desire to be online in your stadium seat is way more widespread than just a few people. Trolls will comment and say “watch the damn game and shut up” but plenty of real sports fans want the replays and closeups they are now accustomed to on TV. And not every seat has a good view of the big screens inside the stadium, and many times those things are showing ads, not replays. Then there is the time standing in line for a beer or bathroom. Why shouldn’t you be able to watch the game you are paying big bucks to be at, instead of being penalized because the stadium doesn’t have enough beer vendors or urinals? How about watching a replay while the game is in one of its lengthy TV timeouts? Or catching up on a play that you missed during halftime? Isn’t just having to listen to Bruno Mars punishment enough?

It will be interesting to see what the user statistics are like when the San Francisco 49ers’ new Levi’s Stadium opens this year, if its much-touted network delivers as hoped. The cynic in me is also guessing that when the NFL finally gets its digital video strategy figured out — meaning they clear the rights contracts and find a way to start charging fans more to watch more video online — the stadium network problems will suddenly be solved, and you’ll be able to watch all the live video you can afford.

We’ll be the first to acknowledge that putting networks in stadiums isn’t easy. Our recent Stadium Tech Reports series is designed to profile those in the industry who are trying to bring a quality wireless experience to their fans, so that others may follow. Already, we see places like Barclays Center and Gillette Stadium pushing the envelope when it comes to features like streaming video. In many ways, getting there is a long road that we’re just at the start of. If there is one bit of analysis I can provide after covering this field for the past 3 years, it’s that I don’t think anyone has gotten the stadium-network thing completely figured out just yet — and that any network put in over the past couple years is probably already in need of an upgrade, due to user demands already exceeding capacity. And that’s before most places are even thinking of providing live video feeds.

So sure, go ahead and block live video if it’s going to crash the network. But stop saying it’s something that just a few fans want, because there’s no proof behind that idea. Until the league and carriers like Verizon offer up real data on stadium network usage, there’s no way of telling exactly how many people at a game want to watch video, and whether it’s just for a replay or two or if they want a constant stream going at their seat. I’d be willing to bet more than a pint with McKenna-Doyle that if she polled an average NFL crowd and asked them if they’d like replays at their seats, a “vast majority” would vote for replays on their phones, and not in favor of settling for jumbotrons and PA announcers as she claims. So again, if you need to block the video, fine, but put the blame for the action on the network’s lack of capacity, and not on the fans who are just trying to enhance their own experience.

Stadium Tech Report: Verizon, AT&T DAS upgrades at MetLife Stadium await Super Bowl Sunday

Verizon branded gate at MetLife Stadium. Credit: Verizon Wireless

Verizon branded gate at MetLife Stadium. Credit: Verizon Wireless

Super Bowl foes the Seattle Seahawks and the Denver Broncos have only had two weeks to prepare for each other. But Verizon Wireless and AT&T have been preparing for the roman-numeral date at MetLife Stadium for more than a year. Will the Verizon Wi-Fi and DAS, and AT&T’s separate DAS be able to handle the wireless needs of the fans at the NFL’s biggest game? Tune in Sunday to see!

We might be one of the only news outlets who care more about the wireless networking at Super Bowls than the game itself, but for many in the stadium tech industry the biggest single game in America’s most popular sport is always somewhat of a wireless watershed. Perhaps at no other event do attendees spend so much time shooting selfies and posting them as they do from the site of Super Bowl Sunday. Even in the expected cold, it should be no different this week at Super Bowl XLVIII at MetLife Stadium in New Jersey, where game time is scheduled for around 6:30 p.m. Eastern.

But well before that, fans will be testing the Verizon-built Wi-Fi network and both of the big carriers’ DAS deployments inside MetLife. According to reps from both companies that we spoke with last week, the carriers are ready.

Verizon spokesman David Samberg told us via email last week that an upgrade of the Verizon DAS in MetLife this past season means there are now more than 500 DAS antennas inside the facility. See some of the photos provided by Verizon that show the clever hiding spots Verizon engineers have found over the last 18 months as they’ve added capacity to a network built just a few years ago.

AT&T has also put in a brand-new DAS over the past year, with another 500-plus antennas of its own. “For the last year or so we’ve been working on our pre-game and game day network playbook in an effort to provide the best possible wireless experience for our customers,” said Michael Maus, assistant vice president of network services at AT&T, via email. “In anticipation of the huge volume of data and voice usage expected [for the Super Bowl], we’ve built a new state of the art antenna system inside the stadium, we’re rolling in portable cell sites both at the stadium, and to support the tailgate areas, and we’ve augmented coverage in New York City to support the activities there.”

Rooftop DAS equipment. Credit: Verizon Wireless

Rooftop DAS equipment. Credit: Verizon Wireless

For Verizon’s own customers, 4G LTE capacity at MetLife has been quadrupled since last year, according to Samberg, who said that all stadium upgrades were completed by October, giving Verizon multiple chances to test its system against crowds at New York Giants and New York Jets home games. So far, Samberg said, so good.

More traffic than last year’s Super Bowl already

While yours truly opined earlier this year that this year’s Super Bowl might not set a wireless traffic record, we didn’t take into account some simple numbers — mainly, that MetLife Stadium’s official capacity of 82,566 is bigger than the Superdome’s 72,003. So, even if it’s cold, having 10,000 more people on hand probably means more bandwidth consumed, even if this year’s game doesn’t have a power blackout in the second half. (And even if it does, Samberg said the network shouldn’t go down since Verizon has backup power supplies on hand.)

Find the DAS antenna! Credit: Verizon Wireless

Find the DAS antenna! Credit: Verizon Wireless

Our only problem with record wireless numbers from Sunday’s game is that we probably won’t ever see an actual number, since Verizon historically shies away from providing a score. Instead it just issues press releases saying things like “way more traffic this year than last!” and then expects us all to believe that without numbers. The good news for fans at the game is that the in-stadium Wi-Fi network, also built by Verizon, is free and open to customers of all carriers, or basically anyone with a device that has a Wi-Fi chip. But Verizon, like big competitor AT&T, has been beefing up its DAS installations significantly because most people try cellular first, even at stadiums, before instructing their phones to find a Wi-Fi network. AT&T, to its credit, usually does deliver a wireless scorecard quickly after big events. So at least from AT&T’s perspective we should find out if this year’s game sets another record.

Aside from the stadium improvements, Verizon will be showing a demonstration of a technology this week that could make DAS more of a competitor to Wi-Fi on the high bandwidth side of things. Called LTE multicast, the technology basically establishes set channels for LTE devices that will “broadcast” video, like a TV channel. (This idea is similar to the StadiumVision Mobile technology Cisco uses at stadiums like Barclays Center.) Theoretically, LTE multicast could let fans use a cellular connection to view multiple video streams, something you would need to use Wi-Fi for it to have any chance of working. But the multicast demo won’t take place at MetLife, but instead at Bryant Park in Manhattan this week. If you are in the city, check out the demo and let us know what you think.

No NFL Mobile at MetLife

And here’s something else you won’t be able to use at MetLife during the Super Bowl: Verizon’s own NFL Mobile app, which outside the stadium will be the only smartphone platform you’ll be able to watch the game on. (The Fox streaming site and app will only work with tablets and desktops or laptops, per the league’s rights agreement with Verizon.) Next year, the rights for NFL Mobile will change and if the local game (like, say, the Super Bowl) is on TV, you’ll be able to use NFL Mobile to watch it even if you’re at the stadium. But not this year! (To give one answer as to why, if you are at the game, you might want to watch it on your cell phone, we say: Bathroom or beer lines!)

More stadium infrastructure photos below:

AT&T DAS antennas at MetLife. Credit: AT&T

AT&T DAS antennas at MetLife. Credit: AT&T

AT&T's new head-end building at MetLife, where its DAS gear is housed. Credit: AT&T

AT&T’s new head-end building at MetLife, where its DAS gear is housed. Credit: AT&T

Inside the AT&T head-end building at MetLife. Cables! Credit: AT&T

Inside the AT&T head-end building at MetLife. Cables! Credit: AT&T

Stadium Tech Report: Wi-Fi, DAS and live video get good reception at Barclays Center

Concessions feature of Barclays Stadium app. Credit: Barclays Center

Concessions feature of Barclays Stadium app. Credit: Barclays Center

Sometimes, the best surprise is no surprise. That’s the case when it comes to technology deployments at the still-new Barclays Center in Brooklyn, where the Wi-Fi, DAS and live video on both fixed and mobile platforms are all performing pretty much as expected.

According to Chip Foley, vice president of building technology for Forest City Ratner Companies (the developer of Barclays Center), perhaps the only mild surprise so far at the just-over-a-year-old Barclays is that the biggest Wi-Fi usage came not during a sporting event, but instead at the MTV Video Music Awards ceremony this past August.

“We had 7,000 people using the Wi-Fi network at the VMAs, and I was a little surprised at that,” said Foley. At Brooklyn Nets games, Foley said, the average Wi-Fi load in the 17,500-seat arena is somewhere between 4,000 and 5,000 users per game. In a recent phone interview, Foley recapped the performance of the stadium’s cutting-edge technology, which also includes one of the first deployments of Cisco’s StadiumVision Mobile, which brings live video feeds to fans using the stadium app. There’s also Cisco-powered digital displays throughout the arena, and a robust DAS deployment to make sure regular cellular connections don’t fail.

HD Wi-Fi attracts 20 percent of attendees

Barclays Center, which opened in September of 2012, had the benefit that few NBA stadiums have in that it was built from the ground up with networking as a key component. If Foley has any regrets about the Cisco Connected Stadium Wi-Fi deployment, it’s that it hasn’t really been fully tested yet. Even during the VMAs, Foley said he was using the in-building Wi-Fi to watch 10 different streaming video views on his laptop, from the red carpet cameras to the behind-the-scenes views of stars getting their awards.

Chip Foley

Chip Foley

“Our goal was to build as robust a network as possible, so that we can handle big needs of one-off events [like the VMAs] as well as the 41+ Nets games every season,” Foley said. With two 1-gigabit backbone lines providing Internet access, Foley said the Barclays network is meeting its goal of being “as fast as your fiber connection at home.”

The only drawback so far seems to be getting more fans to try out the network connection when they are at the games or events. According to Foley, despite advertising and promotions, Nets crowds almost always hit a figure of between 20 percent and 25 percent of them being online, a “Groundhog Day” situation that has Foley wondering whether it’s a natural limit.

“That may just be the number of fans who want to use it [the network]” at a game, Foley said.

The Barclays Center DAS, deployed by ExteNet Systems using gear in part from TE Connectivity, is another non-surprise center for Foley.

“The DAS is great, we never get complaints [about cellular connectivity],” Foley said. “You dread hearing that people can’t send texts. That just hasn’t happened.”

Digital displays, both mobile and fixed

One of the more compelling features of the Barclays tech experience is the implementation of Cisco’s StadiumVision Mobile technology, which brings several live “channels” of video to any fan using the Wi-Fi connection and the stadium app, which was built by WillowTree. With views from the benches, behind the basket and quick replays, Barclays can bring an up-close and personal view to even those far away from the court.

StadiumVision Mobile app being used in Barclays Center. Credit: Barclays Center

StadiumVision Mobile app being used in Barclays Center. Credit: Barclays Center

“StadiumVision Mobile is great for the upper pavilion seats, you can now get a view from a different perspective, and get replays,” Foley said. According to Foley, Cisco engineers tested the technology’s performance to ensure that it worked at every seat in the house.

Fixed digital displays are also a key technology at Barclays, starting with the unique Oculus display built into the striking exterior of the building, and continuing to the hundreds of digital displays inside. Using the Cisco Stadium Vision digital display technology, Barclays Center is able to change and update information on single screens or on all screens on the fly, allowing for greater flexibility in terms of messaging and information like concession-stand prices. Barclays also uses its displays to show train schedules, giving fans better information to plan their departures from events.

“The Stadium Vision displays have been nothing but great for us — we sold a lot of advertising on them even before launch,” Foley said. “It’s fun for our content group to build out content for the L-boards [displays where an L-shaped advertisement brackets other information on the screen], and keep it changing. Restaurant operators can use an iPad to change prices [on their screens] right before an event. They don’t have to talk to us. Overall, it’s a lot less maintenance than I expected or anticipated.”

If Foley had one chance to do anything over again with displays, it would be to add more of them to the original mix. His lesson to future stadium display builders is: If you’re in doubt, put up more.

“We must have had 30-plus meetings regarding [internal] TV locations, with 3D modeling and fly-throughs,” Foley said. “For the most part, I’m happy. But if I could, I would have more clusters [of screens]. Wherever there is one screen now, I wish I had three. People always look at a cluster.”

Adding new screens after the fact, Foley said, isn’t as simple as going to Best Buy to pick up a discount TV.

“You might be able to buy a TV for a couple hundred bucks on Black Friday, but no one tells you that to put that in a venue, once you get past union costs, connectivity and everything else, it’s about $5,000,” Foley said. “It’s way more money to add them now.”

What’s next: iBeacon, Google Glass and more analytics

What’s in the future for Barclays technology? For starters, Foley will oversee deployment of Wi-Fi services for the outside spaces surrounding the arena.

“It’d be nice to have Wi-Fi for ticket scanning outside the venue,” said Foley. “That’s one of those things that you don’t understand the need for it until you open the stadium and see what happens.”

Barclays is also looking into testing the Apple iBeacon technology, which can send text messages to devices in very close proximity. Technologies like iBeacon and even digital signage must also cross internal administrative hurdles, such as simply training sales forces and alerting advertisers to the opportunities.

“For some of the streams, there’s the question of ‘how do we sell this’ — the team has never done this and sponsors may not be aware,” Foley said. “You also have to figure out things like how many notifications and emails should we send out. You don’t want to send out too many, because that turns people off.”

Foley said the Barclays social media team is also at the start of a process of mining statistics from places like Twitter, Facebook and other social media streams, to get a better handle on what fans are using the technology for and how the experience might be improved. One possible way is through a Google Glass application, something Foley agreed might not be for everyone.

“I’m fascinated by the possibility of something like an XML stats feed [in Google Glass] where you’d still be able to watch the game,” Foley said. “We’re getting closer! It’s not for everybody, but some portion of the population is probably thinking that way.”

Stadium Tech Report: DAS, Wi-Fi puts end to no-signal problem at Denver’s Sports Authority Field

PeytonThese days, Denver’s Sports Authority Field at Mile High is the new home of the NFL’s most prolific signal-caller. With a record season for passing yards and passing touchdowns, Broncos quarterback Peyton Manning is recognizable for his animated pointing, shouting and line-of-scrimmage audibles, the ultimate practicioner of last-second communication.

Not too long ago, the fans at Mile High might have had to resort to the same tactics to communicate, using hand waving or shouting, since getting a cell signal was next to impossible. “Forget making a phone call, you couldn’t even send a text,” said Rick Seifert, communications manager for the Broncos’ stadium management company. “And it wasn’t just the fans. We [the staff] couldn’t make calls in the stadium to do our jobs.”

But in 2012, the Broncos changed all that with the installation of a full-featured distributed antenna system (DAS) deployed by TE Connectivity, and a fan-facing Wi-Fi network installed by Verizon Wireless.

Russ Trainor

Russ Trainor

The Broncos also put in a huge new digital scoreboard and robust back-end connectivity provided by Comcast as part of their blitz of networking improvements, and this past fall, AT&T joined in by upgrading its connection to the stadium’s DAS. By next year the Broncos hope to add AT&T and Sprint to its roster of Wi-Fi service providers, reflecting what vice president of information technology Russ Trainor sees as a “never ending growth” of wireless in-stadium consumption.

All carriers on board, slowly

One of the biggest problems with DAS deployments in stadiums is convincing major cellular carriers to work together. Since each carrier wants to deploy systems to do the best job for its customers, there is often a difference in opinion on strategy and operations, which is often followed by similar snags in contract negotiations. Trainor said that the stadium, built in 2001, presented unique RF challenges to wireless with its primarily exposed-steel construction. Verizon and Sprint were the first carriers to sign up for the neutral DAS, followed by AT&T this fall.

DAS equipment at Sports Authority Field. Credit: Denver Broncos

DAS equipment at Sports Authority Field. Credit: Denver Broncos

“It was tough to get them [all the major carriers] to agree on DAS, but we have good engineers on the back end and we came up with a nice solution for everybody,” said Trainor. While the antennas and stadium network are neutral, each carrier provides its own back-end gear, much of which at Mile High had to be placed in a building built outside the facility specifically to house telecom gear. In many stadium DAS deployments, the telecom gear can take up thousands of square feet, which can be challenging to find in facilities built before such needs were known.

“There’s no room inside for all the space they [the carriers] wanted,” Seifert said.

The Wi-Fi network, deployed by Verizon, uses Cisco equipment and is also a neutral host infrastructure, meaning that other carriers could use it to provide Wi-Fi connectivity to their clients if they so choose. According to Seifert, AT&T and Sprint will offer Wi-Fi services to customers next season, in part to answer the consistently growing demand. Like in other stadiums, fans at Sports Authority Field know what to do when they finally find bandwidth: Use more.

Steady growth in wireless use

When Sports Authority Field is at its listed capacity of 76,125 on game days, it becomes the 14th-largest city in Colorado, Trainor said. The team has already seen 525,000 downloads of its mobile application, which provides such in-stadium features as four different video replay angles, a connection to the NFL Network’s RedZone channel, and a direct link to the radio feed from hometown sports station KOA. The application is geo-fenced to ensure that the video rights are only used inside the stadium, and to give fans there a unique game-day experience.

Wi-Fi antennas on stadium overhang. Credit: Denver Broncos

Wi-Fi antennas on stadium overhang. Credit: Denver Broncos

According to Trainor, the team usually sees an average of 4,000 simultaneous connections on the Verizon Wi-Fi network on game days, though on colder days when fans need to wear gloves that number can drop in half. Trainor said the Cisco infrastructure is designed to accomodate 25,000 concurrent connections, a number the team hasn’t yet reached. However, the team did have to double the back-end capacity already for the Wi-Fi network, which is being used more as more fans find it.

“Word of mouth really gets [usage] going,” said Trainor, who noted that at a Kenny Chesney concert last year, the stadium crew saw data uploads outpace data downloads for the first time — a sure sign that fans in attendance were using the network to do things like share pictures and videos with their social-network connections.

“We haven’t seen any true bottlenecks yet, but usage is consistently rising, game after game, for concerts, soccer and football,” Trainor said.

Rick Seifert

Rick Seifert

A good sign from the Wi-Fi networking statistics is a shift in usage from the often crowded 2.4 GHz bands to the 5 GHz bands, which Trainor said is likely due to fans using the latest 5 series iPhones, which support the 5 GHz Wi-Fi frequency. And no matter what happens to the Broncos in the playoffs, Trainor and Seifert know what they will be doing this summer: Upgrading the network components, in the never-ending battle to provide bandwidth.

“Verizon and Sprint have already made significant upgrades to their DAS deployments because of demand and changes in technology, like LTE,” said Seifert. “And next year we’ll probably see AT&T circle back again. It’s very dynamic.”

“As smart phones get smarter it’s a never-ending challenge” to provide connectivity, Trainor said. “It’s a job that’s never finished.”

‘Big Four’ wireless carriers sign on to use DAS at Niners’ Levi’s Stadium

Levi's StadiumWhenever you are putting a neutral DAS in a stadium, one of the biggest challenges is convincing the major cellular carriers to participate in your deployment. Even before their new arena has opened, the San Francisco 49ers have scored a victory of sorts with the news that the four biggest U.S. cellular carriers — AT&T, Verizon, Sprint and T-Mobile — have all agreed to use the neutral-hosted DAS at Levi’s Stadium.

According to DAS Group Professionals, who are behind the DAS deployment at Levi’s, the system of small cellular antennas will also be synchronized with the stadium’s Wi-Fi coverage to produce “seamless” coverage for fans. Though we are historically wary of promises made before networks go live, the fact that the big players in U.S. cellular all agreed to use the neutral DAS shows at least in some way that the folks putting together the technology at Levi’s have a convincing story.

We’ll have more from DGP and the Levi’s DAS deployment at some point in the future, so stay tuned for the stadium launch that has everyone in the stadium tech marketplace watching and waiting.