Why the NFL is blocking streaming at the Super Bowl: Blame the network, not the fans

In case you are wondering why you won’t be able to watch the Super Bowl live on your phone while you’re at the game, Jon Brodkin at Ars Technica has a good story about why the NFL is blocking streaming video inside MetLife Stadium. To quickly recap, NFL CIO Michelle McKenna-Doyle told Brodkin that streaming video takes up too much bandwidth, and that it could hamper overall wireless communications on game day, so the league is proactively blocking live feeds at the game.

While I agree with her assessment of the situation and the solution (blocking live video), I don’t agree with her claim that the “vast majority of our fans want to watch the game on the field, watch the replays on the jumbo board, and participate in the event more than they want to be checking their phone,” and I’m surprised that Brodkin didn’t put up more of a challenge to this claim. Her follow-on claim that the league is doing the “vast majority” a favor by blocking the few video viewers rings hollow and reminds me of the old “data hogs” arguments the carriers used to use against people who were exercising their rights to their unlimited data contracts. My point: don’t blame fans who want to watch live video as being the people ruining the network for everyone else. Put the blame where it deserves to be, namely on the in-stadium networks that can’t yet handle the demands of a large crowd that wants video at the game.

We’ve talked before about why people want to stay connected while at the game. It’s not for everyone, but the desire to be online in your stadium seat is way more widespread than just a few people. Trolls will comment and say “watch the damn game and shut up” but plenty of real sports fans want the replays and closeups they are now accustomed to on TV. And not every seat has a good view of the big screens inside the stadium, and many times those things are showing ads, not replays. Then there is the time standing in line for a beer or bathroom. Why shouldn’t you be able to watch the game you are paying big bucks to be at, instead of being penalized because the stadium doesn’t have enough beer vendors or urinals? How about watching a replay while the game is in one of its lengthy TV timeouts? Or catching up on a play that you missed during halftime? Isn’t just having to listen to Bruno Mars punishment enough?

It will be interesting to see what the user statistics are like when the San Francisco 49ers’ new Levi’s Stadium opens this year, if its much-touted network delivers as hoped. The cynic in me is also guessing that when the NFL finally gets its digital video strategy figured out — meaning they clear the rights contracts and find a way to start charging fans more to watch more video online — the stadium network problems will suddenly be solved, and you’ll be able to watch all the live video you can afford.

We’ll be the first to acknowledge that putting networks in stadiums isn’t easy. Our recent Stadium Tech Reports series is designed to profile those in the industry who are trying to bring a quality wireless experience to their fans, so that others may follow. Already, we see places like Barclays Center and Gillette Stadium pushing the envelope when it comes to features like streaming video. In many ways, getting there is a long road that we’re just at the start of. If there is one bit of analysis I can provide after covering this field for the past 3 years, it’s that I don’t think anyone has gotten the stadium-network thing completely figured out just yet — and that any network put in over the past couple years is probably already in need of an upgrade, due to user demands already exceeding capacity. And that’s before most places are even thinking of providing live video feeds.

So sure, go ahead and block live video if it’s going to crash the network. But stop saying it’s something that just a few fans want, because there’s no proof behind that idea. Until the league and carriers like Verizon offer up real data on stadium network usage, there’s no way of telling exactly how many people at a game want to watch video, and whether it’s just for a replay or two or if they want a constant stream going at their seat. I’d be willing to bet more than a pint with McKenna-Doyle that if she polled an average NFL crowd and asked them if they’d like replays at their seats, a “vast majority” would vote for replays on their phones, and not in favor of settling for jumbotrons and PA announcers as she claims. So again, if you need to block the video, fine, but put the blame for the action on the network’s lack of capacity, and not on the fans who are just trying to enhance their own experience.

Stadium Tech Report: Verizon, AT&T DAS upgrades at MetLife Stadium await Super Bowl Sunday

Verizon branded gate at MetLife Stadium. Credit: Verizon Wireless

Verizon branded gate at MetLife Stadium. Credit: Verizon Wireless

Super Bowl foes the Seattle Seahawks and the Denver Broncos have only had two weeks to prepare for each other. But Verizon Wireless and AT&T have been preparing for the roman-numeral date at MetLife Stadium for more than a year. Will the Verizon Wi-Fi and DAS, and AT&T’s separate DAS be able to handle the wireless needs of the fans at the NFL’s biggest game? Tune in Sunday to see!

We might be one of the only news outlets who care more about the wireless networking at Super Bowls than the game itself, but for many in the stadium tech industry the biggest single game in America’s most popular sport is always somewhat of a wireless watershed. Perhaps at no other event do attendees spend so much time shooting selfies and posting them as they do from the site of Super Bowl Sunday. Even in the expected cold, it should be no different this week at Super Bowl XLVIII at MetLife Stadium in New Jersey, where game time is scheduled for around 6:30 p.m. Eastern.

But well before that, fans will be testing the Verizon-built Wi-Fi network and both of the big carriers’ DAS deployments inside MetLife. According to reps from both companies that we spoke with last week, the carriers are ready.

Verizon spokesman David Samberg told us via email last week that an upgrade of the Verizon DAS in MetLife this past season means there are now more than 500 DAS antennas inside the facility. See some of the photos provided by Verizon that show the clever hiding spots Verizon engineers have found over the last 18 months as they’ve added capacity to a network built just a few years ago.

AT&T has also put in a brand-new DAS over the past year, with another 500-plus antennas of its own. “For the last year or so we’ve been working on our pre-game and game day network playbook in an effort to provide the best possible wireless experience for our customers,” said Michael Maus, assistant vice president of network services at AT&T, via email. “In anticipation of the huge volume of data and voice usage expected [for the Super Bowl], we’ve built a new state of the art antenna system inside the stadium, we’re rolling in portable cell sites both at the stadium, and to support the tailgate areas, and we’ve augmented coverage in New York City to support the activities there.”

Rooftop DAS equipment. Credit: Verizon Wireless

Rooftop DAS equipment. Credit: Verizon Wireless

For Verizon’s own customers, 4G LTE capacity at MetLife has been quadrupled since last year, according to Samberg, who said that all stadium upgrades were completed by October, giving Verizon multiple chances to test its system against crowds at New York Giants and New York Jets home games. So far, Samberg said, so good.

More traffic than last year’s Super Bowl already

While yours truly opined earlier this year that this year’s Super Bowl might not set a wireless traffic record, we didn’t take into account some simple numbers — mainly, that MetLife Stadium’s official capacity of 82,566 is bigger than the Superdome’s 72,003. So, even if it’s cold, having 10,000 more people on hand probably means more bandwidth consumed, even if this year’s game doesn’t have a power blackout in the second half. (And even if it does, Samberg said the network shouldn’t go down since Verizon has backup power supplies on hand.)

Find the DAS antenna! Credit: Verizon Wireless

Find the DAS antenna! Credit: Verizon Wireless

Our only problem with record wireless numbers from Sunday’s game is that we probably won’t ever see an actual number, since Verizon historically shies away from providing a score. Instead it just issues press releases saying things like “way more traffic this year than last!” and then expects us all to believe that without numbers. The good news for fans at the game is that the in-stadium Wi-Fi network, also built by Verizon, is free and open to customers of all carriers, or basically anyone with a device that has a Wi-Fi chip. But Verizon, like big competitor AT&T, has been beefing up its DAS installations significantly because most people try cellular first, even at stadiums, before instructing their phones to find a Wi-Fi network. AT&T, to its credit, usually does deliver a wireless scorecard quickly after big events. So at least from AT&T’s perspective we should find out if this year’s game sets another record.

Aside from the stadium improvements, Verizon will be showing a demonstration of a technology this week that could make DAS more of a competitor to Wi-Fi on the high bandwidth side of things. Called LTE multicast, the technology basically establishes set channels for LTE devices that will “broadcast” video, like a TV channel. (This idea is similar to the StadiumVision Mobile technology Cisco uses at stadiums like Barclays Center.) Theoretically, LTE multicast could let fans use a cellular connection to view multiple video streams, something you would need to use Wi-Fi for it to have any chance of working. But the multicast demo won’t take place at MetLife, but instead at Bryant Park in Manhattan this week. If you are in the city, check out the demo and let us know what you think.

No NFL Mobile at MetLife

And here’s something else you won’t be able to use at MetLife during the Super Bowl: Verizon’s own NFL Mobile app, which outside the stadium will be the only smartphone platform you’ll be able to watch the game on. (The Fox streaming site and app will only work with tablets and desktops or laptops, per the league’s rights agreement with Verizon.) Next year, the rights for NFL Mobile will change and if the local game (like, say, the Super Bowl) is on TV, you’ll be able to use NFL Mobile to watch it even if you’re at the stadium. But not this year! (To give one answer as to why, if you are at the game, you might want to watch it on your cell phone, we say: Bathroom or beer lines!)

More stadium infrastructure photos below:

AT&T DAS antennas at MetLife. Credit: AT&T

AT&T DAS antennas at MetLife. Credit: AT&T

AT&T's new head-end building at MetLife, where its DAS gear is housed. Credit: AT&T

AT&T’s new head-end building at MetLife, where its DAS gear is housed. Credit: AT&T

Inside the AT&T head-end building at MetLife. Cables! Credit: AT&T

Inside the AT&T head-end building at MetLife. Cables! Credit: AT&T

Stadium Tech Report: DAS, Wi-Fi puts end to no-signal problem at Denver’s Sports Authority Field

PeytonThese days, Denver’s Sports Authority Field at Mile High is the new home of the NFL’s most prolific signal-caller. With a record season for passing yards and passing touchdowns, Broncos quarterback Peyton Manning is recognizable for his animated pointing, shouting and line-of-scrimmage audibles, the ultimate practicioner of last-second communication.

Not too long ago, the fans at Mile High might have had to resort to the same tactics to communicate, using hand waving or shouting, since getting a cell signal was next to impossible. “Forget making a phone call, you couldn’t even send a text,” said Rick Seifert, communications manager for the Broncos’ stadium management company. “And it wasn’t just the fans. We [the staff] couldn’t make calls in the stadium to do our jobs.”

But in 2012, the Broncos changed all that with the installation of a full-featured distributed antenna system (DAS) deployed by TE Connectivity, and a fan-facing Wi-Fi network installed by Verizon Wireless.

Russ Trainor

Russ Trainor

The Broncos also put in a huge new digital scoreboard and robust back-end connectivity provided by Comcast as part of their blitz of networking improvements, and this past fall, AT&T joined in by upgrading its connection to the stadium’s DAS. By next year the Broncos hope to add AT&T and Sprint to its roster of Wi-Fi service providers, reflecting what vice president of information technology Russ Trainor sees as a “never ending growth” of wireless in-stadium consumption.

All carriers on board, slowly

One of the biggest problems with DAS deployments in stadiums is convincing major cellular carriers to work together. Since each carrier wants to deploy systems to do the best job for its customers, there is often a difference in opinion on strategy and operations, which is often followed by similar snags in contract negotiations. Trainor said that the stadium, built in 2001, presented unique RF challenges to wireless with its primarily exposed-steel construction. Verizon and Sprint were the first carriers to sign up for the neutral DAS, followed by AT&T this fall.

DAS equipment at Sports Authority Field. Credit: Denver Broncos

DAS equipment at Sports Authority Field. Credit: Denver Broncos

“It was tough to get them [all the major carriers] to agree on DAS, but we have good engineers on the back end and we came up with a nice solution for everybody,” said Trainor. While the antennas and stadium network are neutral, each carrier provides its own back-end gear, much of which at Mile High had to be placed in a building built outside the facility specifically to house telecom gear. In many stadium DAS deployments, the telecom gear can take up thousands of square feet, which can be challenging to find in facilities built before such needs were known.

“There’s no room inside for all the space they [the carriers] wanted,” Seifert said.

The Wi-Fi network, deployed by Verizon, uses Cisco equipment and is also a neutral host infrastructure, meaning that other carriers could use it to provide Wi-Fi connectivity to their clients if they so choose. According to Seifert, AT&T and Sprint will offer Wi-Fi services to customers next season, in part to answer the consistently growing demand. Like in other stadiums, fans at Sports Authority Field know what to do when they finally find bandwidth: Use more.

Steady growth in wireless use

When Sports Authority Field is at its listed capacity of 76,125 on game days, it becomes the 14th-largest city in Colorado, Trainor said. The team has already seen 525,000 downloads of its mobile application, which provides such in-stadium features as four different video replay angles, a connection to the NFL Network’s RedZone channel, and a direct link to the radio feed from hometown sports station KOA. The application is geo-fenced to ensure that the video rights are only used inside the stadium, and to give fans there a unique game-day experience.

Wi-Fi antennas on stadium overhang. Credit: Denver Broncos

Wi-Fi antennas on stadium overhang. Credit: Denver Broncos

According to Trainor, the team usually sees an average of 4,000 simultaneous connections on the Verizon Wi-Fi network on game days, though on colder days when fans need to wear gloves that number can drop in half. Trainor said the Cisco infrastructure is designed to accomodate 25,000 concurrent connections, a number the team hasn’t yet reached. However, the team did have to double the back-end capacity already for the Wi-Fi network, which is being used more as more fans find it.

“Word of mouth really gets [usage] going,” said Trainor, who noted that at a Kenny Chesney concert last year, the stadium crew saw data uploads outpace data downloads for the first time — a sure sign that fans in attendance were using the network to do things like share pictures and videos with their social-network connections.

“We haven’t seen any true bottlenecks yet, but usage is consistently rising, game after game, for concerts, soccer and football,” Trainor said.

Rick Seifert

Rick Seifert

A good sign from the Wi-Fi networking statistics is a shift in usage from the often crowded 2.4 GHz bands to the 5 GHz bands, which Trainor said is likely due to fans using the latest 5 series iPhones, which support the 5 GHz Wi-Fi frequency. And no matter what happens to the Broncos in the playoffs, Trainor and Seifert know what they will be doing this summer: Upgrading the network components, in the never-ending battle to provide bandwidth.

“Verizon and Sprint have already made significant upgrades to their DAS deployments because of demand and changes in technology, like LTE,” said Seifert. “And next year we’ll probably see AT&T circle back again. It’s very dynamic.”

“As smart phones get smarter it’s a never-ending challenge” to provide connectivity, Trainor said. “It’s a job that’s never finished.”

‘Big Four’ wireless carriers sign on to use DAS at Niners’ Levi’s Stadium

Levi's StadiumWhenever you are putting a neutral DAS in a stadium, one of the biggest challenges is convincing the major cellular carriers to participate in your deployment. Even before their new arena has opened, the San Francisco 49ers have scored a victory of sorts with the news that the four biggest U.S. cellular carriers — AT&T, Verizon, Sprint and T-Mobile — have all agreed to use the neutral-hosted DAS at Levi’s Stadium.

According to DAS Group Professionals, who are behind the DAS deployment at Levi’s, the system of small cellular antennas will also be synchronized with the stadium’s Wi-Fi coverage to produce “seamless” coverage for fans. Though we are historically wary of promises made before networks go live, the fact that the big players in U.S. cellular all agreed to use the neutral DAS shows at least in some way that the folks putting together the technology at Levi’s have a convincing story.

We’ll have more from DGP and the Levi’s DAS deployment at some point in the future, so stay tuned for the stadium launch that has everyone in the stadium tech marketplace watching and waiting.

Stadium Tech Report: Boingo, AT&T answer call for more DAS bandwidth at Chicago’s Soldier Field

Soldier FieldWhether or not the Chicago Bears will beat the Green Bay Packers this Sunday is a question still answerable only by higher deities. But if those powers that be decide to unleash weather conditions of biblical proportions again, at the very least the Soldier Field folks can be confident that their in-stadium network will stand up to the challenge.

Earlier this year, tornado weather hit Soldier Field and caused an unusual 2-hour delay during the Nov. 17 Bears-Ravens game. While the deluge was too dangerous for football, for Doug Lodder and Luca Serra, it was a big test to see if the stadium wireless network upgrades they’d recently helped install could stand up to a huge, unexpected traffic surge.

“We thought we’d probably see some good [network] usage with the weather delay and tornados,” said Lodder, vice president of business development at Boingo Wireless, which recently concluded a big network upgrade at the home of the NFL’s Chicago Bears.

“We saw double the usual traffic from both AT&T and Verizon customers during the 2-hour delay,” said Serra, director of sponsorship and media for SMG, the outside entity that manages Soldier Field for its owners, the Chicago Park District. “During that break, we didn’t hear a single complaint [about the wireless network] from the media or the fans. It was a great test — not a desired one in any way, shape or form, but a great test.”

Upgrades needed at ‘oldest’ big stadium Wi-Fi network

Luca Serra, SMG

Luca Serra, SMG

Even though you can’t really plan or design for surges that might happen during extreme situations, Serra and Lodder and others running the Soldier Field network knew last year that an overall upgrade was needed. Home of one of the earliest (if not the first) big-stadium DAS and Wi-Fi networks, Soldier Field was showing 80 percent year-over-year traffic growth on the Wi-Fi network, Lodder said in a recent joint phone interview with Serra.

Along with DAS partner AT&T, SMG and Boingo spent last year adding more DAS capacity so that Solider Field now has 250 DAS antennas to go along an undisclosed number of new Wi-Fi APs. Though AT&T did some specific upgrades to its DAS presence at Soldier Field (including adding support for its new 4G LTE network), all the major wireless carriers are represented in the DAS via the neutral hosting of Boingo.

Why did the firms decide Soldier Field needed an upgrade? While Lodder said its DAS, which first started operating in 2011, was “pretty cutting edge at the time,” during the past couple years, data usage at Soldier Field has skyrocketed.

“Everyone has a device, and there’s a lot more social engagement and sharing,” Lodder said. “We’ve seen a huge increase in data use.”

“With this large increase in use, we said ‘let’s address the issue’ and not let it get to a point where it’s a problem,” Serra said.

Moving fast and finding space

DAS gear at Soldier Field. Credit: Boingo Wireless

DAS gear at Soldier Field. Credit: SMG/Soldier Field

Both Serra’s and Lodder’s teams were familiar with the need to balance performance and aesthetics at the venerable stadium perched on the edge of Lake Michigan. Since the network was put in after Soldier Field’s extensive renovation was finished in 2003, special attention needed to be paid to placements of Wi-Fi and DAS antennas.

“It wasn’t just bolting APs on,” Serra said. “We had to be very protective of the aesthetics of the building.”

For Lodder, that sometimes means that the best placement isn’t always available.

“There’s always challenges with architecture at stadiums,” Lodder said. “In a perfect world, there might be a certain place you want to put an antenna. In the real world, you can almost always count on changing that 20 percent of the time.”

Doug Lodder, Boingo

Doug Lodder, Boingo

A bigger construction concern of late is where to put the extensive amount of back-end telco gear needed to support the DAS infrastructure and upgrades. “Carrier base stations need several thousand square feet of space, and those need to be rooms with connectivity and power,” Lodder said. At Solider Field, Serra said some unfinished space north of the stadium proper was converted into a DAS equipment room, since there wasn’t enough available space in the main building. Lodder said for the recent upgrade, additional power upgrades were installed as well.

Still, even with all the challenges the Boingo, AT&T and SMG teams were able to go from design to finished construction of the recent upgrade in 45 days this past summer, in part thanks to some interesting work schedules.

“After one preseason [football] game we had scaffolding going up at night, with three 24-hour shifts,” Lodder said. “It was a pretty accelerated build.”

There is no such thing as free Wi-Fi

One other interesting twist to the Boingo Wi-Fi network at Soldier Field is its cost — the company charges a $1.99 per day fee for Wi-Fi use, or you can purchase a regular Boingo monthly plan for $7.95 which will also provide access at Soldier Field.

Wi-Fi antennas hidden under stands at Soldier Field. Credit: SMG/Soldier Field

Wi-Fi antennas hidden under stands at Soldier Field. Credit: SMG/Soldier Field

While many if not most professional teams are making Wi-Fi services free to paying ticketholders, Serra and Lodder said that the Bears aren’t.

“The Bears are a tenant of the Park District, and no one [from the team] has ever said ‘we’re going to pay to turn it [Wi-Fi] on for free,’ ” Serra said. Lodder noted that in the end, someone somehow has to pay the cost of connectivity.

“Someone’s always got to pay for it — Wi-Fi isn’t free,” Lodder said. “Some teams may feel they’ve made enough on the tickets to offer [Wi-Fi] free, while in other cases users pay either with money or with eyeballs, by viewing some ads.”

While the NFL itself is still somewhat vague on its league-wide plans for in-stadium Wi-Fi — other than saying that teams should provide it — Serra and Lodder are confident that Soldier Field will be at the top of the chart when it comes to network performance.

“If and when they [the NFL] roll out standards for network performance, we’re going to exceed them,” Lodder said.

AT&T adds 185 Wi-Fi antennas in Charlotte’s Bank of America Stadium upgrade

Bank of America StadiumA year ago, folks who run the Charlotte Panthers’ Bank of America Stadium were telling us how much the fans there liked the Wi-Fi network installed by AT&T. It’s possible Panthers fans like it even more now, with 185 new Wi-Fi access points to help improve coverage.

We haven’t talked to anyone in Charlotte this time around, but a press release from AT&T shares some interesting stats about wireless usage at BofA Stadium. According to AT&T, so far this season fans in Charlotte have sent 1.9 million MB across the Wi-Fi stadium network, 50 percent more data than during the same period in 2012. And the in-stadium DAS usage grew even more, by 90 percent compared to a year ago, according to AT&T.

What does that mean for stadium owners and operators? It may simply mean that whatever loads you think you are designing for, design for more. If AT&T and the Panthers had to go from 460 APs to 645 in less than a year, that’s a good sign that stadium Wi-Fi networks are like Fields of Dreams: If you build it, they will come, and they will download.