Stadium Tech Report: Verizon, AT&T DAS upgrades at MetLife Stadium await Super Bowl Sunday

Verizon branded gate at MetLife Stadium. Credit: Verizon Wireless

Verizon branded gate at MetLife Stadium. Credit: Verizon Wireless

Super Bowl foes the Seattle Seahawks and the Denver Broncos have only had two weeks to prepare for each other. But Verizon Wireless and AT&T have been preparing for the roman-numeral date at MetLife Stadium for more than a year. Will the Verizon Wi-Fi and DAS, and AT&T’s separate DAS be able to handle the wireless needs of the fans at the NFL’s biggest game? Tune in Sunday to see!

We might be one of the only news outlets who care more about the wireless networking at Super Bowls than the game itself, but for many in the stadium tech industry the biggest single game in America’s most popular sport is always somewhat of a wireless watershed. Perhaps at no other event do attendees spend so much time shooting selfies and posting them as they do from the site of Super Bowl Sunday. Even in the expected cold, it should be no different this week at Super Bowl XLVIII at MetLife Stadium in New Jersey, where game time is scheduled for around 6:30 p.m. Eastern.

But well before that, fans will be testing the Verizon-built Wi-Fi network and both of the big carriers’ DAS deployments inside MetLife. According to reps from both companies that we spoke with last week, the carriers are ready.

Verizon spokesman David Samberg told us via email last week that an upgrade of the Verizon DAS in MetLife this past season means there are now more than 500 DAS antennas inside the facility. See some of the photos provided by Verizon that show the clever hiding spots Verizon engineers have found over the last 18 months as they’ve added capacity to a network built just a few years ago.

AT&T has also put in a brand-new DAS over the past year, with another 500-plus antennas of its own. “For the last year or so we’ve been working on our pre-game and game day network playbook in an effort to provide the best possible wireless experience for our customers,” said Michael Maus, assistant vice president of network services at AT&T, via email. “In anticipation of the huge volume of data and voice usage expected [for the Super Bowl], we’ve built a new state of the art antenna system inside the stadium, we’re rolling in portable cell sites both at the stadium, and to support the tailgate areas, and we’ve augmented coverage in New York City to support the activities there.”

Rooftop DAS equipment. Credit: Verizon Wireless

Rooftop DAS equipment. Credit: Verizon Wireless

For Verizon’s own customers, 4G LTE capacity at MetLife has been quadrupled since last year, according to Samberg, who said that all stadium upgrades were completed by October, giving Verizon multiple chances to test its system against crowds at New York Giants and New York Jets home games. So far, Samberg said, so good.

More traffic than last year’s Super Bowl already

While yours truly opined earlier this year that this year’s Super Bowl might not set a wireless traffic record, we didn’t take into account some simple numbers — mainly, that MetLife Stadium’s official capacity of 82,566 is bigger than the Superdome’s 72,003. So, even if it’s cold, having 10,000 more people on hand probably means more bandwidth consumed, even if this year’s game doesn’t have a power blackout in the second half. (And even if it does, Samberg said the network shouldn’t go down since Verizon has backup power supplies on hand.)

Find the DAS antenna! Credit: Verizon Wireless

Find the DAS antenna! Credit: Verizon Wireless

Our only problem with record wireless numbers from Sunday’s game is that we probably won’t ever see an actual number, since Verizon historically shies away from providing a score. Instead it just issues press releases saying things like “way more traffic this year than last!” and then expects us all to believe that without numbers. The good news for fans at the game is that the in-stadium Wi-Fi network, also built by Verizon, is free and open to customers of all carriers, or basically anyone with a device that has a Wi-Fi chip. But Verizon, like big competitor AT&T, has been beefing up its DAS installations significantly because most people try cellular first, even at stadiums, before instructing their phones to find a Wi-Fi network. AT&T, to its credit, usually does deliver a wireless scorecard quickly after big events. So at least from AT&T’s perspective we should find out if this year’s game sets another record.

Aside from the stadium improvements, Verizon will be showing a demonstration of a technology this week that could make DAS more of a competitor to Wi-Fi on the high bandwidth side of things. Called LTE multicast, the technology basically establishes set channels for LTE devices that will “broadcast” video, like a TV channel. (This idea is similar to the StadiumVision Mobile technology Cisco uses at stadiums like Barclays Center.) Theoretically, LTE multicast could let fans use a cellular connection to view multiple video streams, something you would need to use Wi-Fi for it to have any chance of working. But the multicast demo won’t take place at MetLife, but instead at Bryant Park in Manhattan this week. If you are in the city, check out the demo and let us know what you think.

No NFL Mobile at MetLife

And here’s something else you won’t be able to use at MetLife during the Super Bowl: Verizon’s own NFL Mobile app, which outside the stadium will be the only smartphone platform you’ll be able to watch the game on. (The Fox streaming site and app will only work with tablets and desktops or laptops, per the league’s rights agreement with Verizon.) Next year, the rights for NFL Mobile will change and if the local game (like, say, the Super Bowl) is on TV, you’ll be able to use NFL Mobile to watch it even if you’re at the stadium. But not this year! (To give one answer as to why, if you are at the game, you might want to watch it on your cell phone, we say: Bathroom or beer lines!)

More stadium infrastructure photos below:

AT&T DAS antennas at MetLife. Credit: AT&T

AT&T DAS antennas at MetLife. Credit: AT&T

AT&T's new head-end building at MetLife, where its DAS gear is housed. Credit: AT&T

AT&T’s new head-end building at MetLife, where its DAS gear is housed. Credit: AT&T

Inside the AT&T head-end building at MetLife. Cables! Credit: AT&T

Inside the AT&T head-end building at MetLife. Cables! Credit: AT&T

Fox: We’re ready for big Super Bowl streaming audience

While nobody can predict how things like a stadium blackout or a polar vortex might affect the broadcast of an event like the Super Bowl, executives at Fox are doing all they can to prepare to make sure this year’s online coverage of the NFL’s championship game goes off as well as possible.

With viewership expected to exceed last year’s total of 3 million unique online viewers, the live stream of Super Bowl XLVIII will be available on iPads in the Fox Sports Go app and via a browser at FoxSportsGo.com. The online feed will be available free to anyone with an Internet connection starting at 12 a.m. ET on Sunday, Feb. 2, and will also include access to a Spanish-language broadcast.

But remember: The live streaming from Fox is only available for desktops or laptops, or via an iPad using the FoxSportsGo app. Because of NFL rights contracts, to watch the game live on a smartphone you need to be a Verizon customer and have the $5 per month premium version of the NFL Mobile app installed.

No matter which platform you choose to use, as always the Super Bowl should be a compelling story, even if that tale is something other than the game between the Denver Broncos and the Seattle Seahawks. Super Bowl XLVIII is scheduled to start around 6:30 p.m. ET from MetLife Stadium in New Jersey, where snow and cold may be part of the equation.

“It should be interesting, with [possible] weather in New Jersey,” said Clark Pierce, senior vice president of mobile and advanced platforms for Fox Sports, in a phone interview earlier this week. “We’re ready for traffic to spike at some moment when something happens, and we get a lot of concurrent users. We’ll see how we do.”

Tablet audience keeps fueling growth

Granted, the online viewer numbers for the Super Bowl are completely dwarfed by the regular TV viewers, which for last year’s game totaled 164 million. But having 4 million unique viewers online is a huge digital audience, and it’s one that Pierce said Fox has been planning for over the past few years.

“We know what CBS did [for streaming] and we know what NBC did,” Pierce said of the networks which had the Super Bowl broadcast the past 2 years. “And we’re working with [content delivery network provider] Akamai, and it’s not their first rodeo. So I think we’re ready for whatever can happen.”

Perhaps the biggest number of online viewers for this year’s Super Bowl will come via Apple iPads, the market leader in the tablet form factor. While Fox has not yet released viewer numbers by device or platform for its restricted streaming of the NFC playoff games, it did say it had a record 2.4 million visitors to its FoxSports.com website on championship Sunday, a 53 percent increase over the previous year. Pierce said increases in digital viewers are partially due to the expanding tablet market, and tablet owners getting more comfortable watching sports on a smaller screen.

“The concept of watching TV on a smaller screen is another year down the road and people are just that much more used to it,” said Pierce, who also said that online viewers may choose tablets over laptops or desktops due to clearer pictures.

“People are getting spoiled by Apple and Android devices with really high screen resolution,” Pierce said.

Streaming delay inevitable

What doesn’t work so well with any live streaming option is trying to use it as a “second screen” alongside a live TV broadcast. Because of the technical necessities of putting a live signal online — which includes mainly taking the TV feed and encoding it to the Internet — delays between “live” TV and the show online can be from 20 seconds to a minute or more, Pierce said.

“It does take time to take a big HD picture and encode it into IP,” Pierce said. “There’s just no way around it.”

In the future, Pierce hopes to help Fox add more features to its online offerings, maybe making them something like what broadcast engineers see in the event production studios, with multiple monitors offering different feeds and live stats. Still, just having a free online broadcast is a huge leap forward from the recent past.

“It’s been exciting to build it, and in the future our team hopes to add more content and features,” Pierce said. “The horizon is pretty exciting.”

NFL Championship games live streams: Fox for Niners-Seahawks, CBS for Broncos-Patriots, Verizon NFL Mobile for both

Are you ready for some streaming football? In case you won’t be near a TV for either of Sunday’s championship games, you still may have a chance to watch the games live on a PC, a tablet or a smartphone. Here are the details:

If you’re a customer of a participating provider for Fox Sports Go, you can watch Sunday’s NFC championship game between the San Francisco 49ers and the Seattle Seahawks on the Web or on an Apple iPad using the Fox Sports Go website or mobile app. The Fox Sports Go website address is foxsportsgo.com, and the list of participating providers includes AT&T U-Verse, Comcast Xfinity, Suddenlink, Optimum, Midcontinent Communications, and Wow!. Coverage starts with the FOX NFL Sunday pregame show at 6 p.m. ET. Joe Buck, Troy Aikman, Pam Oliver, and Erin Andrews will call the game at 6:30 p.m. If you’re like me, the mute button on Buck/Aikman comes into play at about 6:35.

CBS will also live-stream its coverage of the AFC championship game from Denver, starting at 3 p.m. ET on Sunday. The games can be watched online at CBSSports.com’s NFL page, or via the CBSSports app, which shows the games via a web page. I did see a little disclaimer that says live streaming is only available via iOS mobile devices.

If you are a Verizon Wireless customer and want to watch the games on your 4G LTE smartphone, you can sign up for Verizon’s NFL Mobile app and its live NFL service, which costs $5 a month extra to whatever cell plan you have. NFL Mobile customers will be able to see live streaming action from both championship games as well as the Super Bowl. Though it suffered glitches earlier this season, Verizon’s NFL Mobile app has been rock-solid for me, with lots of air time during my daughter’s Sunday afternoon music lessons. Don’t get caught without football this weekend, these games are possible classics you don’t want to miss.

Fox Sports Go will live-stream Seahawks-Saints and Niners-Panthers this weekend; CBS will live-stream Patriots-Colts, Broncos-Chargers

Screen shot 2014-01-11 at 9.52.57 AMIf you’re a customer of a participating provider for Fox Sports Go, you can watch this weekend’s NFC playoff games either on the Web or on an Apple iPad using the Fox Sports Go website or mobile app. Here at Mobile Sports Report we love this kind of flexibility, since it lets us watch games on the big desktop PC screen if and when other family members want to use the main TV to watch Harry Potter movies. The Fox Sports Go website address is foxsportsgo.com, and the list of participating providers includes AT&T U-Verse, Comcast Xfinity, Suddenlink, Optimum, Midcontinent Communications, and Wow!.

CBS will also live-stream its coverage of the AFC divisional games this weekend, starting with the Colts-Patriots game at 8:15 p.m. ET on Saturday followed by the Broncos-Chargers game Sunday at about 4:40 p.m. Sunday. The games can be watched online at CBSSports.com’s NFL page, or via the CBSSports app, which I believe gets around the Verizon phone-ban by just showing the games via a web page. I did see a little disclaimer that says live streaming is only available via iOS devices. Someone out there give it a shot (I have an Android phone) and let us know if it works on an iPhone. You do not need to subscribe to any TV service to see the CBS live streams.

For Fox, Saturday’s Seattle Seahawks vs. New Orleans Saints game (4:30 p.m. ET start) and Sunday’s San Francisco 49ers vs. Carolina Panthers tilt (1 p.m. ET start) will both be live streamed, according to a Fox news release. To see the stream you need to validate your pay-TV subscription, and once that’s done you can watch all the other Fox games. The Super Bowl, which Fox is broadcasting this year, will not require a cable subscription.

Though the Fox Sports Go app is available for both the iPad and the iPhone, the Fox news release very specifically notes that the game is available only for the tablet version of the app. We are guessing here but we suspect that Verizon’s deal with the league for its NFL Mobile app — the only app for smartphone live NFL action — precludes Fox’s ability to offer live streaming to a phone-type device. All playoff games this weekend, including the AFC games on CBS (according to my phone), will also be available to NFL Mobile subscribers. NFL Mobile requires a Verizon 4G LTE phone and a $5 per month NFL Mobile subscription.

Someday, the league will get all these rights simplified. But not this weekend.

Stadium Tech Report: DAS, Wi-Fi puts end to no-signal problem at Denver’s Sports Authority Field

PeytonThese days, Denver’s Sports Authority Field at Mile High is the new home of the NFL’s most prolific signal-caller. With a record season for passing yards and passing touchdowns, Broncos quarterback Peyton Manning is recognizable for his animated pointing, shouting and line-of-scrimmage audibles, the ultimate practicioner of last-second communication.

Not too long ago, the fans at Mile High might have had to resort to the same tactics to communicate, using hand waving or shouting, since getting a cell signal was next to impossible. “Forget making a phone call, you couldn’t even send a text,” said Rick Seifert, communications manager for the Broncos’ stadium management company. “And it wasn’t just the fans. We [the staff] couldn’t make calls in the stadium to do our jobs.”

But in 2012, the Broncos changed all that with the installation of a full-featured distributed antenna system (DAS) deployed by TE Connectivity, and a fan-facing Wi-Fi network installed by Verizon Wireless.

Russ Trainor

Russ Trainor

The Broncos also put in a huge new digital scoreboard and robust back-end connectivity provided by Comcast as part of their blitz of networking improvements, and this past fall, AT&T joined in by upgrading its connection to the stadium’s DAS. By next year the Broncos hope to add AT&T and Sprint to its roster of Wi-Fi service providers, reflecting what vice president of information technology Russ Trainor sees as a “never ending growth” of wireless in-stadium consumption.

All carriers on board, slowly

One of the biggest problems with DAS deployments in stadiums is convincing major cellular carriers to work together. Since each carrier wants to deploy systems to do the best job for its customers, there is often a difference in opinion on strategy and operations, which is often followed by similar snags in contract negotiations. Trainor said that the stadium, built in 2001, presented unique RF challenges to wireless with its primarily exposed-steel construction. Verizon and Sprint were the first carriers to sign up for the neutral DAS, followed by AT&T this fall.

DAS equipment at Sports Authority Field. Credit: Denver Broncos

DAS equipment at Sports Authority Field. Credit: Denver Broncos

“It was tough to get them [all the major carriers] to agree on DAS, but we have good engineers on the back end and we came up with a nice solution for everybody,” said Trainor. While the antennas and stadium network are neutral, each carrier provides its own back-end gear, much of which at Mile High had to be placed in a building built outside the facility specifically to house telecom gear. In many stadium DAS deployments, the telecom gear can take up thousands of square feet, which can be challenging to find in facilities built before such needs were known.

“There’s no room inside for all the space they [the carriers] wanted,” Seifert said.

The Wi-Fi network, deployed by Verizon, uses Cisco equipment and is also a neutral host infrastructure, meaning that other carriers could use it to provide Wi-Fi connectivity to their clients if they so choose. According to Seifert, AT&T and Sprint will offer Wi-Fi services to customers next season, in part to answer the consistently growing demand. Like in other stadiums, fans at Sports Authority Field know what to do when they finally find bandwidth: Use more.

Steady growth in wireless use

When Sports Authority Field is at its listed capacity of 76,125 on game days, it becomes the 14th-largest city in Colorado, Trainor said. The team has already seen 525,000 downloads of its mobile application, which provides such in-stadium features as four different video replay angles, a connection to the NFL Network’s RedZone channel, and a direct link to the radio feed from hometown sports station KOA. The application is geo-fenced to ensure that the video rights are only used inside the stadium, and to give fans there a unique game-day experience.

Wi-Fi antennas on stadium overhang. Credit: Denver Broncos

Wi-Fi antennas on stadium overhang. Credit: Denver Broncos

According to Trainor, the team usually sees an average of 4,000 simultaneous connections on the Verizon Wi-Fi network on game days, though on colder days when fans need to wear gloves that number can drop in half. Trainor said the Cisco infrastructure is designed to accomodate 25,000 concurrent connections, a number the team hasn’t yet reached. However, the team did have to double the back-end capacity already for the Wi-Fi network, which is being used more as more fans find it.

“Word of mouth really gets [usage] going,” said Trainor, who noted that at a Kenny Chesney concert last year, the stadium crew saw data uploads outpace data downloads for the first time — a sure sign that fans in attendance were using the network to do things like share pictures and videos with their social-network connections.

“We haven’t seen any true bottlenecks yet, but usage is consistently rising, game after game, for concerts, soccer and football,” Trainor said.

Rick Seifert

Rick Seifert

A good sign from the Wi-Fi networking statistics is a shift in usage from the often crowded 2.4 GHz bands to the 5 GHz bands, which Trainor said is likely due to fans using the latest 5 series iPhones, which support the 5 GHz Wi-Fi frequency. And no matter what happens to the Broncos in the playoffs, Trainor and Seifert know what they will be doing this summer: Upgrading the network components, in the never-ending battle to provide bandwidth.

“Verizon and Sprint have already made significant upgrades to their DAS deployments because of demand and changes in technology, like LTE,” said Seifert. “And next year we’ll probably see AT&T circle back again. It’s very dynamic.”

“As smart phones get smarter it’s a never-ending challenge” to provide connectivity, Trainor said. “It’s a job that’s never finished.”

Stadium Tech Report: Boingo, AT&T answer call for more DAS bandwidth at Chicago’s Soldier Field

Soldier FieldWhether or not the Chicago Bears will beat the Green Bay Packers this Sunday is a question still answerable only by higher deities. But if those powers that be decide to unleash weather conditions of biblical proportions again, at the very least the Soldier Field folks can be confident that their in-stadium network will stand up to the challenge.

Earlier this year, tornado weather hit Soldier Field and caused an unusual 2-hour delay during the Nov. 17 Bears-Ravens game. While the deluge was too dangerous for football, for Doug Lodder and Luca Serra, it was a big test to see if the stadium wireless network upgrades they’d recently helped install could stand up to a huge, unexpected traffic surge.

“We thought we’d probably see some good [network] usage with the weather delay and tornados,” said Lodder, vice president of business development at Boingo Wireless, which recently concluded a big network upgrade at the home of the NFL’s Chicago Bears.

“We saw double the usual traffic from both AT&T and Verizon customers during the 2-hour delay,” said Serra, director of sponsorship and media for SMG, the outside entity that manages Soldier Field for its owners, the Chicago Park District. “During that break, we didn’t hear a single complaint [about the wireless network] from the media or the fans. It was a great test — not a desired one in any way, shape or form, but a great test.”

Upgrades needed at ‘oldest’ big stadium Wi-Fi network

Luca Serra, SMG

Luca Serra, SMG

Even though you can’t really plan or design for surges that might happen during extreme situations, Serra and Lodder and others running the Soldier Field network knew last year that an overall upgrade was needed. Home of one of the earliest (if not the first) big-stadium DAS and Wi-Fi networks, Soldier Field was showing 80 percent year-over-year traffic growth on the Wi-Fi network, Lodder said in a recent joint phone interview with Serra.

Along with DAS partner AT&T, SMG and Boingo spent last year adding more DAS capacity so that Solider Field now has 250 DAS antennas to go along an undisclosed number of new Wi-Fi APs. Though AT&T did some specific upgrades to its DAS presence at Soldier Field (including adding support for its new 4G LTE network), all the major wireless carriers are represented in the DAS via the neutral hosting of Boingo.

Why did the firms decide Soldier Field needed an upgrade? While Lodder said its DAS, which first started operating in 2011, was “pretty cutting edge at the time,” during the past couple years, data usage at Soldier Field has skyrocketed.

“Everyone has a device, and there’s a lot more social engagement and sharing,” Lodder said. “We’ve seen a huge increase in data use.”

“With this large increase in use, we said ‘let’s address the issue’ and not let it get to a point where it’s a problem,” Serra said.

Moving fast and finding space

DAS gear at Soldier Field. Credit: Boingo Wireless

DAS gear at Soldier Field. Credit: SMG/Soldier Field

Both Serra’s and Lodder’s teams were familiar with the need to balance performance and aesthetics at the venerable stadium perched on the edge of Lake Michigan. Since the network was put in after Soldier Field’s extensive renovation was finished in 2003, special attention needed to be paid to placements of Wi-Fi and DAS antennas.

“It wasn’t just bolting APs on,” Serra said. “We had to be very protective of the aesthetics of the building.”

For Lodder, that sometimes means that the best placement isn’t always available.

“There’s always challenges with architecture at stadiums,” Lodder said. “In a perfect world, there might be a certain place you want to put an antenna. In the real world, you can almost always count on changing that 20 percent of the time.”

Doug Lodder, Boingo

Doug Lodder, Boingo

A bigger construction concern of late is where to put the extensive amount of back-end telco gear needed to support the DAS infrastructure and upgrades. “Carrier base stations need several thousand square feet of space, and those need to be rooms with connectivity and power,” Lodder said. At Solider Field, Serra said some unfinished space north of the stadium proper was converted into a DAS equipment room, since there wasn’t enough available space in the main building. Lodder said for the recent upgrade, additional power upgrades were installed as well.

Still, even with all the challenges the Boingo, AT&T and SMG teams were able to go from design to finished construction of the recent upgrade in 45 days this past summer, in part thanks to some interesting work schedules.

“After one preseason [football] game we had scaffolding going up at night, with three 24-hour shifts,” Lodder said. “It was a pretty accelerated build.”

There is no such thing as free Wi-Fi

One other interesting twist to the Boingo Wi-Fi network at Soldier Field is its cost — the company charges a $1.99 per day fee for Wi-Fi use, or you can purchase a regular Boingo monthly plan for $7.95 which will also provide access at Soldier Field.

Wi-Fi antennas hidden under stands at Soldier Field. Credit: SMG/Soldier Field

Wi-Fi antennas hidden under stands at Soldier Field. Credit: SMG/Soldier Field

While many if not most professional teams are making Wi-Fi services free to paying ticketholders, Serra and Lodder said that the Bears aren’t.

“The Bears are a tenant of the Park District, and no one [from the team] has ever said ‘we’re going to pay to turn it [Wi-Fi] on for free,’ ” Serra said. Lodder noted that in the end, someone somehow has to pay the cost of connectivity.

“Someone’s always got to pay for it — Wi-Fi isn’t free,” Lodder said. “Some teams may feel they’ve made enough on the tickets to offer [Wi-Fi] free, while in other cases users pay either with money or with eyeballs, by viewing some ads.”

While the NFL itself is still somewhat vague on its league-wide plans for in-stadium Wi-Fi — other than saying that teams should provide it — Serra and Lodder are confident that Soldier Field will be at the top of the chart when it comes to network performance.

“If and when they [the NFL] roll out standards for network performance, we’re going to exceed them,” Lodder said.