December 22, 2014

Stadium Tech Report: Arizona Cardinals get stadium ready for Super Bowl with Wi-Fi upgrade

University of Phoenix Stadium. Credit all photos: Arizona Cardinals.

University of Phoenix Stadium. Credit all photos: Arizona Cardinals. (click on photos for larger image)

As they get set to host their second Super Bowl this February, the IT team at the University of Phoenix Stadium in Glendale, Ariz., knows now what they didn’t know then: The big game requires a big wireless network. Bigger than you think.

“It’s funny to look back now on that first Super Bowl,” said Mark Feller, vice president of information technology for the Arizona Cardinals, speaking of Roman numeral game XLII, held in the barely 2-year-old facility on Feb. 3, 2008. With a couple Fiesta Bowls and one BCS championship game (2007) under his belt in a facility that opened with Wi-Fi and DAS, Feller said he and his team “thought we had a good handle” on what kind of network was required for a Super Bowl crowd.

The NFL, he said, begged to differ. Those college games might have been big, but the Super Bowl was bigger.

“We had the Fiesta Bowl that year at night, and when the game was over there were people from the NFL there wanting to know when they could set up,” said Feller in a recent phone interview. “This year, we’re much better prepared. We know what the water temperature is this time.”

Rip and replace, with more and better gear

Wi-Fi railing antennas

Wi-Fi railing antennas

For Super Bowl XLIX, scheduled to take place in Glendale on Feb. 1, 2015, Feller and his team have not just tuned up their network — they have done a full rip and replace of the Wi-Fi system, installing new Cisco gear from back end to front, in order to support a wireless game-day demand that is historically second to none. Integrator CDW has led the Wi-Fi effort and Daktronics and ProSound did the installation of new video screens, and neutral host Crown Castle has overseen a revamp of the DAS system, again with more antennas added to bolster coverage. In all, there has been more than $8 million in wireless improvements before this year, Feller said, as well as another $10 million for two new video boards that are each three times larger than what was there before.

“The last three or four years there have been things we knew we needed to improve [before the Super Bowl],” Feller said. After extensive work with the NFL’s technical team — this time well before the Fiesta Bowl — Feller oversaw a “top to bottom” refurbishment that included replacing core Cisco networking gear with newer gear, and new and more Wi-Fi access points that now total somewhere north of the 750 mark, with some more to be added before the big game. The new network, which was in place for the start of the current NFL season, has undergone testing by CDW at each home game, Feller said. CDW also plans to expand the network outside the stadium before the Super Bowl, in part to handle the extra events that take place not just on game day but in the days leading up to the game.

“The plan is to install more [coverage] outside, in the plaza areas,” Feller said.

When it opened in 2006, the $455 million University of Phoenix Stadium was one of the first with full-bowl Wi-Fi, using Cisco gear from the inside out. “Cisco was in here before they called it [their solution] ‘connected stadium’,” Feller said. From core network switches to firewalls to edge switches, this year there is all new Cisco gear in the venue, as well as new 3700 series APs, with panel antennas and antennas in handrails.

“Handrail [antennas] are sometimes a bit of a challenge, because you need to drill through concrete that’s 40 feet up in the air, behind another ceiling,” said Feller, describing one particular design challenge. Another one was mounting antennas on drop rods from the catwalks below the stadium’s retractable roof, to serve the upper-area seating. There are also some new Wi-Fi APs on the front row of the seating bowl, pointing up into the crowd.

“It was a fun project,” Feller said.

Stadium with roof open

Stadium with roof open


All on board for the DAS

The upgrade for the stadium’s DAS, led by Crown Castle, was just finished a few weeks ago, Feller said, and included more coverage outside the stadium as well, with antennas placed on light poles and on the stadium’s shell.

“Crown Castle did a great job of managing the carriers” on what is a 48-sector DAS, Feller said. “It [the upgrade] really required a lot of creative thinking from their engineers.”

Since the stadium was originally designed with wireless in mind, Feller and his team didn’t need to build new head end room for the DAS upgrades. “But I wouldn’t say we have plenty of space left,” he added. “We’ve got a lot of new equipment.”

Though all the major carriers are expected to be on the DAS by the big game, league partner Verizon Wireless should have some special projects up its sleeve for the big game, including another demonstration of its LTE Broadcast technology, which optimizes things like live video over LTE cellular links.

New Cardinals app a preview of Super Bowl version?

The Cardinals also had a new version of the game-day team app for this season, built by stadium-app leader YinzCam. According to Feller the new app supports three different live video feeds, as well as instant replays.

Wi-Fi antenna on railing

Wi-Fi antenna on railing

“It’s really cool to have that ability to watch things like a touchdown pass at the end of the game,” Feller said. And while no details have yet been revealed, in an interview with NFL CIO Michelle McKenna-Doyle earlier this year MSR learned that the league and YinzCam are working on a Super Bowl app with its own new bells and whistles. (Stay tuned for more info on the Super Bowl app.)

In addition to two more regular-season home games in December, the University of Phoenix Stadium will have at least a couple more dry runs to help test the network, during the Dec. 31 Fiesta Bowl and during the NFL’s Pro Bowl, scheduled for Jan. 25. And though the Cardnials lost to the Atlanta Falcons Sunday, at 9-3 they are still tied with the Green Bay Packers for the best record in the NFC, something that has the Phoenix faithful optimistic about the postseason.

“We’re going to get some more test runs, on New Year’s Eve and during the Pro Bowl,” Feller said. “And maybe some home playoff games as well!”

(more photos below)

Wi-Fi antenna in roof rafters

Wi-Fi antenna in roof rafters

More antennas in rafters

More antennas in rafters

Wi-Fin antenna under overhang

Wi-Fi antenna under overhang

Stadium Tech Report — NFL stadium technology reports — AFC East

Editor’s note: The following team-by-team capsule reports of NFL stadium technology deployments are an excerpt from our most recent Stadium Tech Report, THE FOOTBALL ISSUE. To get all the capsules in one place as well as our featured reports, interviews and analysis, download your free copy of the full report today.

AFC EAST

Reporting by Chris Gallo

Buffalo Bills
Ralph Wilson Stadium
Seating Capacity: 71,757
Wi-Fi – No
DAS – Yes, 200 antennas
Beaconing – No

The Buffalo Bills have had a busy offseason. After the passing of longtime owner Ralph Wilson, the organization was bought by Buffalo husaband-and-wife team of Terry and Kim Pegula for $1.4 billion. Even while the future of the team was uncertain, the stadium named after its longtime owner became fan-friendlier. New gates to enter the stadium, HD video boards, and increased cell service are just a few of the improvements. No Wi-Fi, but Ralph Wilson Stadium does have over 200 DAS antennas.

New England Patriots
Gillette Stadium
Seating Capacity: 68,756
Wi-Fi-Yes
DAS-Yes
Beaconing – No

The New England Patriots are doing everything to get fans off the couch and in Gillette Stadium, with Wi-Fi outfitted by Extreme Networks, a team-centric Game Day Live mobile app, and a squad that’s won the AFC East 5 years in a row. Is it time for another Patriots Super Bowl run?

Miami Dolphins
Sun Life Stadium
Seating Capacity: 75,540
Wi-Fi – Yes, 1,100 access points
DAS-Yes
Beaconing – Yes

Near the end of the 2013 season, Sun Life Stadium became one of the NFL’s first venues to feature beacon technology. The Qualcomm Gimbal contextual awareness platform delivers coupons for concessions as fans walk by and alerts them of shorter wait times on the concourse. Plus, AT&T upgraded the stadium with more than 1,100 Wi-Fi access points and DAS antennas a year ago. All of this has the Dolphins delivering one of the better wireless game day experiences. The search to find a new Dan Marino, however, is still a work in progress.

New York Jets
MetLife Stadium
Seating Capacity: 82,500
Wi-Fi – Yes, 850 access points
DAS – Yes, over 600 antennas
Beaconing – No

There are lots of benefits to hosting a Super Bowl – including the improved connectivity of your stadium. After AT&T and Verizon spent over a year outfitting MetLife Stadium with their own DAS deployments, the stadium saw a 60 percent increase in wireless data from the previous Super Bowl. Safe to say the stadium is well-equipped to easily connect fans with 850 Wi-Fi access points and more than 600 DAS antennas. MetLife Stadium enters its fourth season and continues to make the fan experience unforgettable. Now can the Jets make 2014 an unforgettable season and find their way back to the playoffs?

Niners’ home opener tops Super Bowl for Wi-Fi data traffic with 3.3 Terabytes

Fans take pictures of opening kickoff from southwest concourse. Credit, all photos: Paul Kapustka, MSR

Fans take pictures of opening kickoff from southwest concourse. Credit, all photos: Paul Kapustka, MSR

The first regular-season home game for the San Francisco 49ers in their new home, Levi’s Stadium, produced more Wi-Fi traffic and far more actual fan-to-network connections than the most-recent Super Bowl, according to statistics from the Niners’ tech team.

Dan Williams, vice president of technology for the 49ers, said the Levi’s Stadium Wi-Fi network carried 3.3 Terabytes of data during Sunday night’s game between the Niners and the Chicago Bears, topping the 3.2 TB mark reported from Super Bowl XLVIII in February. According to Williams, out of the 70,799 that filled Levi’s Stadium Sunday, more than 30,000 fans connected to the Wi-Fi network at some point, with peak usage of 19,000 fans all connecting at one time occurring just before the 5:30 p.m. local time kickoff. According to the Super Bowl stats, the peak number of fans on Wi-Fi at that game at MetLife Stadium in New Jersey was 13,500.

“We’re pretty excited by Sunday,” said Williams, who said that the Wi-Fi network stood up well even under peak data transfer rates of 3.1 Gbps right before kickoff, and another 2.6 Gbps peak around 7:30 p.m. Around the peaks, network traffic stayed “well over 1 gig per second for three and a half hours,” Williams said.

North scoreboard screen at Levi's Stadium.

North scoreboard screen at Levi’s Stadium.

During the Niners’ first preseason game against the Denver Broncos, the Levi’s Stadium Wi-Fi network carried 2.13 TB of data, and during the Aug. 24 preseason game against San Diego there was another 1.96 TB of Wi-Fi data. The figures do not include any reporting from the stadium’s DAS network, which carries cellular traffic from AT&T, Verizon Wireless and T-Mobile customer. If preseason games are any indication, Williams expects to see numbers in the terabyte range for DAS traffic as well.

The Wi-Fi numbers from Sunday showed that fans quickly figured out a name change in the network name (or SSID). During preseason games, the Wi-Fi network was identified as “Levi’s Stadium” in a device list of available networks; on Sunday the free stadium network used the name “xfinitywifi,” reflecting the brand of Wi-Fi sponsor Comcast. Some fans might have been confused since the “xfinitywifi” SSID is the same one used by Comcast for its public Wi-Fi networks.

“Some folks may have been scratching their heads,” said Williams. “We changed the name last Monday before the opener.”

Replay app gets 7,800 views

As previously reported by MSR, the instant replay feature of the Levi’s Stadium app had its debut Sunday, and according to Williams fans watched 7,800 replays via the app. The top replay view was of the early touchdown pass from Niners QB Colin Kaepernick to Michael Crabtree, which Williams said was viewed more than 1,000 times.

Fans on southwest concourse take photos of live action.

Fans on southwest concourse take photos of live action.

As MSR reported, the replay feature was somewhat limited in functionality, not working at all until late in the first half and then only offering the last two plays plus some scoring highlights for viewing. Previously, team executives had said the replay feature would offer multiple camera angles and multiple replay reviews all at the same time. According to Williams, more features will be added to the replay function in the near future.

“It’s not the finished product, by any means,” Williams said. “You’ll see some more polish on it.”

The most-used feature in the stadium app, Williams said, continues to be the food and beverage features, which allow fans to either purchase concessions for express line pickup, or to have their orders delivered to their seats. Williams said the Niners delivered 2,100 food orders to fans Sunday, the most for the Niners so far.

Perhaps the best news for Williams was the lack of complaints about the wireless network, which the team had asked fans to tweet about if they were experiencing problems. Though some fans with older devices that only work on the 2.4 GHz wireless bands might not see the same speeds as those with newer devices (which use the more roomy 5 GHz bands), Williams said his team only got a couple complaints about network issues, and one of those was solved before they could respond.

“Overall it just was a really good [wireless] experience,” said Williams, who always ends by noting that networks are never completely finished products. But with its Super Bowl-beating performance Sunday, the Levi’s Stadium network appears in midseason form.

“I think we’re close,” Williams said.

View from the north porch

View from the north porch

Holy Terabyte! First football crowd at Levi’s Stadium uses 2.13 TB of Wi-Fi traffic, with nearly 25K fans on Wi-Fi at once

Levi's Stadium from Section 244. All photos: Paul Kapustka, Mobile Sports Report

Levi’s Stadium from Section 244. All photos: Paul Kapustka, Mobile Sports Report

All those predictions about Silicon Valley people using a stadium network more than other fans? It looks like they’re true.

The network numbers are in for the first football game at Levi’s Stadium, and they are pretty amazing: According to Dan Williams, the vice president of technology for the San Francisco 49ers, the Levi’s Stadium Wi-Fi network carried 2.13 terabytes of data during last Sunday’s preseason game, with a peak of 24,775 fans on the Wi-Fi network at the same time. Those numbers are comparable to the latest Super Bowl at MetLife Stadium in New Jersey, where according to AT&T and Verizon there was approximately 2.5 TB of data used on wireless networks.

The kicker to the Niners’ stats — they do NOT include any traffic figures from the Levi’s Stadium DAS, the distributed antenna system that is meant to provide enhanced cellular coverage in the stadium. What follows is an email Q-and-A with Williams, who kindly answered our extensive list of questions. The real question is, if Niners fans are hitting terabyte levels during preseason games, what’s going to happen when the season starts for real? And the instant replay function in the team app is live? Read on for some great insight from Williams. Additional editor’s note: The companies talked about include Aruba Networks, the provider of Wi-Fi gear; Brocade, provider of back-end networking gear and integration; DAS Group Professionals, the integrator and deployment team behind the DAS (the network of small antennas that improve in-building cellular coverage).

Mobile Sports Report: what was the peak number for simultaneous Wi-Fi connections? The average?
Dan Williams: We peaked at 24,775 (roughly 38% of attendance) concurrent connections with an average of 16,862 (roughly 25% of attendance).

MSR: When did connections spike? When did they start and then tail off?
Williams: We had two spikes, 1:02 p.m. [editor's note: kickoff was 1 p.m.] with a system wide peak of 2.3Gbps and then again at 1:53 p.m. with 1.7Gbps. We averaged more than 1Gbps for more than two hours.

Niners VP of technology Dan Williams attempts to fix my Droid 4 Wi-Fi issues (while trying not to laugh at the fact that I actually have and use a Droid 4)

Niners VP of technology Dan Williams attempts to fix my Droid 4 Wi-Fi issues (while trying not to laugh at the fact that I actually have and use a Droid 4)

MSR: What was the total data tonnage on the Wi-Fi network?
Williams: We offloaded 2.13 Terabytes during the event.

MSR: What were the usage patterns with the app — which feature did people use most?
Williams: We had a great deal of usage throughout Sunday. The food ordering app usage was top of the list with ticketing being a close second while video would take third from a feature standpoint.

MSR: What are the plans with the instant replay feature… when will it be live (and can you explain why it was held back)?
Williams: We felt a lot of folks were happy with the livestream, so we wanted to focus more on a couple of core features with food ordering and ticketing a bit more at this point. Replays will be available to all by the first regular season home game.

MSR: Can you explain exactly how the location feature works… does it require Bluetooth to be on?
Williams: The location service is mainly built around low-energy Bluetooth, BLE. We have a number of beacons placed throughout the open areas and points-of-interest which allow the app to identify your location through proximity. Aruba helped us build this as well. GPS is also used but the primary resource is Bluetooth. The app prompts users to enable Bluetooth to provide improved location awareness.

One of the big screens in Levi's Stadium.

One of the big screens in Levi’s Stadium.

MSR: Can you provide any stats on the DAS performance?
Williams: The DAS held up really well. Like WiFi, we found some areas that need tuning. Unlike WiFi, the carriers protect a lot of their specific data but they have told us they are very happy with the system DGP helped us with. It is important to note our DAS and WiFi have been built to compliment each other and I think between Aruba and DGP, we did that very well. Most come here looking to connect to WiFi but our story internally has been we are going to have an awesome connectivity play regardless of medium.

MSR: Could you guys see any [more] of the 2.4 GHz issues like the one I had?
Williams: As you know, 2.4GHz is limited with non-overlapping channels so we suspect a number of legacy devices may have some problems. That said, we had a ratio of 2:1 with respects to 5GHz to 2.4GHz [usage] which shows a good deal of 2.4GHz usage. We know we still have some optimizations to do in the upper bowl and upper concourse while we continue to fine-tune the main bowl and concourse as we noticed our cell edge was weaker than expected when the stands were full. Our Aruba team did a great job capturing real-time data during the event as there is really no other way to test this stuff without a full venue. We will make some tweaks and continue to learn more from every event we host. Between Aruba, Brocade, and the 49er tech staff, we are not resting on our laurels. We know there is more to do.

AT&T sees massive traffic on new Kentucky Derby DAS deployment from Mobilitie

Churchill DownsWe’ve got a more in-depth review of the wireless experience at the Kentucky Derby on the way, but we did want to share with our readers the somewhat amazing stats from the AT&T wireless network over the weekend of racing at Churchill Downs. With a new DAS from Mobilitie in place, AT&T said it saw a total of 2 Terabytes of traffic over the weekend and a stunning total of 180 Gigabytes of traffic during the peak hour of 5-6 p.m. EDT, just before the 140th Kentucky Derby race went off on May 3.

According to a press release, that peak hour of wireless traffic was the most ever for AT&T at any in-venue event, including the various Super Bowls. What’s even more impressive is that the new DAS also hosted traffic from Verizon Wireless, which did not provide any statistics from the event; however, it’s a good guess that Verizon customers among the 260,000 attendees during race weekend were doing the same things AT&T customers were, taking pictures and sending messages from one of the pure “bucket list” events in sports.

Like we said, more details on the network deployment and challenges at Churchill Downs, in our next Stadium Tech Report. Stay tuned!

NFL Mobile users watched 10.3 million video streams during Super Bowl week

Remember the Football on your phone video from last August? Pretty funny, right? Imagine, people wanting to watch football on their phones, no matter where they go. Well last week a whole bunch of them did just that. According to the NFL, more than 10 million video streams were watched on the Verizon NFL Mobile platform during Super Bowl week, a jump of more than 400 percent from the year before.

Football on your phone? You bet!

For some reason we can’t get the league’s media arm or Verizon Wireless to provide the one stat we really want — how many people watched live Super Bowl action on a smartphone? Verizon at least is consistent — they have never provided any kind of statistic on NFL Mobile usage. Today the NFL Media folks issued a press release with all kinds of “record” numbers (we will post the whole thing below if you want to dig through it), but no discrete number for the game itself. We’ll get to the reason for that in a bit. But for the overall stats, we can sum up the numbers quickly: If there was football programming available last week, a lot of people watched it. And if it was available online or to mobile devices, so much the better.

What really drove traffic across all NFL Media properties this year was the NFL Mobile package. Remember, this year the “NFL Mobile from Verizon” app was actually available for smartphones from any carrier; the catch was, you could only get live game action if you were a Verizon subscriber and paid $5 more a month. Everyone else, including Verizon customers, could see highlights and NFL Network video content, like features and reports from Super Bowl week. The opening up of the app is probably the biggest reason why unique users of NFL Mobile properties during Super Bowl week increased 88 percent this year versus last, 11.2 million users compared to 6.0 million, according to NFL statistics.

Here’s where the stats get interesting: While it’s impressive that video streams across all NFL Media properties during Super Bowl week increased 56 percent this year compared to last, 34.0 million to 21.8 million, what’s really mind-boggling is that 10.3 million of those streams this year were consumed via the NFL Mobile app, a 416 percent increase over last year.

To repeat: Almost ONE-THIRD OF ALL NFL MEDIA VIDEO STREAMS were watched last week… on a phone.

One possible reason why neither the league nor Verizon wants to release actual game-day live action viewing numbers for NFL Mobile is that they may not be that impressive. Remember, only Verizon customers who ponied up the $5 extra “premium” fee could have watched the Fox simulcast on their phones, so it’s a smaller subset to begin with. And really, for the big game itself, most likely you were on a couch watching a big screen. (We here at MSR HQ did find the NFL Mobile live feed effective for when we had to roam into the kitchen for more snacks, or for other “breaks” necessary during the game. But we didn’t watch more than a few minutes of the game on the phone.) Plus, the Fox stream was available to tablets using its app or for PCs or laptops watching online, so that probably took away some potential phone-watchers of live game action.

During non-game times, however, smartphones appear to be leading the mobile video explosion. Even though tablets seem to make more sense for watching sports while mobile, it’s pretty clear that people are watching a lot of NFL video on the thing that never leaves their pocket or purse — their phone. Are other sports taking note? And now do you know why the NFL is pushing toward NFL Now? Stay tuned. And keep your phone and checkbook handy.

(full press release content below)

COVERAGE OF SUPER BOWL XLVIII SETS VIEWERSHIP & TRAFFIC RECORDS ACROSS NFL MEDIA

Wall-to-wall coverage of Super Bowl XLVIII produces double-digit spikes for NFL Network, NFL.com, and NFL Mobile

NFL Media’s 11th year covering the Super Bowl produced record-breaking results across all platforms.

NFL NETWORK

Providing expert analysis, the latest news and reports, special guest appearances, and matchup-related programming leading up to Super Bowl XLVIII, NFL Network served up 140 total hours – including a record 86 live hours – of programming from 12 sets in eight locations across New York City and New Jersey, utilizing a roster of 40 on-air personalities with a combined 17 Super Bowl rings.

For Super Bowl week*, NFL Network averaged 175,000 viewers in Total Day and 226,000 viewers in Primetime – up +20% and +31%, respectively, compared to last year making this the most-watched Super Bowl week ever on NFL Network.

With more than 41 hours of Super Bowl coverage, SUPER BOWL LIVE averaged 141,000 viewers and up +60% over last year’s average audience (88,000 viewers).

On Super Bowl Sunday, NFL GAMEDAY MORNING averaged 657,000 viewers – up +10% over last year’s telecast (596,000 viewers), making this the most-watched NFL GAMEDAY MORNING ever.

NFL Network’s postgame audience peaked from 10:30PM-11:00PM with an average of 1.14 million viewers, up + 25% over last year’s highest postgame peak (906,000 viewers)

NFL Network averaged 313,000 viewers in Primetime on Super Bowl Sunday – up +52% compared to last year’s performance.

For the entire 2013 postseason, NFL Network averaged 148,000 viewers in Total Day – up +13% compared to last year’s performance (131,000 viewers) making this the most-watched postseason ever on NFL Network.

NFL DIGITAL MEDIA

Across all internet-connected devices, including PC’s, tablets, and smartphones, the official digital properties of the National Football League delivered original video programming, the latest news and information, as well as unprecedented access to players, celebrities and musicians. NFL Digital Media’s offerings included NFL Mobile from Verizon which provided fans access to live, streaming video of Super Bowl XLVIII and NFL Network, including exclusive Super Bowl content and commercials.

For Super Bowl week, visits to NFL Digital Media properties increased 24% versus last year, bolstered by a 149% increase in visits to NFL Mobile.

Unique users of NFL Digital Media properties rose 22% during the week versus last year (27.8M vs. 22.8M), driven by 88% growth in NFL Mobile Properties (11.2M vs. 6.0M).

Unique users and visits to NFL Digital Media properties were driven by an array of content around Super Bowl XLVII. The two highest performing long-form NFL Digital Media features were:

Ø Judy Battista’s piece on Super Bowl XLVIII’s impact on Peyton Manning’s legacy

Ø Mike Silver’s story on the Seahawks dominant defense being fueled by the tight bond amongst the ‘Legion of Boom’

Video Streams during Super Bowl week across all NFL Digital Media properties increased 56% versus last year (34.0M vs. 21.8M).

Video consumption to NFL Mobile from Verizon reached 10.3M streams for a 416% increase over last year’s Super Bowl Week.

The highest performing video of the week featured NFL Network analysts making predictions for Super Bowl XLVIII. The video was viewed more than 800k times.

Seen in 195 countries, NFL Game Pass consumption grew 49% year-over-year across all devices and 57% on desktop.

*Super Bowl Week is defined as January 27-February 2, 2014

– end press release —

And yes, we know the Football On Your Phone video was a promo for DirecTV’s Sunday Ticket, which is neither here nor there when it comes to Verizon NFL Mobile numbers. But the video is funny enough to watch again: