December 19, 2014

Stadium Tech Report: Arizona Cardinals get stadium ready for Super Bowl with Wi-Fi upgrade

University of Phoenix Stadium. Credit all photos: Arizona Cardinals.

University of Phoenix Stadium. Credit all photos: Arizona Cardinals. (click on photos for larger image)

As they get set to host their second Super Bowl this February, the IT team at the University of Phoenix Stadium in Glendale, Ariz., knows now what they didn’t know then: The big game requires a big wireless network. Bigger than you think.

“It’s funny to look back now on that first Super Bowl,” said Mark Feller, vice president of information technology for the Arizona Cardinals, speaking of Roman numeral game XLII, held in the barely 2-year-old facility on Feb. 3, 2008. With a couple Fiesta Bowls and one BCS championship game (2007) under his belt in a facility that opened with Wi-Fi and DAS, Feller said he and his team “thought we had a good handle” on what kind of network was required for a Super Bowl crowd.

The NFL, he said, begged to differ. Those college games might have been big, but the Super Bowl was bigger.

“We had the Fiesta Bowl that year at night, and when the game was over there were people from the NFL there wanting to know when they could set up,” said Feller in a recent phone interview. “This year, we’re much better prepared. We know what the water temperature is this time.”

Rip and replace, with more and better gear

Wi-Fi railing antennas

Wi-Fi railing antennas

For Super Bowl XLIX, scheduled to take place in Glendale on Feb. 1, 2015, Feller and his team have not just tuned up their network — they have done a full rip and replace of the Wi-Fi system, installing new Cisco gear from back end to front, in order to support a wireless game-day demand that is historically second to none. Integrator CDW has led the Wi-Fi effort and Daktronics and ProSound did the installation of new video screens, and neutral host Crown Castle has overseen a revamp of the DAS system, again with more antennas added to bolster coverage. In all, there has been more than $8 million in wireless improvements before this year, Feller said, as well as another $10 million for two new video boards that are each three times larger than what was there before.

“The last three or four years there have been things we knew we needed to improve [before the Super Bowl],” Feller said. After extensive work with the NFL’s technical team — this time well before the Fiesta Bowl — Feller oversaw a “top to bottom” refurbishment that included replacing core Cisco networking gear with newer gear, and new and more Wi-Fi access points that now total somewhere north of the 750 mark, with some more to be added before the big game. The new network, which was in place for the start of the current NFL season, has undergone testing by CDW at each home game, Feller said. CDW also plans to expand the network outside the stadium before the Super Bowl, in part to handle the extra events that take place not just on game day but in the days leading up to the game.

“The plan is to install more [coverage] outside, in the plaza areas,” Feller said.

When it opened in 2006, the $455 million University of Phoenix Stadium was one of the first with full-bowl Wi-Fi, using Cisco gear from the inside out. “Cisco was in here before they called it [their solution] ‘connected stadium’,” Feller said. From core network switches to firewalls to edge switches, this year there is all new Cisco gear in the venue, as well as new 3700 series APs, with panel antennas and antennas in handrails.

“Handrail [antennas] are sometimes a bit of a challenge, because you need to drill through concrete that’s 40 feet up in the air, behind another ceiling,” said Feller, describing one particular design challenge. Another one was mounting antennas on drop rods from the catwalks below the stadium’s retractable roof, to serve the upper-area seating. There are also some new Wi-Fi APs on the front row of the seating bowl, pointing up into the crowd.

“It was a fun project,” Feller said.

Stadium with roof open

Stadium with roof open


All on board for the DAS

The upgrade for the stadium’s DAS, led by Crown Castle, was just finished a few weeks ago, Feller said, and included more coverage outside the stadium as well, with antennas placed on light poles and on the stadium’s shell.

“Crown Castle did a great job of managing the carriers” on what is a 48-sector DAS, Feller said. “It [the upgrade] really required a lot of creative thinking from their engineers.”

Since the stadium was originally designed with wireless in mind, Feller and his team didn’t need to build new head end room for the DAS upgrades. “But I wouldn’t say we have plenty of space left,” he added. “We’ve got a lot of new equipment.”

Though all the major carriers are expected to be on the DAS by the big game, league partner Verizon Wireless should have some special projects up its sleeve for the big game, including another demonstration of its LTE Broadcast technology, which optimizes things like live video over LTE cellular links.

New Cardinals app a preview of Super Bowl version?

The Cardinals also had a new version of the game-day team app for this season, built by stadium-app leader YinzCam. According to Feller the new app supports three different live video feeds, as well as instant replays.

Wi-Fi antenna on railing

Wi-Fi antenna on railing

“It’s really cool to have that ability to watch things like a touchdown pass at the end of the game,” Feller said. And while no details have yet been revealed, in an interview with NFL CIO Michelle McKenna-Doyle earlier this year MSR learned that the league and YinzCam are working on a Super Bowl app with its own new bells and whistles. (Stay tuned for more info on the Super Bowl app.)

In addition to two more regular-season home games in December, the University of Phoenix Stadium will have at least a couple more dry runs to help test the network, during the Dec. 31 Fiesta Bowl and during the NFL’s Pro Bowl, scheduled for Jan. 25. And though the Cardnials lost to the Atlanta Falcons Sunday, at 9-3 they are still tied with the Green Bay Packers for the best record in the NFC, something that has the Phoenix faithful optimistic about the postseason.

“We’re going to get some more test runs, on New Year’s Eve and during the Pro Bowl,” Feller said. “And maybe some home playoff games as well!”

(more photos below)

Wi-Fi antenna in roof rafters

Wi-Fi antenna in roof rafters

More antennas in rafters

More antennas in rafters

Wi-Fin antenna under overhang

Wi-Fi antenna under overhang

Niners’ home opener tops Super Bowl for Wi-Fi data traffic with 3.3 Terabytes

Fans take pictures of opening kickoff from southwest concourse. Credit, all photos: Paul Kapustka, MSR

Fans take pictures of opening kickoff from southwest concourse. Credit, all photos: Paul Kapustka, MSR

The first regular-season home game for the San Francisco 49ers in their new home, Levi’s Stadium, produced more Wi-Fi traffic and far more actual fan-to-network connections than the most-recent Super Bowl, according to statistics from the Niners’ tech team.

Dan Williams, vice president of technology for the 49ers, said the Levi’s Stadium Wi-Fi network carried 3.3 Terabytes of data during Sunday night’s game between the Niners and the Chicago Bears, topping the 3.2 TB mark reported from Super Bowl XLVIII in February. According to Williams, out of the 70,799 that filled Levi’s Stadium Sunday, more than 30,000 fans connected to the Wi-Fi network at some point, with peak usage of 19,000 fans all connecting at one time occurring just before the 5:30 p.m. local time kickoff. According to the Super Bowl stats, the peak number of fans on Wi-Fi at that game at MetLife Stadium in New Jersey was 13,500.

“We’re pretty excited by Sunday,” said Williams, who said that the Wi-Fi network stood up well even under peak data transfer rates of 3.1 Gbps right before kickoff, and another 2.6 Gbps peak around 7:30 p.m. Around the peaks, network traffic stayed “well over 1 gig per second for three and a half hours,” Williams said.

North scoreboard screen at Levi's Stadium.

North scoreboard screen at Levi’s Stadium.

During the Niners’ first preseason game against the Denver Broncos, the Levi’s Stadium Wi-Fi network carried 2.13 TB of data, and during the Aug. 24 preseason game against San Diego there was another 1.96 TB of Wi-Fi data. The figures do not include any reporting from the stadium’s DAS network, which carries cellular traffic from AT&T, Verizon Wireless and T-Mobile customer. If preseason games are any indication, Williams expects to see numbers in the terabyte range for DAS traffic as well.

The Wi-Fi numbers from Sunday showed that fans quickly figured out a name change in the network name (or SSID). During preseason games, the Wi-Fi network was identified as “Levi’s Stadium” in a device list of available networks; on Sunday the free stadium network used the name “xfinitywifi,” reflecting the brand of Wi-Fi sponsor Comcast. Some fans might have been confused since the “xfinitywifi” SSID is the same one used by Comcast for its public Wi-Fi networks.

“Some folks may have been scratching their heads,” said Williams. “We changed the name last Monday before the opener.”

Replay app gets 7,800 views

As previously reported by MSR, the instant replay feature of the Levi’s Stadium app had its debut Sunday, and according to Williams fans watched 7,800 replays via the app. The top replay view was of the early touchdown pass from Niners QB Colin Kaepernick to Michael Crabtree, which Williams said was viewed more than 1,000 times.

Fans on southwest concourse take photos of live action.

Fans on southwest concourse take photos of live action.

As MSR reported, the replay feature was somewhat limited in functionality, not working at all until late in the first half and then only offering the last two plays plus some scoring highlights for viewing. Previously, team executives had said the replay feature would offer multiple camera angles and multiple replay reviews all at the same time. According to Williams, more features will be added to the replay function in the near future.

“It’s not the finished product, by any means,” Williams said. “You’ll see some more polish on it.”

The most-used feature in the stadium app, Williams said, continues to be the food and beverage features, which allow fans to either purchase concessions for express line pickup, or to have their orders delivered to their seats. Williams said the Niners delivered 2,100 food orders to fans Sunday, the most for the Niners so far.

Perhaps the best news for Williams was the lack of complaints about the wireless network, which the team had asked fans to tweet about if they were experiencing problems. Though some fans with older devices that only work on the 2.4 GHz wireless bands might not see the same speeds as those with newer devices (which use the more roomy 5 GHz bands), Williams said his team only got a couple complaints about network issues, and one of those was solved before they could respond.

“Overall it just was a really good [wireless] experience,” said Williams, who always ends by noting that networks are never completely finished products. But with its Super Bowl-beating performance Sunday, the Levi’s Stadium network appears in midseason form.

“I think we’re close,” Williams said.

View from the north porch

View from the north porch

NFL Mobile users watched 10.3 million video streams during Super Bowl week

Remember the Football on your phone video from last August? Pretty funny, right? Imagine, people wanting to watch football on their phones, no matter where they go. Well last week a whole bunch of them did just that. According to the NFL, more than 10 million video streams were watched on the Verizon NFL Mobile platform during Super Bowl week, a jump of more than 400 percent from the year before.

Football on your phone? You bet!

For some reason we can’t get the league’s media arm or Verizon Wireless to provide the one stat we really want — how many people watched live Super Bowl action on a smartphone? Verizon at least is consistent — they have never provided any kind of statistic on NFL Mobile usage. Today the NFL Media folks issued a press release with all kinds of “record” numbers (we will post the whole thing below if you want to dig through it), but no discrete number for the game itself. We’ll get to the reason for that in a bit. But for the overall stats, we can sum up the numbers quickly: If there was football programming available last week, a lot of people watched it. And if it was available online or to mobile devices, so much the better.

What really drove traffic across all NFL Media properties this year was the NFL Mobile package. Remember, this year the “NFL Mobile from Verizon” app was actually available for smartphones from any carrier; the catch was, you could only get live game action if you were a Verizon subscriber and paid $5 more a month. Everyone else, including Verizon customers, could see highlights and NFL Network video content, like features and reports from Super Bowl week. The opening up of the app is probably the biggest reason why unique users of NFL Mobile properties during Super Bowl week increased 88 percent this year versus last, 11.2 million users compared to 6.0 million, according to NFL statistics.

Here’s where the stats get interesting: While it’s impressive that video streams across all NFL Media properties during Super Bowl week increased 56 percent this year compared to last, 34.0 million to 21.8 million, what’s really mind-boggling is that 10.3 million of those streams this year were consumed via the NFL Mobile app, a 416 percent increase over last year.

To repeat: Almost ONE-THIRD OF ALL NFL MEDIA VIDEO STREAMS were watched last week… on a phone.

One possible reason why neither the league nor Verizon wants to release actual game-day live action viewing numbers for NFL Mobile is that they may not be that impressive. Remember, only Verizon customers who ponied up the $5 extra “premium” fee could have watched the Fox simulcast on their phones, so it’s a smaller subset to begin with. And really, for the big game itself, most likely you were on a couch watching a big screen. (We here at MSR HQ did find the NFL Mobile live feed effective for when we had to roam into the kitchen for more snacks, or for other “breaks” necessary during the game. But we didn’t watch more than a few minutes of the game on the phone.) Plus, the Fox stream was available to tablets using its app or for PCs or laptops watching online, so that probably took away some potential phone-watchers of live game action.

During non-game times, however, smartphones appear to be leading the mobile video explosion. Even though tablets seem to make more sense for watching sports while mobile, it’s pretty clear that people are watching a lot of NFL video on the thing that never leaves their pocket or purse — their phone. Are other sports taking note? And now do you know why the NFL is pushing toward NFL Now? Stay tuned. And keep your phone and checkbook handy.

(full press release content below)

COVERAGE OF SUPER BOWL XLVIII SETS VIEWERSHIP & TRAFFIC RECORDS ACROSS NFL MEDIA

Wall-to-wall coverage of Super Bowl XLVIII produces double-digit spikes for NFL Network, NFL.com, and NFL Mobile

NFL Media’s 11th year covering the Super Bowl produced record-breaking results across all platforms.

NFL NETWORK

Providing expert analysis, the latest news and reports, special guest appearances, and matchup-related programming leading up to Super Bowl XLVIII, NFL Network served up 140 total hours – including a record 86 live hours – of programming from 12 sets in eight locations across New York City and New Jersey, utilizing a roster of 40 on-air personalities with a combined 17 Super Bowl rings.

For Super Bowl week*, NFL Network averaged 175,000 viewers in Total Day and 226,000 viewers in Primetime – up +20% and +31%, respectively, compared to last year making this the most-watched Super Bowl week ever on NFL Network.

With more than 41 hours of Super Bowl coverage, SUPER BOWL LIVE averaged 141,000 viewers and up +60% over last year’s average audience (88,000 viewers).

On Super Bowl Sunday, NFL GAMEDAY MORNING averaged 657,000 viewers – up +10% over last year’s telecast (596,000 viewers), making this the most-watched NFL GAMEDAY MORNING ever.

NFL Network’s postgame audience peaked from 10:30PM-11:00PM with an average of 1.14 million viewers, up + 25% over last year’s highest postgame peak (906,000 viewers)

NFL Network averaged 313,000 viewers in Primetime on Super Bowl Sunday – up +52% compared to last year’s performance.

For the entire 2013 postseason, NFL Network averaged 148,000 viewers in Total Day – up +13% compared to last year’s performance (131,000 viewers) making this the most-watched postseason ever on NFL Network.

NFL DIGITAL MEDIA

Across all internet-connected devices, including PC’s, tablets, and smartphones, the official digital properties of the National Football League delivered original video programming, the latest news and information, as well as unprecedented access to players, celebrities and musicians. NFL Digital Media’s offerings included NFL Mobile from Verizon which provided fans access to live, streaming video of Super Bowl XLVIII and NFL Network, including exclusive Super Bowl content and commercials.

For Super Bowl week, visits to NFL Digital Media properties increased 24% versus last year, bolstered by a 149% increase in visits to NFL Mobile.

Unique users of NFL Digital Media properties rose 22% during the week versus last year (27.8M vs. 22.8M), driven by 88% growth in NFL Mobile Properties (11.2M vs. 6.0M).

Unique users and visits to NFL Digital Media properties were driven by an array of content around Super Bowl XLVII. The two highest performing long-form NFL Digital Media features were:

Ø Judy Battista’s piece on Super Bowl XLVIII’s impact on Peyton Manning’s legacy

Ø Mike Silver’s story on the Seahawks dominant defense being fueled by the tight bond amongst the ‘Legion of Boom’

Video Streams during Super Bowl week across all NFL Digital Media properties increased 56% versus last year (34.0M vs. 21.8M).

Video consumption to NFL Mobile from Verizon reached 10.3M streams for a 416% increase over last year’s Super Bowl Week.

The highest performing video of the week featured NFL Network analysts making predictions for Super Bowl XLVIII. The video was viewed more than 800k times.

Seen in 195 countries, NFL Game Pass consumption grew 49% year-over-year across all devices and 57% on desktop.

*Super Bowl Week is defined as January 27-February 2, 2014

– end press release —

And yes, we know the Football On Your Phone video was a promo for DirecTV’s Sunday Ticket, which is neither here nor there when it comes to Verizon NFL Mobile numbers. But the video is funny enough to watch again:

Analytics and big data of prime interest to stadium networks

Editor’s note: Please welcome guest poster R. Scott Raynovich, editor of the Rayno Report, which has just launched a free in-depth report on networking, analytics and big data. Read on for Rayno’s take on the value of intelligent analytics, and take advantage of his new free report. — PK

By R. Scott Raynovich, Rayno Report

You can’t throw a stone on the Internet without hitting some big data. But what’s interesting about big data, and its cousin, analytics, is that the technology has hundreds of applications in hundreds of markets, from digital marketing to security. And there’s a huge, untapped market that is unfolding in networking, including in-stadium and on-campus networks.

People are talking about Software-Defined Networking (SDN), and of course, everybody’s favorite, Network Function Virtualization. The discussion around many of these technologies is about commoditization and reduced cost of hardware. But that’s actually not where the value is created: The value is going to be created in opening up of the system and allowing for a lot more exchange of data and information and intelligent analytics.

Extreme Networks has made a smart move, I believe, by focusing on this with its recent partnership with the NFL. Extreme talks about the business use cases for analytics, such as analyzing what your customers are doing, monitoring performance, and overall, providing a better product. If you look at the logic of it, it’s this that networking vendors have as much data and information as anybody, because it passes through their pipes.

Analytics and the data center: a natural marriage

I recently completed an early study in this, which is included in a new report: Analytics and the Software-Defined Data Center. My research indicates that a new crop of technology is emerging that could tie together storage, computing, and networking, by providing cutting-edge measurement and analysis that allows managers to gain more insight into what’s going on in data centers and networks, and eventually automate their management.

See Extreme’s fancy graphic below on the Super Bowl. This just scratches the surface. What it shows, if anything, is how much information and analysis can be extracted from just about any network application. Imagine the data flowing through a ecommerce data center, or a digital media service. Businesses now have instant access to everything everybody is doing. Or how do some of the largest Websites in the world monitoring their networks for security breaches?

The answer is mostly in measurement and analytics technologies.

What’s interesting is that we’ve just started using this concept to make our networks and data-centers systems and more self-reliant. There is a long way to go, and you will start hearing more about this in the coming months.

Top 10 Benefits of Purview Application Awareness at Super Bowl XLVIII

(Source: Extreme Networks)

Friday Grab Bag: Super Bowl prop bets, anyone?

Sure you have purchased a square or two in the office pool and made a friendly wager on the outcome of the game, but that is scratching the surface. Did you know you can bet on how long the National Anthem will be or if the singer will wear gloves? Will the game go overtime?

A look at which are good and bad can be found in a number of places but I liked the ones published in SportsOnEarth and one that was in Football Outsiders as they give some context to what you should and should not do.

Is the Super Bowl a boon on the local economy?
Every year we see a number of time-honored myths rolled out about the Super Bowl. Top day for avocado consumption. No. Top day for spousal abuse. No. That it is a financial boon to the host city/area. Maybe no as well.

A piece in the New York Times points out that since past predictions of prosperity by the NFL were destroyed once the methodology was public the same might be true now. The current estimate, between $550 million and $600 was determined in secret, they will not say who did it or how they arrived at that number. However advertisers are flooding the host city so that they can be seen prior to the event so maybe they know something they are just not telling us.

Winter Olympics broadcast schedule
In case you actually can watch the games from the comfort of your sofa or a cozy corner pub here is a complete viewers’ guide to the events, when they will be broadcast. It should be noted by figure skating fans that the first events take place prior to the opening ceremony.

For a breakdown on what will be interesting as well as the challenges that the broadcaster NBC could face look over to Sports Illustrated and a piece by Richard Deitsch. Will the network let possible bad news from the surrounding area taint its broadcasts?

Why no respect for American soccer players?
With the World Cup now looming on the sports horizon ESPNFC did a piece on why it seems that European based leagues do not value American players and why increasingly that feeling is being reciprocated.

The nice thing about the piece is that it does not jump to a conclusion but points out how different development systems, short earning window and other factors all play in the decisions that players make in where they want to play.

Another black eye for Sochi?
The upcoming Winter Olympics in Russia has received a series of bad news with intolerance, corruption and potential terrorism issues rising to the forefront. Now a recent piece from ABC shows that there could be another piece of bad news on the horizon.

They are saying that one of the key figures in helping Russia win the games over Austria and South Korea was a man named Gafur Rakhimov. The trouble with Rakhimov is that he is considered by U.S. authorities to be one of the top four or five people in the heroin trade and is under indictment in Uzbekistan.

How to watch the Super Bowl online, or on your phone

Just in case your TV goes on the blink this weekend, don’t worry, you can still watch Sunday’s Super Bowl XLVIII between the Denver Broncos and the Seattle Seahawks — either via an online stream, or via your smartphone if you are a Verizon customer.

SUPER BOWL XLVIII KICKOFF: 6:30 p.m. ET

TV: FOX

WATCH THE SUPER BOWL LIVE ONLINE HERE

Fox, which is showing the game on regular TV, will also make the broadcast available online via its Fox Sports Go app and website. Usually, you need a pay TV subscription to see the Fox feed, but it will be free to all viewers on Super Bowl Sunday. The same feed will also be available at NFL.com and at SuperBowl.com, just in case you need an alterate website address. If you want to watch on an iPad, you will want the Fox Sports Go app.

Smartphone viewing via Verizon NFL Mobile

Remember, you can’t watch the game on a smartphone via the Fox app. That’s because Verizon Wireless has the rights to live action on smartphones, via its NFL Mobile app. To view the game live, you must A) be a Verizon customer, B) have the NFL Mobile app installed, and C) have paid the $5 per month premium NFL Mobile fee.

HERE IS THE VERIZON NFL MOBILE INFO PAGE

Remember, both the Fox website feed and the NFL Mobile app feed will be significantly behind the live TV broadcast, anywhere from 20 seconds to more than a minute. And, no, you won’t be able to watch the live feed if you are at the game.