Analysis: The year of the big stadium Wi-Fi upgrade

Carolina Panthers director of IT James Hammond shows off a new under-seat Wi-Fi AP at Bank of America Stadium. Credit: Carolina Panthers

Carolina Panthers director of IT James Hammond shows off a new under-seat Wi-Fi AP at Bank of America Stadium. Credit: Carolina Panthers

Even in the midst of several brand-new stadium debuts and the future-proofed wireless networks inside them, there is a separate, yet distinct trend emerging in the big-stadium, wireless connectivity world: Call it the year of the big upgrade.

Our profile in our latest STADIUM TECH REPORT of Bank of America Stadium in Charlotte, N.C., is a case in point: Thanks to the never-ending demand for more connectivity for fans, stadiums that deployed networks just a few years ago are now finding that those old systems already need upgrades or replacements, typically at a much higher cost than the original network. In addition to BofA Stadium, the New England Patriots’ home, Gillette Stadium, also got a Wi-Fi makeover this past summer, going from about 400 Wi-Fi APs to well over a thousand, with most of the new ones deployed under seats.

According to Fred Kirsch, who oversees the Gillette Stadium network, some of the under-seat placements there were especially tricky, since granite underneath the stands didn’t allow for the ability to drill through the concrete. A workaround involving an above-ground enclosure was envisioned and manufactured, underlining the custom complexity of network deployment found from stadium to stadium. No two are the same, and what works at one may or may not work at another.

But what is common across all these large venues is the ever-increasing need for bandwidth, a moving target that has yet to slow down or stabilize. Last year the story that turned everyone’s head was the need by carriers to upgrade their DAS infrastructure at Levi’s Stadium ahead of Super Bowl 50 – this coming just a year after the stadium had opened for business. While the demands of a Super Bowl (especially Super Bowl 50, which set records for DAS and Wi-Fi usage) are perhaps much different than everyday events, it’s still a safe bet that for many stadiums with Wi-Fi networks – especially the early movers – 2016 has become a year of reckoning, or biting the bullet and writing more checks for more coverage, perhaps seemingly too soon after the initial rollout.

Getting ready for Super Bowl LI

In Houston, NRG Stadium finally has Wi-Fi, and not a moment too soon, with Super Bowl LI on the near horizon. Since the venue didn’t have Wi-Fi prior to this season it’s not really an upgrade but it’s hard to understate the challenge of putting in a Super Bowl-ready network in just one summer, a construction calendar shortened by the fact that integrator 5 Bars and equipment vendor Extreme Networks had to wait until after the NCAA Men’s Final Four was over to begin installing cabling and APs. At of the start of the NFL season the Wi-Fi network is already live at NRG Stadium, and is sure to go through weekly tweaks as the league marches on toward its championship game.

Gillette Stadium before the Sept. 11 game vs. the Miami Dolphins. Credit: Steve Milne, AP, via Patriots.com

Gillette Stadium before the Sept. 11 game vs. the Miami Dolphins. Credit: Steve Milne, AP, via Patriots.com

And while attention-grabbing new stadiums like US Bank Stadium in Minneapolis and Mercedes-Benz Stadium in Atlanta are planning big network capacity from the get-go, some new stadiums like T-Mobile Arena in Las Vegas have upgrade thinking planned in from the start, with the idea that the network will never really be a finished product, at least until they stop making new phones or developing new apps. Of course, that future isn’t happening anytime soon, with the Apple iPhone 7 announcement with the new double-lens camera coming in just before the start of another football season.

New phones and new apps mean more bandwidth demands, leading even those who already have stadium networks to keep wondering if what they’ve installed is enough. We suspect this may be an ongoing story line for the foreseeable future, so – stay tuned here to Mobile Sports Report for the latest success stories and lessons learned from those who have already jumped in or jumped back in to the deployment fray.

Editor’s note: This column is from our latest STADIUM TECH REPORT, which is available for free download from our site. Read about Wi-Fi deployments at Bank of America Stadium, Mercedes-Benz Stadium and more!

New Report: Carolina Panthers build new Wi-Fi and DAS; Mercedes-Benz Stadium update, and more!

Q3thumbMobile Sports Report is pleased to announce the Q3 issue of our STADIUM TECH REPORT series, the ONLY in-depth publication created specifically for the stadium technology professional and the stadium technology marketplace.

In addition to our historical in-depth profiles of successful stadium technology deployments, our Q3 issue for 2016 has additional news and analysis, including a look at Wi-Fi analytics at the Mall of America, and a story about how the Cleveland Browns found $1 million in ROI using new analytics software from YinzCam. Download your FREE copy today!

Inside the report our editorial coverage also includes:

— Bank of America Stadium profile: An in-depth look at the Carolina Panthers’ decision to bring new Wi-Fi and DAS networks in-house;
— Mercedes-Benz Stadium profile: An early look at the technology being built into the new home of the Atlanta Falcons, with an emphasis on fiber;
— T-Mobile Arena photo essay: A first look at the newest venue on the famed Las Vegas Strip;
— Avaya Stadium profile: How the stadium’s Wi-Fi network became the star of the MLS All-Star game.

We’d like to take a quick moment to thank our sponsors, which for this issue include Mobilitie, Crown Castle, SOLiD, CommScope, JMA Wireless, Corning, Samsung Business, Xirrus, Huber+Suhner, ExteNet Systems, DAS Group Professionals and Boingo Wireless. Their generous sponsorship makes it possible for us to offer this content free of charge to our readers. We’d also like to thank you for your interest and support.

Niners fans use 2.76 TB of Wi-Fi for season opener at Levi’s Stadium

Niners fans celebrate a touchdown in the season opener. Credit: LevisStadium.com.

Niners fans celebrate a touchdown in the season opener. Credit: LevisStadium.com.

Fans at the San Francisco 49ers’ home opener used 2.76 terabytes of data on the Wi-Fi network at Levi’s Stadium, according to statistics provided by the Niners’ network team.

The Sept. 12 Monday Night Football game, a 28-0 win for the Niners over the visiting Los Angeles Rams, was well below the 10.1-TB mark recorded during Super Bowl 50, held at Levi’s Stadium back in February. Still, the 2.76 TB is a healthy regular-season game mark, with 16,681 unique users of the Wi-Fi network as well as a maximum concurrent number of users of 11,987.

The Niners also added some interesting new social twists to the Levi’s Stadium experience this year, including the ability for fans to use the stadium app to respond to poll questions (like voting on the next song to be played during timeouts) posted on the stadium’s large digital displays. While it’s not known if the feature was used during the regular season opener, according to the network team 30,000 fans participated in big-board polls during the Niners’ preseason game on Aug. 26 versus the Green Bay Packers. Full stats from the Niners networking team below.

Screen Shot 2016-09-23 at 12.01.42 PM

Wi-Fi network powers rich data collection at Mall of America

Fans greet One Direction at Mall of America. Credit: Tony Nelson (click on any photo for a larger image)

Fans greet One Direction at Mall of America. Credit: Tony Nelson (click on any photo for a larger image)

If you’re shopping for mobile customer data, why not just go to the mall?

That’s what Minnesota’s Mall of America did, not by finding a service that sells such information, but by investing in a massive and complex Wi-Fi network, designed and deployed by AmpThink using Cisco gear. And the service is free for Mall visitors. While being an attractive guest feature, the service simultaneously provides the Mall with enough data to fill digital warehouses with information about what people do both online and in the real world while on the property.

According to Janette Smrcka, IT director for Mall of America, though the Mall is only at the start of its data analysis program, it is already seeing interesting results that will likely help the Mall better connect with its visitors and, in all likelihood, improve business results for Mall tenants. A recent month-long sampling of customer behavior data gave the Mall a tremendous amount of insight on how activities such as promotions and events affect visitor behavior, information the Mall wouldn’t have had without its Wi-Fi network.

“We’re just at the beginning of being able to use all of this valuable data and translate it into actionable information,” said Smrcka in an interview at Mall of America, located in Bloomington, nine miles south of Minneapolis.

Almost a Super Bowl of data every week

How much data are we talking about? In the world of stadium networks, the most recent Super Bowl at Levi’s Stadium in Santa Clara, Calif. set a single-day record with 26 terabytes of wireless data used – 15.9 TB on cellular networks and 10.1 TB on the stadium Wi-Fi. At Mall of America, from the launch of their Wi-Fi network during Thanksgiving weekend last year until May 2016, nearly 320,000 unique Mall visitors connected to the network, using a total of 486 TB of traffic – almost a Super Bowl of data per week.

Wi-Fi AP visible below Mall of America sign. Credit: Paul Kapustka / MSR

Wi-Fi AP visible below Mall of America sign. Credit: Paul Kapustka / MSR

While the Mall might not match the single-day crush of a Super Bowl, the steady stream of visitors (Mall officials estimate that the six million square-foot facility sees an average of 109,000 visitors on weekdays and 160,000 on weekends) produces some staggering numbers. According to a recent public presentation about the network, Mall of America claimed that one month of Wi-Fi usage on its network equaled a full year of Wi-Fi activity on an NBA-sized stadium network.

According to Smrcka, the Mall knew it needed Wi-Fi connectivity as a table stakes amenity, but it was mindful of fitting performance to price, while achieving return on investment justification in the process.

“We knew we needed something but the challenge was the cost,” Smrcka said. “We knew we couldn’t charge for the service.” Smrcka also said the Mall has seen other malls try and fail with initial Wi-Fi deployments due to subpar service, prompting guest disdain.

“We’ve seen some of our peers put Wi-Fi in, and not do it very well, and get lots of complaints,” Smrcka said. “It’s like airport Wi-Fi. Sure it’s there, but just try using it.”

With AmpThink, Mall of America found a partner who knew the need for high-quality deployments. The design and install company has been behind the Wi-Fi networks at several Super Bowls, as well as recent networks built in stadiums like Kyle Field at Texas A&M and US Bank Stadium in Minneapolis.

Wi-Fi 'ball' visible in middle of theme park area. Credit: Paul Kapustka / MSR

Wi-Fi ‘ball’ visible in middle of theme park area. Credit: Paul Kapustka / MSR

However, this massive Wi-Fi network, along with the unique challenges of implementing it at the Mall, also needed the elusive component of return on investment.

Not easy to build inside a mall

On a recent early morning tour of the Mall, AmpThink president Bill Anderson showed Mobile Sports Report some of the challenges inherent to one of the world’s biggest shopping venues. For starters, there was the need for custom enclosures that fit with the facility’s overall aesthetics as well as cutting through double firewalls (the real kind of firewalls, not the software kind) to keep safety codes intact.

To cover one of the Mall’s more unique spaces, the 7-acre theme park in the center of the facility, AmpThink had to design and build enclosures that look like big lollipops. This “Wi-Fi ball on a stick” design fit the park’s design aesthetics while providing coverage in and around the various rides and amusement spaces. AmpThink also figured out how to fit a Wi-Fi AP inside digital sign kiosks, so the kiosks could connect to the network and therefore the Cisco StadiumVision system for digital display management. In many places, the APs included beacons inside, setting up the network for device proximity capabilities.

From an RF perspective, Anderson said one of the toughest challenges was keeping interference to a minimum between APs on different floors of the multi-level mall. Another large challenge was simply the logistics of construction, with separate scheduling needed for the many hundreds of Mall tenants.

The mall's many levels make it a tough place to tune RF. Credit: Paul Kapustka / MSR

The mall’s many levels make it a tough place to tune RF. Credit: Paul Kapustka / MSR

“We own the space above their ceilings, and would need to get in the stores to run cable through,” said Smrcka. Coordinating construction was a challenge at times, like when teenage clerks didn’t relay scheduling messages to their store managers, further complicated by the need to have security officers present to keep an eye on inventory.

Anderson said that AmpThink deployment teams also needed to make sure they cleaned up after putting in APs, as any drywall dust found on store facades could result in complaints from store owners. Despite the extra hurdles, deployment of the network, composed of more than 600 APs, started in July of last year and launched just before Thanksgiving. Then the data started pouring in. Now, what to do with all that information?

Putting students to work

To help figure out how to best use the stream of information coming its way, the Mall conducted a study of its data in a partnership with graduate students at the Carlson Analytics Lab at the University of Minnesota’s Carlson School of Management. By mapping visitor behavior using Wi-Fi and beacon activity – tracking where shoppers arrived, where they walked and how long they spent in different areas of the Mall – Mall of America and the student researchers were able to uncover interesting stats on things like in-mall promotions, events and store appeal.

Mall kiosk with Wi-Fi inside to drive the Cisco StadiumVision software. Credit: Paul Kapustka / MSR

Mall kiosk with Wi-Fi inside to drive the Cisco StadiumVision software. Credit: Paul Kapustka / MSR

According to the Mall’s presentation at the recent SEAT Conference in Las Vegas, one analysis showed that if visitors were offered free admission to the amusement park, they actually spent 40 percent of their time at the Mall somewhere other than the amusement park — a sign that free amusement park entry could spur more shopping. The analysis also showed that during events at the Mall – according to the Mall, it hosts more than 400 special events a year – visitors stayed at the Mall on average 1.4 times longer than visitors who did not attend an event. The Mall also found out that 39 percent of event attendees visited the Mall’s food courts, compared to 25 percent of non-event visitors.

During its presentation, the Mall also showed screenshots of interactive “heat maps” showing exactly where visitors entered, where they walked, and how long they stayed. This information was gathered by the Wi-Fi AP beacons, which allowed for accurate device location tracking. With such information at their fingertips, the Mall sees a future where the network helps initiate new features for assisted shopping and custom experiences for visitors without resorting to historic feedback systems like surveys or focus groups.

Data driving the future

“This data is golden when it comes to describing shopper behavior,” said Smrcka, who also talked about the deployment at the SEAT Conference in Las Vegas. Shopper surveys, she said, have proven to sometimes not be reliable, and “who has the time to sit in a focus group for hours?”

How many malls do you know that have a One Direction tribute photo? Credit: Paul Kapustka / MSR

How many malls do you know that have a One Direction tribute photo? Credit: Paul Kapustka / MSR

This information enhances other Mall guest behavior and experience initiatives. “We already have a social media command team watching geo-located social media posts,” Smrcka said. They also employ a text messaging system, which visitors use to send messages to customized numbers to communicate if a bathroom needs servicing, set a reminder for their parking spot, or to find out where the closest gelato stand is.

And while Mall of America, like other bricks and mortar retailers, competes every day against online shopping, Smrcka said there are plenty of people who still want to see and feel the goods they are purchasing. Mall of America plans to use their Wi-Fi network’s data to make the guest experience even better and study the feasibility of possible future services like personal shopping, valet parking, curbside pickup and home delivery.

Moreover, Smrcka and her team can better segment and target its visitors and their entertainment, dining and shopping needs.

“What we’re able to do [from analytics] is still changing from month to month,” Smrcka said. “But the data really empowers a team like ours.”

NRG Stadium Wi-Fi ‘soft launches’ at Texans preseason game

Screen Shot 2016-08-26 at 10.35.17 AMThe Houston Texans gave their fans more to cheer about with a 16-9 preseason victory over the New Orleans Saints last Saturday, but what might have made a lot of people happy at NRG Stadium was the unofficial debut of the stadium’s new Wi-Fi network, which was available in a sort of “soft launch” mode.

We say “sort of,” because according to people who were at the game there was pretty heavy promotion of the new network’s availability, with bandwidth sponsor Comcast distributing flyers in seat cup-holders as well as making in-stadium announcements about the wireless connectivity. NRG Stadium had been one of the few NFL venues without Wi-Fi, but with the Super Bowl headed to Houston at the end of this season installing Wi-Fi became a priority.

Starting after the Final Four concluded this past spring, integrator 5 Bars and Wi-Fi gear provider Extreme Networks got busy, eventually installing approximately 1,250 Wi-Fi APs inside NRG Stadium. According to 5 Bars representatives, many of the APs in the seating bowl were installed under the seats, a deployment method that is becoming a trend in larger stadiums.

Though we don’t have any stats yet (since the network isn’t really “officially” launched) we did hear from network sources that there was a good uptake on the system, and we are looking forward to watching the Wi-Fi’s performance this season leading up to Super Bowl 51 in February. If any fans out there hit another Texans game anytime soon, send us a speedtest of the Wi-Fi!

Optical fiber, under-seat Wi-Fi will power wireless connectivity at Atlanta’s Mercedes-Benz Stadium

Aerial photo of Mercedes-Benz Stadium under construction. Credit all photos and artist renderings: Merecedes-Benz Stadium (Click on any photo for a larger image)

Aerial photo of Mercedes-Benz Stadium under construction. Credit all photos and artist renderings: Merecedes-Benz Stadium (Click on any photo for a larger image)

Once just a series of drawings on a blueprint, Atlanta’s new Mercedes-Benz Stadium is getting more real by the day, with walls being added to steel beams, and wires for the internal networks being pulled into place.

Though the June 2017 opening day still is many months away, thanks to thoughtful planning many elements of the stadium’s network have already been tested, thanks to a facility created by stadium network officials to test components under situations as close to “live” as they could possibly get. That lab environment helped the network team make its final decisions on vendors and deployment methods, like going under-seat for deployment of most of the 1,000 Wi-Fi APs that will be in the stadium’s bowl area, part of a planned total of 1,800 APs in the entire venue.

In a recent interview with Jared Miller, chief technology officer at AMB Sports and Entertainment (the entity named for Arthur Blank, the owner of the Atlanta Falcons), Mobile Sports Report got an exclusive update on the construction progress so far for the new $1.5 billion facility, along with new details about the internal network deployment, which will be using more optical fiber than any previous stadium network we know of.

Like the network built at Texas A&M’s Kyle Field, the network inside Mercedes-Benz Stadium will have a single optical core for Wi-Fi, cellular and video, using the Corning ONE platform and deployed by lead network integrator IBM along with Corning.

Wall panels being added to Mercedes-Benz Stadium in Atlanta

Wall panels being added to Mercedes-Benz Stadium in Atlanta

Miller also confirmed our earlier report that YinzCam software would be used to host the stadium’s IPTV deployment, but vendor choices for Wi-Fi gear and a stadium app have yet to be named.

As construction teams continue to hustle toward completion of the building, here are more details from our conversation with Miller about how the Falcons’ tech team went through the process of determining the products and methods that would allow them to construct a network able to “push the limits” on fan connectivity.

Under-seat for Wi-Fi, with handrail heat sinks

In our early August conversation with Miller, he was happy to report that the planned 4,000 miles of optical fiber were finally starting to be threaded into the new building. “We’re making great progress with a ton of yellow cable,” Miller said.

While the overall architecture at the network core in Mercedes-Benz Stadium will be similar to the one IBM and Corning deployed at Kyle Field, Miller said that in Atlanta his team is pushing fiber even farther to the edge, “with only the last couple feet at most being copper.”

Interior suite construction with fiber cable visible

Interior suite construction with fiber cable visible

Miller said optical fiber, which can carry more data traffic at faster speeds than copper cable, is a necessary infrastructure underpinning for facilities like Mercedes-Benz Stadium that expect to host the biggest events like the Super Bowl and college football championship games. Mercedes-Benz Stadium is already slated to host Super Bowl LIII, the 2018 College Football Playoff Championship, and the 2020 Final Four.

“I really believe [fiber] gives us the foundation to grow and react in the future, to handle technologies we don’t even know about yet,” Miller said.

On the Wi-Fi side of things, Miller said that Mercedes-Benz Stadium will also mimic Kyle Field’s extensive use of under-seat APs in the bowl seating areas. Miller said the stadium will have 1,000 APs serving the seating areas and another 800 for the rest of the venue, for a total Wi-Fi AP count of 1,800.

Since the Mercedes-Benz Stadium network will be using more optical equipment closer to the edge, Miller said that his team used 3D printing experiments to craft custom enclosures for the under-seat APs, both to ensure they didn’t act as debris “traps” and also to add elements like an internal heat sink to diffuse the warmth from the extra electrical components. The heat sink solution involved attaching the AP elements to metal chair railings to dissipate heat, Miller said.

Testing the network before the building is built

After announcing its partnership with IBM in early 2015 as lead technology integrator, the stadium network team spent 6 months reworking the network design, Miller said, a process that confirmed the choice of optical networking at the core. Then to help the network team select gear and components, the Mercedes-Benz Stadium organization built a “full-scale lab facility” that Miller said allowed his team to build multiple live networks to test gear for performance and interaction with other network elements.

Artist rendering of outside of building

Artist rendering of outside of building

“The lab enabled us to see firsthand how gear behaved, not just alone but together [with other products],” said Miller, who added that at one time the network team had three simultaneous running stadium networks inside the lab.

“We were able to bring in different endpoint devices, like POS systems, and know how it’s going to behave [in a network],” Miller said. Plus, the network gave eventual business users of the planned gear time to get hands-on experience and training well before the stadium opens its doors.

On the DAS side of the network buildout, Miller said the stadium has an on-site, raised-floor room for DAS gear with “ample room” for future growth.

“One of those things we learned was that DAS [needs] always double,” Miller said.

YinzCam software for IPTV

Though the stadium hasn’t yet announced a provider for a game-day stadium application, Miller did confirm that Mercedes-Benz Stadium will use YinzCam software to control its IPTV system, which will cover the 2,500 or so TV screens inside the building.

Artist rendering of Falcons game configuration with roof open and 'halo' video board visible

Artist rendering of Falcons game configuration with roof open and ‘halo’ video board visible

“YinzCam is just the most intuitive and capable content management system,” Miller said.

Video is going to be a big part of the stadium from all angles, beginning with the one-of-a-kind “halo board,” a circular screen that will sit inside the retractable roof lines. For standard TV placements, Miller said Mercedes-Benz Stadium will use mainly 50-inch screens and will work with YinzCam to ensure the screens can be seen.

In the stadium’s suites, TV screens will be controlled by a tablet application; Miller said that Mercedes-Benz Stadium is also “contemplating adding the ability to control TV screens with a mobile app,” like the system YinzCam deployed at Texas A&M.

Friendly food pricing and more to come

Though Miller’s concerns are mostly technological in nature, he said there are still a lot of improvements coming to the stadium “that are not always reliant on brute technology,” like the new lower-priced food menus the Falcons announced earlier this year that seem to harken another era with $2 Cokes and $2 hot dogs. Miller said the stadium team continues to get feedback from a fans’ council, which has tagged the arrival and departure experience as one of the main pain points that needs fixing.

Artist rendering of window wall with view to city

Artist rendering of window wall with view to city

Mercedes-Benz Stadium will try to alleviate ingress and egress issues by doing things like creating “ticketed spaces” perhaps on the big outdoor plazas where many fans can congregate even before entering the stadium doors. By creating such spaces, Miller said fans might be able to enter the stadium more rapidly without the logjams that sometimes occur.

“We’re going to study arrival patterns and see what it looks like,” Miller said. “We have one more season to test those kind of things.”

Another amenity that may emerge is the use of wireless charging stations at a number of locations, to combat a scenario that Miller said often happens at marquee events, mainly fans’ phones draining their batteries as they compete with other devices to connect to a wireless network.

“We are focusing on providing amazing connectivity and pushing the limits,” Miller said. “We are looking at all kinds of options to allow fans to stay connected and not be separated from their device.”