Verizon goes under concrete to bolster NRG Stadium DAS for Super Bowl LI

Nodes on wheels, or NOWs, provide extra coverage for Verizon Wireless in Houston for Super Bowl LI. Credit: Verizon Wireless

Nodes on wheels, or NOWs, provide extra coverage for Verizon Wireless in Houston for Super Bowl LI. Credit: Verizon Wireless

In a slight twist from its strategy for last year’s Super Bowl, Verizon Wireless has installed DAS antennas underneath the concrete flooring of lower-tier seats at Houston’s NRG Stadium, to provide extra bandwidth for the expected high wireless data usage at Super Bowl LI.

Last year at Levi’s Stadium in Santa Clara, Calif., Verizon beefed up its distributed antenna system (DAS) with under-seat antennas it designed specifically for use in stadiums. The idea of mounting antennas under seats, a growing trend in the stadium Wi-Fi world, is gaining traction as another method of bringing signals closer to fans, especially in places (like lower bowl seats) where there are no overhangs or other places to mount gear.

And while Verizon has been preparing for Sunday’s big game at NRG Stadium for years, that didn’t stop the company from “continually tweaking” its network preparations, according to Leo Perreault, executive director of network operations for Verizon’s South Central market, a region that stretches from west of Florida to Arizona, including Houston. In a phone interview this week, Perreault said that Verizon installed the under-concrete antennas during the middle of the 2016 football season, giving the company “some good experience” with the deployment ahead of Sunday’s game.

Under concrete = easier install and maintenance

A view inside the head end room that runs Verizon's NRG Stadium DAS. Credit: Verizon Wireless

A view inside the head end room that runs Verizon’s NRG Stadium DAS. Credit: Verizon Wireless

It might not be well known outside of wireless networking circles, but signals will travel through concrete; many early stadium Wi-Fi designs (and some current ones, including a new network installed at the Pepsi Center in Denver) use antennas mounted under concrete floors, pointing up. Though fixed under-seat antennas can provide better coverage, Perreault said the ease of deployment made putting the additional DAS antennas underneath the floor a better option in Houston.

“This way [under the concrete] is non-intrusive,” Perrault said, noting that the devices are also not affected by stadium power-washing units. The decision may have been influenced by the fact that NRG Stadium’s new Wi-Fi network had a big issue with moisture in under-seat AP placements, forcing a mid-season rip and replace for all the under-seat Wi-Fi APs.

Even though antennas under concrete are not as powerful, Perrault said Verizon is “very pleased with the performance. It’s a good compromise.”

Biggest stadium DAS?

Between the game being the Super Bowl and it being in Texas, there’s no shortage of hyperbole surrounding the game and all its attendant facets, including the network technology. But when Perreault claims that the DAS Verizon has installed for NRG Stadium “might be the largest we have anywhere,” that might be true since it also serves adjacent properties including the NRG Convention Center, the NRG Arena and an outdoor DAS in the surrounding spaces. In addition to Houston Texans games, NRG Park (which includes the stadium) is also host to the Houston Livestock Show and Rodeo, which humbly bills itself as the “world’s largest livestock show and richest regular-season rodeo.”

An alien spaceship, or a temporary cell tower from Verizon? You choose.

An alien spaceship, or a temporary cell tower from Verizon? You choose.

Inside NRG Stadium, Perreault said the new Verizon DAS (built before the 2015 season) has more than 900 antennas. As neutral host, Verizon will also provide access to AT&T and T-Mobile on its network; Sprint, which built a previous DAS at NRG, will continue to run on that system.

Outside the stadium and around Houston, Verizon has done the usual big-event preparations, with lots of permanent and temporary macro network improvements, and portable units like COWs (cells on wheels) and smaller NOWs (nodes on wheels). You can review all the Verizon preparations in a company blog post.

As previously reported in MSR, Verizon also helped foot part of the bill for the new NRG Stadium Wi-Fi network, a deal that will give Verizon a reserved claim to 40 percent of the Wi-Fi network’s capacity, according to Perreault.

Whether or not Super Bowl LI breaks the wireless data consumption records set at last year’s game remains to be seen, but Perreault said there doesn’t seem to be any slowing down yet of the perpetual growth in wireless data use at stadiums, especially at big events like the Super Bowl.

“Fans just seem to find ways to consume whatever additional bandwidth you provide,” he said.

Sprint sees 797 GB at college playoff title game; will have separate DAS for Super Bowl

Ready for the playoffs and Super Bowl! Credit: 5 Bars

Ready for the playoffs and Super Bowl! Credit: 5 Bars

More results from fan wireless usage at the recent College Football Playoff championship game is trickling in, with Sprint claiming it saw 797 gigabytes of data traffic from its sites in and around Raymond James Stadium in Tampa.

Added to the 3.8 TB reported by AT&T on its networks, we now have a running total of approximately 4.6 TB of DAS usage for the Jan. 9 game between Clemson and Alabama, which Clemson won 35-31 on a last-second TD. We are still waiting for reports from Verizon Wireless and T-Mobile, so stay tuned.

We also have yet to receive any reports of Wi-Fi traffic from Raymond James Stadium officials, so it’s still undetermined how “big” of a wireless event the college championship game was this year. In the past two years, the CFP championship was among the top Wi-Fi single-day usage totals, but so far the Raymond James Stadium folks haven’t responded to any calls or emails requesting information. Anyone who was at the game who wants to comment on the Wi-Fi performance, feel free to jump in to the comments below.

Sprint on its own DAS at NRG Stadium

We also learned from Sprint and from NRG Stadium officials that Sprint will be on its own DAS and small cell network for the Super Bowl, and not on the new Verizon DAS that was installed last year. In a Sprint blog post the company said it saw 637 GB of data on its NRG Stadium network for a December Houston Texans game, ahead of what it saw on its networks during last year’s Super Bowl at Levi’s Stadium in Santa Clara, Calif.

According to David Moore, manager of information services for NRG Park, Sprint installed the original DAS in the venue, ahead of Super Bowl XXXVIII in 2004. AT&T and Verizon “migrated” to the new Verizon DAS last season, and T-Mobile will also be on the Verizon DAS, according to Moore.

The MSR Interview: San Francisco Giants CIO Bill Schlough

AT&T Park CIO Bill Schlough shows off his World Series bling.

AT&T Park CIO Bill Schlough shows off his World Series bling.

Who better to talk about stadium Wi-Fi than the guy who was there when it all started? Our guest for our first MSR Interview (part of our Stadium Tech Report Podcast series) is San Francisco Giants senior vice president and chief information officer Bill Schlough, who goes old-school talking about stadium Wi-Fi back in 2004… and brings it to the current day with stats from the most recent season at AT&T Park. Plus, his thoughts on game-day apps and why great connectivity is the real winner. Listen in now!

Hear Bill talk about:

— New Wi-Fi records set… during the Warriors’ playoff run

— Why going under-seat with Wi-Fi was a necessary thing to do

— How the Giants are experimenting with virtual reality

— Why he thinks great connectivity matters most (even more than stadium-app features)

Some story links that offer some history about AT&T Park’s networks from MSR:

S.F. Giants add more Wi-Fi, ‘virtual reality experience’ to AT&T Park for 2016 season

SF Giants fans used 78.2 TB of Wi-Fi data at AT&T Park during 2015 season

Stadium Tech Report: World Series set new wireless records at AT&T Park

Stadium Tech Report: San Francisco’s AT&T Park lives up to its wireless reputation

Giants: NLCS stadium Wi-Fi usage at AT&T Park quadrupled since 2012

SUBSCRIBE TO THE PODCAST:

Here is the link to the podcast on iTunes!

Stadium Tech Report: Carolina Panthers take ownership of DAS, Wi-Fi at Bank of America Stadium

James Hammond, director  of IT for the Panthers, poses next to an under-seat Wi-Fi AP. Credit all photos: Carolina Panthers

James Hammond, director of IT for the Panthers, poses next to an under-seat Wi-Fi AP. Credit all photos: Carolina Panthers

“The fan is the most valuable member of our team,” Jerry Richardson, owner of the Carolina Panthers, is fond of saying.

And it’s become the virtual mission statement for the Charlotte, N.C.-based National Football League franchise. So even though its home field, the Bank of America Stadium, was built relatively recently (1996), technology has come a long way in two decades. And as the Panthers began a four-phase renovation in 2014, they did it with fans’ MVP status in mind, according to James Hammond, director of IT for the Panthers. “It was time for some changes,” he said.

While Carolina was among the first NFL stadiums to install fan-facing Wi-Fi and enhanced cellular networks, the previous DAS and Wi-Fi systems weren’t keeping up with demand and that was starting to adversely impact the Panthers fan experience, Hammond said.

“We chose to perform a rip-and-replace on both DAS and Wi-Fi and take ownership in-house,” Hammond explained. Because the Panthers own and operate BofA Stadium, making those moves was a lot easier than if they were tenants.

Time for an upgrade

Editor’s note: This profile is part of our latest STADIUM TECHNOLOGY REPORT, which includes more stadium profiles as well as looks at Wi-Fi at the Mall of America, and analytics software being used by the Cleveland Browns. DOWNLOAD YOUR FREE COPY of the report today!

The first “fan-centric improvements,” as Hammond called them, came in 2014 in the form of escalators, video big boards and a distributed audio system. As part of the second phase of upgrades, the Panthers then used the 2015 offseason to renovate the club-level suites and tore out the old DAS system while they were at it. And after a careful evaluation of different DAS solutions, they shortlisted two vendors: CommScope and Corning.

CommScope ultimately got the nod; the Panthers then had to decide between the vendor’s ION-B and ION-U DAS systems. “We went with the ION-U, which was quite new and cutting edge at that point, since it had NEMA-rated remotes,” Hammond said. Other systems lacked that kind of weatherproofing and would require additional enclosures – and expense.

CommScope's ION-U powers the new DAS at Bank of America Stadium.

CommScope’s ION-U powers the new DAS at Bank of America Stadium.

“We started over with all new fiber and coax. We did the decommissioning and construction in 90 days, which was pretty quick for a ground-up project,” he said. Beam Wireless Inc. did the design, integration and optimization and is handling the ongoing maintenance of the DAS system; Optical Telecom installed the DAS gear. BofA Stadium now has 256 DAS remotes and more than 600 DAS indoor and outdoor antennas.

AT&T, Verizon and Sprint are the participating DAS carriers; T-Mobile is weighing whether to join the mix during the 2017 off-season.

The Panthers have divided BofA Stadium into 48 DAS zones: 16 zones for the upper bowl, 16 in the lower bowl, and another 16 for concourses, suites, clubs, and offices. Not all zones are used exclusively; carriers choose simulcast patterns that place multiple zones into sectors, and can change them as capacity requirements dictate, Hammond told Mobile Sports Report.

“With some minor design changes to the interior areas, we can accommodate nearly 70 zones,” he explained. “At present the most sectors in use by a carrier is 32. This means the carrier simulcasts across a mix of our 48 zones in order to match them up to 32 carrier sectors.”

Once the new DAS was built and the first couple of events were analyzed, carriers began asking for more frequencies and additional DAS sectors to continue meeting ever-growing demand. In response to the new carrier requests, the first round of DAS upgrades were implemented in the spring of 2016, Hammond said. During the 2015 season, DAS bandwidth was running around 2 GB during games. Hammond said, “With these latest DAS upgrades, we expect the bandwidth numbers to be even higher.”

A DAS remote in a NEMA-rated enclosure.

A DAS remote in a NEMA-rated enclosure.

The impact of the new DAS system was felt immediately upon its debut in July 2015. “It was a much better experience for fans who noticed the improved cellular experience,” Hammond said. Another unexpected benefit: The upgraded DAS helped mitigate bottlenecks with the old Wi-Fi system, which Hammond characterized as “under-designed.”

Going under seat for Wi-Fi upgrade

Unfortunately, there wasn’t time to address any Wi-Fi upgrades before the 2015 football season began, but the Panthers issued an RFP for new Wi-Fi in August 2015 in preparation for Phase 3 renovations that would also include security upgrades and renovations to the upper concourse.

Interested vendors needed to ensure high bandwidth rates as well as high take-rates that allowed three different ISPs (Time Warner Cable/Charter, Level 3 Communications and Windstream) to deliver in excess of 10 GB, though Hammond said they’re starting at a 7-GB threshold.

The Wi-Fi award went to Aruba, now HP Enterprise, in December 2015, and construction began in January 2016 after the last postseason game, when the Panthers beat the Arizona Cardinals to win the NFC championship and a trip to Super Bowl 50.

Similar to Levi’s Stadium and the Dodgers Stadium, the Panthers chose underseat AP enclosures; BofA Stadium sports 770 AP enclosures in the upper and lower bowls out of a total of 1,225 APs, all to ensure maximum coverage and minimal dead spots. The Panthers selected AmpThink to do the Wi-Fi integration and construction; the turnkey contractor also designed and fabricated a custom enclosure for the APs.

Indoor access point inside the stadium.

Indoor access point inside the stadium.

One other innovation in the Panthers’ Wi-Fi installation is that the underseat enclosure is mounted to the riser — the vertical part of the step — but looks like it’s on the tread, the horizontal part, which is intended as a waterproofing measure. “The riser is easier to seal and isn’t affected by pressure washers, which you’re doing constantly with an outdoor stadium,” Hammond said. “And by running pipe through the riser, you don’t have gravity working against you,” which helps keep out water, he explained.

Panthers fans access the stadium Wi-Fi through a portal page after accepting the team’s terms and conditions. From there, they are whitelisted and can automatically join the Wi-Fi network for the rest of the season. Hammond said a fan’s email is requested but not required by the portal page, and there’s a small incentive offered to encourage fans’ email subscriptions.

The new Wi-Fi system got a workout with a soccer game at BofA Stadium at the end of July 2016, then with a Panthers’ Fan Fest the following week. “All the indicators were good, and fan feedback about the system was excellent,” Hammond said. But he cautioned that the two events were not “full bowl” events with smaller attendance numbers (~50,000) than a regular season football game (75,000+). “We will continue to optimize and tune settings as we learn more during events with higher attendance,” Hammond said.

Total budget so far for the technology upgrades totals about $16 million; the DAS build-out was just under $10 million; Wi-Fi was a little more than $6 million, which included additional wired infrastructure, according to the team.

Beacons coming next

And the Panthers aren’t done making technology improvements to their stadium. Phase 4 looks to add Bluetooth beacons and do some refinement of the Panthers app. “My goal during the upcoming season is to look at options for location-aware services,” Hammond said. Some APs have beacons built in; other may need to be added to get the granularity the Panthers want for location awareness.

Hammond also wants to give fans more things to do with the Panthers app and also optimize it for push notifications, even with something as basic and useful as restroom and concessions location information. “As we learn more about fans individually, we can direct them to things of particular interest to them,” he added.

“So far, we are very pleased with the performance of the Wi-Fi and DAS systems,” Hammond said, noting the Panthers will continue to tune frequencies, add zones and increase bandwidth where needed. It’s the sort of attention that smart sporting franchises pay to their most valued team members.

car1

Ookla shares Speedtest data from CenturyLink Field, other stadiums

Ookla ad banner being flown over CenturyLink Field in Seattle. Credit: Ookla

Ookla ad banner being flown over CenturyLink Field in Seattle. Credit: Ookla

Anyone who follows Mobile Sports Report knows that I use the Speedtest app from Ookla to measure stadium network performance whenever I visit a sporting venue. While my one-man tests do show some measure of network power, I always dreamed of harnessing the results from many fans at the same game to see a better picture of the network performance.

Well, Speedtest’s creators think along the same lines, and conducted an experiment during an Aug. 25 Seattle Seahawks preseason game at CenturyLink Field in Seattle. You can read their very thorough post and neat results here, with some interesting twists — for instance, the cellular networks are way faster than the CenturyLink Wi-Fi, according to the Ookla results.

UPDATE: Ookla responded to our email and let us know that on Aug. 25, there were 252 Speedtests at CenturyLink Field, a great sampling to draw results from. Ookla also talked about tests from 12 different events at CenturyLink Field, and said in the email that across those events it saw 1,143 tests conducted.

Ookla also published some test result totals from other stadiums as well, including Levi’s Stadium, AT&T Stadium and Bank of America Stadium, but didn’t say when those tests were recorded, or how many tests were taken.

What we really like, however, is that Ookla’s tests show what our stadium tech report surveys have been showing — that overall, in-stadium network performance is steadily improving. Over time, more data like this can help dispel the still-lingering rumor that stadium networks don’t deliver good connectivity. Now if we could only get Ookla to partner with us to do league-wide or college-comparison speedtests… anyone ready for that idea?

Will cellular carrier aggregation matter in stadium networks?

Kauffman Stadium during 2015 World Series

Kauffman Stadium during 2015 World Series

Over the past few days, both Sprint and Verizon Wireless have made announcements about a technique called “carrier aggregation” (CA for short) for LTE cell networks that basically bonds together different frequency channels to bring more bandwidth to a mobile device. Though the premise sounds great, what we here at MSR HQ haven’t been able to ascertain yet is whether or not this technique will help solve the biggest problem in stadium network situations, namely providing enough capacity for users on the networks installed there.

Sprint has made the most noise this week, with claims of CA demonstrations at Soldier Field in Chicago and Kansas City’s Kauffman Stadium that (they said) showed Sprint devices bonding three different frequency channels to hit download speeds of 230 Mbps, a score way off the charts for any existing stadium networks. (The fastest Wi-Fi and cellular speeds we’ve seen in our short history of stadium tests, by comparison, are in the 60 Mpbs range.) Verizon made a similar announcement about CA being put in across its network, without specifying if the service would be available in stadiums. Other carriers, including AT&T and T-Mobile, are also exploring use of the CA technique. At the very least, some lucky users with newer devices may see leaps in performance thanks to CA deployments, a good thing on any level.

But our bigger question — which hasn’t been answered in the press releases and hasn’t (yet) been answered in email questions to Sprint or Verizon — is whether or not CA will help with overall network capacity, which to us seems to be a more pressing problem at most stadiums as opposed to simple download speeds. I mean, demos are great and it’s cool to see what the upper limits are for one device; but it’d be more impressive if Sprint could guarantee that 230 Mbps mark to every device in the park, should everyone there have a Sprint phone with the capability to perform the CA trick (not all devices in the market today can do so).

Finally using the Clearwire spectrum

What’s also not completely revealed in the press releases is what kind of gear is necessary on the back end of the network to make CA work, and whether or not it makes economic sense to have that gear placed inside stadiums to enable the technique for as many fans as possible. While we understand the basic premise probably better than most (since in a former life yours truly spent several years following and analyzing the Clearwire spectrum holdings at 2.5 GHz) it’s not clear if CA solves any congestion problems, especially for carriers other than Sprint, who only have a limited amount of licensed spectrum in each market they serve.

(Without getting too deep into spectrum geekiness, Sprint on paper probably has more room to grow in the CA space since its 2.5 GHz holdings dwarf other carriers’ licensed bands; but to make use of that spectrum, you need customers with devices that can use that spectrum, and enough cash for a wide network buildout, both of which Sprint may be challenged to find.)

As we understand CA, by bonding channels you can make one device faster since it has more aggregate bandwidth to work with. But it’s not clear that using CA in a stadium environment would make the overall situation any faster than say, three phones using single channels by themselves. Also, since you can’t create new bandwidth, if one phone starts tapping three different channels doesn’t that actually leave less room for other devices that may want to also use those channels? Perhaps with CA the connections would be faster and wouldn’t last as long, thereby freeing up spectrum for other devices; again, there’s not a lot of information yet on the capacity side of the equation, especially in crowded stadiums or at big events where bandwidth needs escalate. If there are any cellular wizards in the audience with more knowledge of the situation, feel free to chime in.

We did get an email response from our old friend John Saw, formerly of Clearwire and now chief technical officer at Sprint. Here’s his explanation of why CA is a good thing for stadiums:

Essentially, sites with bonded channels will drive higher capacities. This will be especially timely and helpful in crowded spaces like Soldier Field where there are surges in capacity demand during live sporting events. Sprint customers with CA enabled phones will enjoy 2X (in the case of 2CA) or 3X (in the case of 3CA) their download speeds, which means that they will get a better data experience with a bigger pipe. But wait – CA will lift all boats and it will also benefit those Sprint customers who have not upgraded to CA enabled phones yet. While they may not enjoy the higher peak speeds enabled by CA phones, their phones will have access to more network resources which means they will also have a better data experience, with no stalling or without that dreaded “windmill effect” in a crowded stadium.

I kind of understand what Saw is talking about here, but I am still having a problem with the math that says all boats will be lifted through the use of CA. Plus, experience and interviews have taught us that across the country, Sprint is behind Verizon and AT&T when it comes to DAS deployments inside stadiums; and, it’s not clear (and hasn’t been answered) whether or not CA can work over a neutral-host DAS deployment where carriers share antennas and other infrastructure.

From an industry-wide standpoint, CA seems like a great thing for all cell phone users since as it progresses devices should be able to utilize whatever bandwidth is around to make performance better. It’s also good to see more technology advancements made on the network side of things, since infrastructure needs all the help it can get to keep up with devices. But right now, we’re not sure if CA is the answer to any of the capacity problems stadium network operators face. Anyone with views that can expand the explanation, feel free to hit the comments section below or send me an email to kaps at mobilesportsreport.com.

https://duwit.ukdw.ac.id/document/pengadaan/slot777/

https://mtsnupakis.sch.id/wp-content/zeusslot/

https://insankamilsidoarjo.sch.id/wp-content/slot-zeus/

https://smpbhayangkari1sby.sch.id/wp-content/slot-zeus/

https://alhikamsurabaya.sch.id/wp-content/slot-thailand/

https://mtsnupakis.sch.id/wp-content/bonus-new-member/

https://smptagsby.sch.id/wp-content/slot-bet-200/

https://lookahindonesia.com/wp-content/bonus-new-member/

https://ponpesalkhairattanjungselor.sch.id/wp-content/mahjong-slot/

https://mtsnupakis.sch.id/wp-content/slot777/

https://sdlabum.sch.id/wp-content/slot777/

https://sdlabumblitar.sch.id/wp-content/bonus-new-member/

https://sdlabumblitar.sch.id/wp-content/spaceman/

https://paudlabumblitar.sch.id/wp-content/spaceman/