January 31, 2015

Stadium Tech Report: AT&T Stadium’s massive antenna deployment delivers solid Wi-Fi, DAS performance

The old saw that says “everything’s bigger in Texas” is not just a stereotype when it comes to wireless networking and AT&T Stadium. Though our visit was brief and we didn’t have the opportunity to do a deep-dive technology tour, the MSR team on hand at the recent College Football Playoff championship game came away convinced that if it’s not the fastest fan-facing stadium network, the Wi-FI and DAS deployments at AT&T Stadium sure are the biggest, at least the largest we’ve ever heard of.

Inside AT&T Stadium at the College Football Playoff championship game. (Click on any photo for larger image) Credit all photos: Paul Kapustka, MSR

Inside AT&T Stadium at the College Football Playoff championship game. (Click on any photo for larger image) Credit all photos: Paul Kapustka, MSR

And in many ways we found, bigger is better, at least when it comes to staying connected inside one of the world’s truly humongous indoor spaces.

If you’ve not seen the stats, allow us to remind you that during the Jan. 12 championship game between the University of Oregon and THE Ohio State University the AT&T Stadium network carried more than 6 terabytes of wireless data, with almost 5 TB of that over the in-house Wi-Fi network. Another 1.4 TB was recorded being used by AT&T customers on the AT&T-hosted neutral DAS, which almost certainly carried another terabyte or two from other carriers on the system, who did not report any statistics. Any way you add it up, it’s the biggest single-day wireless data figure we’ve ever heard for a sports arena, professional or college, in any sport at any time.

Flooding the zone with more antennas and APs

How do you get such a big data number? One way is to make sure that everyone can connect, and one way to get to that point is to flood the zone with antennas and access points. Already the leader in the number of Wi-Fi access points and DAS antennas, AT&T Stadium got another 280 Wi-Fi antennas installed between Thanksgiving and the college championship game, according to John Winborn, CIO for the Dallas Cowboys. Some of those antennas, the staff said, were housed in new under-the-seat enclosures that AT&T’s Foundry designed somewhat specifically for use in the lower bowl of AT&T Stadium, which like other stadiums had previously had issues getting connectivity to seats close to field level.

According to Winborn, the AT&T Stadium now has more than 1,600 Wi-Fi APs in use for football games, and 1,400 antennas in its DAS network. By comparison, Levi’s Stadium in Santa Clara, Calif., perhaps the newest and one of the most technologically savvy venues out there, has 1,200 Wi-Fi APs and 700 DAS antennas in its deployments. Winborn also said that the antenna/AP number at AT&T can also scale up as necessary, especially for events that use up more of the building’s space, like the Final Four basketball tournament held there last spring.

“We scaled up to 1,825 [Wi-Fi] antennas for the Final Four last year,” said Winborn in a recent email, where he guessed that the venue might deploy up to 2,000 Wi-Fi APs when the Academy of Country Music Awards holds its yearly event at AT&T Stadium on April 19.

Hiding Wi-Fi APs an aesthetic priority

John Winborn, CIO for the Dallas Cowboys, poses next to a picture of two other innovators, Tex Schramm and Gil Brandt

John Winborn, CIO for the Dallas Cowboys, poses next to a picture of two other innovators, Tex Schramm and Gil Brandt

For all the extra numbers, one thing we noticed in walking around the building on Jan. 12 was that seeing an exposed Wi-Fi AP is about as common as seeing an albino deer. When we asked Winborn what the toughest thing was about network deployment in the venue, he responded quickly: “Finding ways to hide the APs so Jerry [Jones] doesn’t see them.”

With the price-is-no-object Jones on one side, and AT&T’s corporate image on the other, it’s clear there aren’t too many budgetary concerns when it comes down to spending more to make the network work, or look, better. Put it this way: You are never likely to have a “no signal” problem in a building that has on its outside an AT&T logo the size of the moon, and where AT&T CEO Randall Stephenson can be found wandering around the suite level during big events.

Though the immense space could probably be covered by fewer antennas, it’s worthwhile to remember that when the building was built and opened in 2009, it wasn’t designed with high-speed networking in mind. That means that almost all of the Wi-Fi and DAS deployments are a retrofit, including the ingenious circle of Wi-Fi antennas halfway up the seating bowl, which are covered by a tented ring of fiberglass designed and built specifically for the stadium.

According to Winborn the Wi-Fi network is supported by its own 2 GB backbone, with separate backbones in place for media networks and stadium application use. Winborn also noted that the stadium network runs 3,500 TVs via the Cisco StadiumVision system. Other records from this season include a peak concurrent Wi-Fi user mark of 27,523 (set at the Lions playoff game) and 38,534 unique Wi-Fi connections, that mark set at the season opener against the San Francisco 49ers.

Performance solid, even at rooftop level

The view from the nosebleed section

The view from the nosebleed section

So how fast are the Wi-Fi and DAS networks? In our limited testing time at the CFP game, we found solid connections almost everywhere we tried, including outside the stadium while we (freezingly) waited for the doors to open. Just outside the main ticket gate, we got a Wi-Fi signal of 23.93 Mbps on the download and 39.67 Mbps on the upload. At the same location a Verizon 4G LTE device got a 5.93 Mbps download speed, and a 2.59 Mbps upload speed, but it’s unclear if that was on the stadium DAS or just on the local macro network.

When the doors finally opened at 5:30 p.m. (no idea why Jerry kept us all out in the cold all afternoon) we went inside and got solid connections inside the foyer of the pro shop — 18.23/21.74 on Wi-Fi, 21.05/14.84 on an AT&T 4G LTE device, and 12.65/4.61 on a Verizon 4G LTE phone. (It’s worthwhile to note that all our Wi-Fi speeds were recorded on the Verizon device, a new iPhone 6 Plus.)

Down in our field-level suite, where we were the guests of AT&T, we got marks of 19.43/25.31 on the Wi-Fi, 7.35/11.04 on AT&T 4G and 5.71/4.05 on Verizon 4G. An interesting note here: When Oregon scored a touchdown on its opening drive, we took another Wi-Fi speedtest right after the play and got readings of 4.38/7.79, suggesting that there were many Ducks fans communicating the good news.

Later during the game we wandered up to the “Star Level” suites (floor 6 on the stadium elevator) and got a Wi-Fi mark of 11.57/30.51, and 19.31/13.46 on AT&T 4G. The only place we didn’t get a good Wi-Fi signal was at the nosebleed-level plaza above the south end zone, where we weren’t surprised by the 1.41/1.98 Wi-Fi mark since we didn’t see any place you could put an AP. We did, however, get an AT&T 4G signal of more than 7 Mbps on the download in the same location, meaning that even fans way up at the top of the stadium were covered by wireless, no small feat in such a huge space.

Bottom line: Network in place for whatever’s next

If there is a place where AT&T falls behind other stadiums, it’s in the synchronization of network and app; since it wasn’t built with food delivery in mind, it’s doubtful that AT&T will match Levi’s Stadium’s innovative delivery-to-any-seat feature anytime soon. And even though AT&T Stadium is dominated by the massive over-the-field TV set, fans at the CFP championship game were left literally in the dark during questionable-call replays, since they weren’t shown on the big screen and aren’t supported in the AT&T Stadium app.

What could be interesting is if the technology demonstrated by AT&T at the big college game – LTE Broadcast, which sends a streaming channel of live video over a dedicated cellular link – becomes part of the AT&T Stadium repertoire. From experience, such a channel could be extremely helpful during pregame events, since many fans at the college championship were wandering around outside the stadium unsure of where to go or where to find will-call windows. A “pre-game info” broadcast over LTE Broadcast could eliminate a lot of pain points of getting to the event, while also introducing fans to the network and app for later interaction.

At the very least, AT&T Stadium’s network alone puts it in at least the top three of most-connected football stadiums, alongside Levi’s Stadium and Sun Life Stadium in Miami. Here’s looking forward to continued competition among the venues, with advancements that will only further improve the already excellent wireless fan experience.

More photos from our visit below. Enjoy!

Fans freezing outside waiting for the CFP game to start

Fans freezing outside waiting for the CFP game to start

Creative OSU fan

Creative OSU fan

Plug for the app

Plug for the app

AT&T Stadium NOC aka "the Fishbowl"

AT&T Stadium NOC aka “the Fishbowl”

Sony Club. Now we know where Levi's Stadium got its "club" ideas

Sony Club. Now we know where Levi’s Stadium got its “club” ideas

Panoramic view (click on this one!)

Panoramic view (click on this one!)

A glass (cup?) of bubbly to celebrate the 6 TB event

A glass (cup?) of bubbly to celebrate the 6 TB event

Seahawks hit local Wi-Fi record during playoff game with 2.6 Terabytes of traffic; Verizon maintains cone of silence over its investment in network

Screen Shot 2015-01-16 at 11.05.18 AMThe Wi-Fi network that debuted in CenturyLink Field this season had its highest Wi-Fi traffic day last weekend, when 2.6 terabytes of data was carried during the Seattle Seahawks’ Jan. 10 playoff victory over the Carolina Panthers.

According to the Seahawks’ tech staff, 18,899 of the 68,524 fans in attendance used the Wi-Fi network at some point, with a peak concurrent user mark of 15,662. The peak bandwidth utilization of 1.4 Gbps was reached just after Cam Chancellor sealed the win with his electric 90-yard interception touchdown return, the Seahawks tech staff said.

The Wi-Fi numbers represent traffic on both of the separate Wi-Fi networks in the stadium, one of which is reserved exclusively for Verizon Wireless customers. Verizon, which has declined to comment publicly on the specifics of its partnership with the Seahawks, is believed to have bankrolled a major portion of the Wi-Fi deployment at CenturyLink. Before the Wi-Fi partnership between the Seahawks, Wi-Fi gear provider Extreme Networks and Verizon was officially announced on Oct. 29, Verizon claimed it had “added an in-stadium Wi-Fi system” at CenturyLink prior to the start of the 2014 football season, as part of a national football stadium Wi-Fi map Verizon published on Sept. 4. The Wi-Fi deployment was somewhat of a surprise, since team officials had long said they were looking at 2015 as the year they might pull the trigger on a Wi-Fi expenditure. Apparently, having available funding from Verizon helped push the project forward faster than expected; but again, we have no official confirmation or explanation of the exact fiscal participation level of all the partners involved.

For Seattle fans, having Verizon as a Wi-Fi partner has additional perks — in addition to a separate Wi-Fi network for Verizon customers, all fans at the stadium also have access to the NFL Network’s popular RedZone channel, via the new Seahawks stadium app created by YinzCam. Because of various conflicting rights contracts, RedZone isn’t available in most stadiums for fans to watch. The Seahawks also have a number of live-action and replay views available via the app; however, the stadium’s tech team did not have any metrics on fan use of the app or the number of video replays watched.

Wi-Fi access point antennas visible on poles at CenturyLink Field, Seattle. Credit: Extreme Networks

Wi-Fi access point antennas visible on poles at CenturyLink Field, Seattle. Credit: Extreme Networks

Verizon’s reluctance to comment publicly on its Wi-Fi deployments is no surprise; repeated attempts and queries by MSR for interviews with Verizon executives about Wi-Fi deployments are routinely ignored by Verizon representatives, and public quotes like the one from Bobby Morrison, president for the Pacific Northwest and Alaska at Verizon Wireless, in the official press release, don’t offer any details about Verizon’s level of fiscal commitment to the CenturyLink deployment. Verizon has also declined to comment on its Wi-Fi network deployments at Ford Field in Detroit, and at the Staples Center in Los Angeles.

Verizon executives were also conspicuously absent from a stadium-technology event centered on the Wi-Fi network earlier this week, leading some reports to omit Verizon’s considerable participation in the network’s deployment, something NFL CIO Michelle McKenna-Doyle told MSR about earlier this year.

Though no statistics were available from Verizon about the DAS deployment it also put in at CenturyLink Field this year, AT&T did share some DAS stats from CenturyLink for the Seahawks’ Dec. 14 victory over the San Francisco 49ers. During that game, AT&T customer traffic on the AT&T DAS at CenturyLink was 395 GB, according to AT&T. It will be interesting to see how much wireless traffic this weekend’s NFC championship game between the Seahawks and the Packers will generate — we’ll track down as much of it as we can to see if it compares to the 6 TB mark set at the recent College Football Playoff championship game at AT&T Stadium in Arlington, Texas.

College championship game at AT&T Stadium breaks 6 Terabyte wireless data mark, with almost 5 TB of Wi-Fi traffic

AT&T Stadium before the college football playoff championship game. (Click on any photo for larger image) Credit all photos: Paul Kapustka, MSR

AT&T Stadium before the college football playoff championship game. (Click on any photo for larger image) Credit all photos: Paul Kapustka, MSR

Not only did Monday night’s College Football Playoff championship game crown a new new national title team — it also broke the unofficial record for most wireless traffic at a single sporting event, with more than 6 terabytes of data used by the 85,689 fans in attendance at AT&T Stadium in Arlington, Texas.

John Winborn, chief information officer for the Dallas Cowboys, said the AT&T-hosted Wi-Fi network at AT&T Stadium carried 4.93 TB of traffic during Monday’s game between Ohio State and Oregon, a far higher total than we’ve ever heard of before for a single-game, single-venue event. AT&T cellular customers, Winborn said, used an additional 1.41 TB of wireless data on the stadium DAS network, resulting in a measured total of 6.34 TB of traffic. The real total is likely another terabyte or two higher, since these figures don’t include any traffic from other carriers (Verizon, Sprint, T-Mobile) carried on the AT&T-neutral host DAS. (Other carrier reps, please feel free to send us your data totals as well!)

The national championship numbers blew away the data traffic totals from last year’s Super Bowl, and also eclipsed the previous high-water Wi-Fi mark we knew of, the 3.3 TB number set by the San Francisco 49ers during the opening game of the season at their new Levi’s Stadium facility. Since we’ve not heard of any other event even coming close, we’ll crown AT&T Stadium and the college playoff championship as the new top dog in the wireless-data consumption arena, at least for now.

University of Phoenix Stadium, already with Super Bowl prep under way

University of Phoenix Stadium, already with Super Bowl prep under way

Coincidentally, MSR on Tuesday was touring the University of Phoenix Stadium and the surrounding Westgate entertainment district, which is in the process of getting the final touches on a new complex-wide DAS installed by Crown Castle. The new DAS includes antennas on buildings and railings around the restaurants and shops of the mall-like Westgate complex, as well as inside and outside the UoP Stadium. (We’ll have a full report soon on the new DAS installs, including antennas behind fake air-vent fans on the outside of the football stadium to help handle pre-game crowds).

The University of Phoenix Stadium also had its entire Wi-Fi network ripped and replaced this season, in order to better serve the wireless appetites coming for the big game on Feb. 1. At AT&T Stadium on Monday we learned that the network there had almost 300 new Wi-Fi access points and a number of new DAS antennas installed since Thanksgiving, in anticipation of a big traffic event Monday night. Our exclusive on-the-scene tests of the Wi-Fi and DAS network found no glitches or holes in coverage, which is probably part of the reason why so many people used so much data.

UPDATE: Here is the official press release from AT&T, which basically says the same thing our post does.

Stadium Tech Report: AT&T Stadium network a winner at CFP Championship game

Inside AT&T Stadium at the College Football Championship game. Credit all photos: Paul Kapustka, MSR

Inside AT&T Stadium at the College Football Championship game. Credit all photos: Paul Kapustka, MSR

It’s late here in North Texas and you know by now that Ohio State beat Oregon to win the first non-mythical college football championship. Behind the scenes at AT&T Stadium Monday night, the wireless network in AT&T Stadium was also a winner, standing up to the challenge of the 85,000-plus crowd on both the DAS and Wi-Fi front.

We’ll have a more thorough stadium report when we get time to digest all the info we gathered at the game (and get the network stats back from the AT&T Stadium tech crew) but one thing we learned before the game was that since November, the Wi-Fi network at AT&T Stadium grew by more than 280 access points, on top of a total somewhere in the 1,200 range. According to AT&T network folks the stadium here in Arlington, Texas, has been seeing game-day totals of 3.3 Terabytes of data carried on the Wi-Fi network — leading some here to believe that Monday’s championship game could well surpass 4 TB of data used at a single game, an unofficial record as far we know for a single-day, single facility network.

As guests of AT&T we also got a quick demonstration of LTE broadcast technology, which basically slices the available cellular spectrum into a channel that can provide live streams of video. We’ll have more on this new technology in another separate report, but it is something to watch for facilities that want video options but don’t want to go whole hog on Wi-Fi.

AT&T LTE Broadcast demo, showing a live streaming broadcast of the game

AT&T LTE Broadcast demo, showing a live streaming broadcast of the game

Even though we were housed in a field-level suite your intrepid MSR crew wandered all over the massive facility, and basically found great connectivity wherever we were. Two places stick out in my mind: At the very top of the nosebleed section in the south end zone the Wi-Fi dipped to just 1 Mbps, probably because the roof is so high there is no place for an access point. However, at that same spot the AT&T 4G LTE signal was around 7 Mbps, providing great connectivity in a tough to configure spot.

The other notable spot was in a “star level” suite (about the 6th level of the building), where we got a Wi-Fi signal of 28 Mbps download and 59 (no typo!) Mbps on the upload. Yes, suite people have it better but all around wherever we went we got consistent Wi-Fi signals in the high teens or low 20s, and LTE cellular signals (including Verizon 4G LTE) just under 10 Mbps. Like the Ohio State offense, the network at AT&T Stadium works really well and may have set a new record Monday night. More soon, and more images soon as well. For now, Elvis has left the building.

Outside in the frozen tundra of North Texas, aka Arlington

Outside in the frozen tundra of North Texas, aka Arlington

This place was humming all night long

This place was humming all night long

AT&T 4G LTE speedtest, from the top of the stadium

AT&T 4G LTE speedtest, from the top of the stadium

The view from the nosebleed section

The view from the nosebleed section

Some "suite" Wi-Fi speeds

Some “suite” Wi-Fi speeds

MSR at the College Football Playoff Championships: Send us your speedtests!

ESPN's College Football Playoff Championships stage in downtown Fort Worth, Sunday night. Credit: Paul Kapustka, MSR

ESPN’s College Football Playoff Championships stage in downtown Fort Worth, Sunday night. Credit: Paul Kapustka, MSR

As Twitter followers found out yesterday, MSR is in “north Texas,” aka the Dallas-Fort Worth “Metroplex,” attending tonight’s inaugural College Football Playoff Championship game at AT&T Stadium.

We’re here to see a test of AT&T’s LTE broadcast technology, which will ostensibly make it easier for venues to deliver live video streams via a cellular connection. But we also are going to take advantage of the event to walk around “Jerry’s World” and take speed tests and see how the network at AT&T Stadium performs for a big game. If you are in attendence and know how to do a Wi-Fi or cellular speed test, send us the results (tweet at @paulkaps is the best way, or send me email to kaps at mobilesportsreport.com). Check Twitter for updates during the game.

Washington dropping Huawei for Cisco/Verizon Wi-Fi at FedEx Field, report says

Ming He, Country General Manager for Huawei in the U.S. (left), and Rod Nenner, Vice President of the Washington Redskins (right), pictured together when Huawei announced the team sponsorship and partnership.

Ming He, Country General Manager for Huawei in the U.S. (left), and Rod Nenner, Vice President of the Washington Redskins (right), pictured together when Huawei announced the team sponsorship and partnership.

According to a report from Bill Gertz at the Washington Times, the Washington, D.C. NFL franchise is apparently scrapping a recent deal with Chinese networking gear supplier Huawei to put fan-facing Wi-Fi into FedEx Field, turning instead to U.S. companies Cisco and Verizon.

Gertz, in the “Inside the Ring” column at the Times, said the Washington team’s senior vice president Tony Wyllie said in an email that “We [Washington] are in the process of deploying a stadium-wide Wi-Fi network working with Verizon and Cisco.” Gertz said the team did not elaborate on why the recent deal with Huawei was apparently scrapped before it got started.

Huawei, which claims to have installed Wi-Fi networks in many stadiums worldwide, had not had any large-scale installations at major U.S. venues before announcing the FedEx Field deal. A major competitor to large U.S. networking firms like Cisco, Huawei has been at the center of controversy in recent years, including being tabbed as a security threat by U.S. government officials, and later as a reported target for N.S.A. surveillance.

Under the announced terms of the deal, Huawei was supposed to install Wi-Fi in suite areas this December; a company spokesman said that while there was no official deal announced, Huawei was also supposed to follow that install up with a full-stadium deployment before the 2015 season started. In the initial announcement, the team announced Huawei Enterprise USA as a multi-year team sponsor and “Official Technology Partner.”

We have got calls and emails in to all the interested parties, and will update this story as we hear more.