May 28, 2016

T-Mobile Arena opens in Las Vegas, with 565 Wi-FI APs

T-Mobile Arena, Las Vegas, on the official April 6 opening. All photos: Cox Business (click on any photo for a larger image)

T-Mobile Arena, Las Vegas, on the official April 6 opening. All photos: Cox Business (click on any photo for a larger image)

If they build it, will professional hockey or basketball teams come? The first part of that question has already been answered, with the official opening of T-Mobile Arena in Las Vegas on April 6, just in time for a kickoff concert from the Killers. With 565 Cisco Wi-Fi access points, the arena is well-enabled for wireless access; the big question is, will an NHL or NBA team soon call the place home?

With no deal announced for either league it’s an open-ended question. For now, the 20,000-seat venue, located just west of the strip near the New York New York hotel/casino (right next to the freeway) will have to be satisfied hosting all kinds of events from concerts to one-off sports events like the Harlem Globetrotters (April 19) and a WWE event in June. Fans at any event will be able to use free Wi-Fi provided by Cox Business, which is the “exclusive Technology Integration/Telecommunications Services Provider,” according to a press release from the official opening ceremonies.

While we haven’t visited the arena yet — we are looking forward to a hosted tour during this summer’s SEAT Conference in July — the $375 million multi-purpose venue, owned by a joint venture between AEG and MGM Resorts International, looks pretty cool with its overhanging lounges and outdoor plaza with real, live trees, a rarity on the strip. Inside, the tech underpinnings sound state of the art, beginning with a 10-Gigabit fiber optic network that serves as the arena’s backbone.

Special shrouds for the Wi-Fi APs

Custom shroud for Wi-Fi APs at T-Mobile Arena

Custom shroud for Wi-Fi APs at T-Mobile Arena

According to figures provided to us by Cox Business folks, the 565 Wi-FI APs include a mix of indoor and outdoor models from Cisco, some designed for office-type settings and some designed to withstand outdoor temperatures and weather. According to Cox its on-site engineers also designed a “vanity cover” type of shroud, which is more aesthetically pleasing to the eye while also helping keep the AP safe from “disruptions,” like possibly being bumped or some other physical intrusion.

The arena will also use Cisco’s StadiumVision system to provide synchronized content feeds to the 767 4K-capable digital displays throughout the venue. Thanks to the Cox sponsorship, that content could include “all 60 channels of high-definition news, sports and entertainment content from the Cox cable channel lineup as well as live in-house feeds from the arena,” according to Cox.

We have also heard reports, but have not confirmed with the company, that Mobilitie will be providing the in-venue DAS. Mobilitie’s involvement is not a big surprise, given that the company partnered with MGM in the past to bring Wi-Fi to the resort company’s casinos. Back when the T-Mobile naming sponsorship was announced, there were reports of special discounts and VIP access for T-Mobile customers, but so far none of that information was easily discovered on the arena’s website. Stay tuned for more updates as we get them on the DAS/cellular side of things; anyone who visits the arena soon should take a speedtest and post the results here in the comments.

Jeff Breaux, vice president of western operations, Cox Business, (left) and Derrick R. Hill, vice president, Cox Business/Hospitality Network, gesture toward the exterior digital signage at T-Mobile Arena.

Jeff Breaux, vice president of western operations, Cox Business, (left) and Derrick R. Hill, vice president, Cox Business/Hospitality Network, gesture toward the exterior digital signage at T-Mobile Arena.

UPDATE: WrestleMania 32 sets new Wi-Fi mark at AT&T Stadium; total Wi-Fi + DAS hits 8.6 TB

The Undertaker arrives at AT&T Stadium for WrestleMania 32. Photo: WWE.com

The Undertaker arrives at AT&T Stadium for WrestleMania 32. Photo: WWE.com


UPDATE: Fixes an MSR calculation error on DAS figures.

The 101,763 fans who filled AT&T Stadium Sunday for WrestleMania 32 set new stadium records for Wi-Fi, according to figures provided by AT&T Stadium and AT&T, with 6.77 terabytes of Wi-Fi traffic and an additional 1.9 TB on the AT&T network on the stadium’s DAS for a total wireless figure of 8.6 TB.

The Wi-Fi numbers put Sunday’s signature WWE event (also the biggest WrestleMania by attendance) into second place in our unofficial record-keeping of the largest single-day Wi-Fi traffic stadium events. Only Super Bowl 50 at Levi’s Stadium earlier this year was bigger, with 10.1 TB of Wi-Fi traffic. So far, WrestleMania 32 is also now third in combined Wi-Fi and DAS figures, trailing Super Bowl 50 and Super Bowl XLIX (see charts below).

THE NEW TOP 3 TOTAL USAGE

1. Super Bowl 50, Levi’s Stadium, Santa Clara, Calif., Feb. 7, 2016: Wi-Fi: 10.1 TB; DAS: 15.9 TB; Total: 26 TB
2. Super Bowl XLIX, University of Phoenix Stadium, Glendale, Ariz., Feb. 1, 2015: Wi-Fi: 6.23 TB; DAS: 6.56 TB**; Total: 12.79 TB**
3. WrestleMania 32, AT&T Stadium, Arlington, Texas, April 3, 2016: Wi-Fi: 6.77 TB; DAS: 1.9 TB*; Total: 8.6 TB*

* = AT&T DAS stats only
** = AT&T, Verizon Wireless and Sprint DAS stats only

THE NEW TOP 5 FOR WI-FI

1. Super Bowl 50, Levi’s Stadium, Santa Clara, Calif., Feb. 7, 2016: Wi-Fi: 10.1 TB
2. WrestleMania 32, AT&T Stadium, Arlington, Texas, April 3, 2016: Wi-Fi: 6.77 TB
3. Super Bowl XLIX, University of Phoenix Stadium, Glendale, Ariz., Feb. 1, 2015: Wi-Fi: 6.23 TB
4. Alabama vs. Texas A&M, Kyle Field, College Station, Texas, Oct. 17, 2015: Wi-Fi: 5.7 TB
5. College Football Playoff championship game, AT&T Stadium, Arlington, Texas, Jan. 12, 2015: Wi-Fi: 4.93 TB

Wi-Fi not to blame for stadium entry issues

John Winborn, chief information officer for the Dallas Cowboys Football Club, said in an email that the reported claims of the Wi-Fi being offline Sunday — and that being the reason why entry lines were long and slow — were not true. While Winborn did admit that one single Wi-Fi AP (out of the more than 2,000 in the stadium’s network) was offline and there were “a couple issues” with ticket scanners, he said “there were no Wi-Fi issues that would have had a significant impact on ingress.” Other reports have claimed the doors were opened later due to extended show rehearsals, while commenters on MSR’s posts have claimed that a lack of wristbands for stadium-floor seating also led to seating issues even for fans already inside the main building entrances. So far, we have not seen any official explanation for the delays other than the official apology from the stadium and the WWE:

“To ensure the safety of WWE fans, increased security measures were put in place tonight. We apologize that it may have taken some fans longer than usual to get into AT&T Stadium.”

During Sunday’s event Winborn said the Wi-Fi network saw 20,462 concurrent and 34,951 total user connections, some via a network of 150 temporary Wi-Fi APs installed among the seats on the stadium floor.

Wi-Fi consolidation continues: Brocade buys Ruckus for $1.2 B

Screen Shot 2016-04-04 at 11.08.16 AMFollowing last year’s $3 billion purchase of Aruba Networks by Hewlett Packard, the Wi-Fi gear industry consolidation continued today with the announcement of Brocade buying Ruckus Wireless for $1.2 billion.

For the stadium-networking industry, the news may mean that Ruckus gear may find its way into more potential deals, thanks to the deeper pockets and integration possibilities brought to the table by Brocade, which competes in the back-end network gear market against companies like Cisco. Though it’s not known to have scored many stadium deals, Brocade is the back-end network supplier for Levi’s Stadium, which is a close-by neighbor to Brocade’s San Jose headquarters. It is also part of the network gear being used at the new Golden 1 Center in Sacramento, where Ruckus will provide the Wi-Fi gear, perhaps a deal where the purchase relationship got started.

Ruckus, which had come close but lost two high-profile Wi-Fi deals (for the San Jose Earthquakes’ new stadium, and for the Wi-Fi network at NRG Stadium in Houston), had said publicly that it did not participate in “pay for play” deals, where a gear manufacturer might provide discounts or payments for stadiums to use their equipment. Ruckus has won some other high-profile stadium deals recently, including at Angels Stadium in Anaheim.

Minnesota Vikings pick VenueNext for U.S. Bank Stadium app

Outside view of U.S. Bank Stadium in Minneapolis. Photo: USBankStadium.com.

Outside view of U.S. Bank Stadium in Minneapolis. Photo: USBankStadium.com.

Stadium app developer VenueNext has scored another NFL client, as the Minnesota Vikings announced today that they would use VenueNext technology in the app for the yet-to-open U.S. Bank Stadium.

According to VenueNext and the Vikings, the U.S. Bank Stadium app will support many of the same unique game-day features found in the app VenueNext built for the San Francisco 49ers’ Levi’s Stadium, including beacon-based wayfinding, the ability to order food and drinks via the app for express pickup, digital ticketing and game-day upgrade availability, as well as “robust” video content and a loyalty program tied to game-day activity. One feature at Levi’s Stadium, the ability to have food and drink delivered to fans in their seats, is “still being explored” by the Vikings, according to VenueNext.

Due to open this summer ahead of the 2016 NFL season, U.S. Bank Stadium is slated to host Super Bowl LII on Feb. 4, 2018. A Wi-Fi network with approximately 1,300 Cisco access points will supply wireless connectivity to the 66,200-seat venue, along with a neutral-host DAS built by Verizon Wireless. Aruba is supplying the 2,000 beacons being used inside the venue, and overall network operations will be run by CenturyLink, which will oversee deployment of some 2,000 digital TV displays inside the stadium.

Screenshot of U.S. Bank Vikings app in development. Image: VenueNext

Screenshot of U.S. Bank Vikings app in development. Image: VenueNext

According to VenueNext, app development partners will include Ticketmaster, Aramark for food, point-of-sale solution Appetize, seat upgrade technology from Experience, fan loyalty programs from Skidata and content app developer Adept. The Vikings are the third NFL team to choose VenueNext technology, behind the Niners and the Dallas Cowboys. VenueNext also has built a stadium app for the NBA’s Orlando Magic.

“We look forward to launching this new, dynamically-upgraded app that not only will give all Vikings fans a better experience when consuming team content on their mobile devices but also will allow seamless access to the numerous amenities at U.S. Bank Stadium,” said Vikings Owner/President Mark Wilf in a prepared statement. “Our goals are always to provide the best game day experience possible and to continue developing deeper engagement with all Vikings fans, and the VenueNext technology will help achieve both.”

“We’re excited to extend our reach in the NFL through this collaboration with the Vikings,” said John Paul, CEO & Founder of VenueNext, also in a prepared statement. “We want to become the standard for bringing Silicon Valley innovation to fan experiences, and implementing in a state-of-the-art development like U.S. Bank Stadium brings us closer to that goal.”

Interior look at U.S. Bank Stadium. Photo: USBankStadium.com

Interior look at U.S. Bank Stadium. Photo: USBankStadium.com

Entry issues snarl start of WrestleMania 32 at AT&T Stadium; crowd uses 6.77 TB of Wi-Fi

While Sunday’s WrestleMania 32 eventually ended up with a record attendance for the WWE’s signature event with 101,763 fans in attendance, some snafus with the entry process had fans reporting as long as three-hour waits to get in, with some pegging a Wi-Fi outage that disabled ticket scanners as one of the roots of the problem.

We don’t as of yet have any on-the-record comments from AT&T Stadium so we can’t say exactly what the problem was. UPDATE, 4/4/16, 2:50 p.m.: According to John Winborn, chief information officer for the Dallas Cowboys Football Club (and the person who runs the network inside AT&T Stadium), despite the rumors that emerged there was no Wi-Fi outage at the stadium — in fact Winborn said the crowd used 6.77 TB of Wi-Fi data Sunday, a single-day stadium Wi-Fi total second only to the 10.1 TB used at Levi’s Stadium for Super Bowl 50. (More details on the Wi-Fi consumption coming soon.)

Wi-Fi rumors aside, on Sunday night Twitter and Instagram were full of photos from irate fans who were stuck outside the venue, with some reportedly unable to get inside or to their seats before the actual matches started. According to a blog post at the Fort Worth Star-Telegram, the WWE and the Stadium issued a joint press release halfway through the event that read:

“To ensure the safety of WWE fans, increased security measures were put in place tonight. We apologize that it may have taken some fans longer than usual to get into AT&T Stadium.”

Several other news outlets reported the problem, with all seeming to peg the issue on a Wi-Fi problem with the ticket scanners. Winborn, however, said there were no Wi-Fi outages in the stadium or with the ticket scanners.

According to the Fort Worth Star-Telegram report, the vitriol on Twitter and other social media outlets may not have been entirely correct; according to the report:

Do not believe the viral reports and unsightly online pictures that fans could not get to their seats by the time the early matches began — they could but they preferred to walk around, buy food, drink and merchandise. But the images and the complaints that flooded the Internet about #WrestleMania32 were so plentiful, and negative, joint press release was issued halfway through the event.

With 101,763 finally in their seats the event was the fifth-largest crowd inside AT&T Stadium, and as such most likely produced a large Wi-Fi and DAS traffic number once things finally got underway. We will update this post and have another when we get the wireless traffic stats from the stadium.

Betting the Under (Part 2): Putting Wi-Fi antennas under seats is the hot new trend in stadium wireless networks

Under-seat Wi-Fi AP at Levi's Stadium. Photo: Paul Kapustka, MSR (click on any photo for a larger image)

Under-seat Wi-Fi AP at Levi’s Stadium. Photo: Paul Kapustka, MSR (click on any photo for a larger image)

Part 2 of this story picks up with the decision to put Wi-Fi APs under seats at Levi’s Stadium. If you missed it, here is the link to Part 1.

According to Chuck Lukaszewski, now vice president of wireless strategy and standards at Hewlett Packard Enterprise (formerly very high density architect in the CTO Office of Aruba Networks), Aruba had been testing under-seat AP designs since around 2010, “in one form or another.” There were some initial tests of under-seat AP deployments at Turner Field in Atlanta and at American Airlines Arena in Dallas, but nothing on the scale of AT&T Park’s 2013 deployment, or on the scale Aruba planned to have at Levi’s Stadium when it opened in 2014.

Some of the first under-seat Wi-Fi deployments in other arenas were actually deployed completely under the stands, Lukaszewski said, with signals shooting up through the concrete. Though he said “you could get reasonably good throughput through concrete,” especially for 2.4 GHz frequencies, installing antennas above the concrete was “considerably better,” Lukaszewski said.

Curiously, one of the biggest problems in stadium Wi-Fi deployment — especially for those heavy on overhead antenna use — is negotiating interference between antennas; sometimes, clients can “see” antennas and APs that are across the stadium, and will try to connect to those instead of the AP closest to them, a problem that leads to inefficient bandwidth use. Interference also means you can’t place APs too closely together, making it somewhat of an art to find ways to increase coverage without increasing interference.

Dan Williams, former VP of technology for the San Francisco 49ers, talking networking at Levi's Stadium. Photo: Paul Kapustka, MSR

Dan Williams, former VP of technology for the San Francisco 49ers, talking networking at Levi’s Stadium. Photo: Paul Kapustka, MSR

What Aruba found in its testing, Lukaszewski said, was that under-seat Wi-Fi AP deployments could be far more dense than overhead-centric designs, mainly because the human bodies in the seats would provide beneficial “blocking” of signals, allowing network designers to place APs more closely together, and to be able to re-use the same Wi-Fi channels in more antennas.

“If you can use human bodies to contain signals, you can have much smaller cells,” Lukaszewski said. Under-seat deployments, he said, “allows us to re-use the same channel less than 100 yards away.”

With more channels available for each AP, the difference in the metric Lukaszewski calls “megabytes per fan” can be “profound” for an under-seat design versus an overhead design, he said.

“We do see trends [in stadium network data] of under-seat being able to deliver well over 100 MB per fan per event, while overhead designs [deliver] significantly under 100 MB per fan per event,” said Lukaszewski.

Dan Williams, the former vice president of technology for the San Francisco 49ers, said he and Lukaszewski were in agreement that under-seat was the best method to deploy at Levi’s Stadium.

Kyle Field at Texas A&M. White spots in stands are under-seat AP locations. Photo: Paul Kapustka, MSR

Kyle Field at Texas A&M. White spots in stands are under-seat AP locations. Photo: Paul Kapustka, MSR

“I just did not believe in overhead,” said Williams, who said he brainstormed with Aruba’s Lukaszewski on the under-seat idea, which they both brought to the Wi-Fi design at Levi’s. By using under-seat APs, Williams said, the Levi’s Stadium design looked to provide “cones [of bandwidth] around the audience, immersing [fans] in a signal.”

After beating the previous year’s Super Bowl Wi-Fi total at its NFL regular-season opener in 2014, Levi’s Stadium’s Wi-Fi network more than passed its biggest test ever this year, carrying a record 10.1 terabytes of Wi-Fi data during Super Bowl 50. Those numbers are proof of Lukaszewski’s claim: “By far, under seat is better.”

New deployments trending to under-seat

Editor’s note: This excerpt is from our latest STADIUM TECH REPORT, our long-form PDF publication that combines in-depth stadium tech reports with news and analysis of the hottest topics in the world of stadium and large public venue tech deployments. Enjoy this PART 1 of our lead feature, or DOWNLOAD THE REPORT and read the whole story right now!

Even though under-seat deployments can be considerably more expensive, especially in a retrofit situation where deployment requires coring through concrete, many stadiums are now seeming to agree with another Lukaszewski claim, that “the return absolutely justifies the investment.”

At AT&T Stadium in Arlington, Texas, the Cowboys quicked followed their sister park’s lead and installed under-seat APs in force ahead of that venue’s hosting of the inaugural College Football Playoff championship game in January of 2015. John Winborn, chief information officer for the Dallas Cowboys Football Club, said the team worked with AT&T’s “Foundry” innovation centers to produce a smaller, sleeker under-seat AP enclosure that fit well with the stadium’s commitment to aesthetics.

Back on the baseball side, the Giants now have 1,628 Wi-Fi APs in their park, with the vast majority of them under-seat, in all three decks of seating. And the Giants’ main rival to the south, the Los Angeles Dodgers, also used under-seat APs in a recent Wi-Fi upgrade.

Close-up of conduit running to under-seat AP at Kyle Field. Photo: Paul Kapustka, MSR

Close-up of conduit running to under-seat AP at Kyle Field. Photo: Paul Kapustka, MSR

And if Levi’s Stadium led the way for under-seat Wi-Fi, the new mainly under-seat network at the refurbished Kyle Field at Texas A&M might be the QED on the debate, with ultra-fast network speeds and big data-consumption numbers (including 5.7 TB of Wi-Fi at a game versus Alabama) adding measureable momentum to the under-seat trend. Bill Anderson, CEO of Wi-Fi deployment strategy firm AmpThink, said he was an early disbeliever in under-seat Wi-Fi — until he saw the numbers.

“At first we mocked it, made fun of it,” said Anderson, whose firm has been called in to produce Wi-Fi network designs for several recent Super Bowls, as well as for the Kyle Field design. But when Aruba showed AmpThink the data from under-seat tests and deployments, “that was the ‘a-ha’ moment for us,” Anderson said.

Working with Aruba at Kyle Field, AmpThink was able to collect its own data, which convinced Anderson that under-seat was the way to go if you wanted dense, high-performing networks.

“The really important thing is to get APs closer to the people,” said Anderson. “That’s the future.”

Anderson said some doubters may remain, especially those who try to mix a small amount of under-seat APs with existing overhead deployments, a recipe for lowered success due to the potential interference issues. At Texas A&M, Anderson said AmpThink was able to build a design with far less interference and much greater density than an overhead solution, producing numbers that people have to pay attention to.

“We only know what we’ve observed, but we’re evangelistic supporters” of under-seat designs, Anderson said. “If someone says to you under-seat is hocus-pocus, they’re not looking at the data.”

Not for everyone, but more are trying under-seat

Though proponents of under-seat Wi-Fi all agree on its ability to deliver denser, faster networks, they all also agree that under-seat can be considerably more costly than overhead Wi-Fi, especially in a retrofit situation.

In addition to having to core through concrete seating areas to get conduit to the under-seat APs, the devices themselves need to be sealed, to guard them from weather, drink spills, and the power-washing equipment employed by most stadiums to clean seating areas.

Aruba’s Lukaszewski also noted that under-seat deployments generally use more linear feet of cabling to connect the APs than overhead, which also drives up the cost. Then since under-seat designs tend to use more APs, that also means a higher budget to cover a higher number of devices.

A row shot of the under-seat APs at AT&T Stadium. Photo: Dallas Cowboys

A row shot of the under-seat APs at AT&T Stadium. Photo: Dallas Cowboys

For some stadiums, the construction materials used prohibit the under-seat option from even being tried. At the Green Bay Packers’ legendary Lambeau Field, a late-1950s construction design that used lots of concrete and rebar — as well as part of the stadium’s bottom sitting directly in the ground — meant that under-seat Wi-Fi wasn’t an option, according to Wayne Wichlacz, director of information technology for the Packers.

Other stadiums, like the University of Nebraska’s Memorial Stadium, don’t have enough space between the stadium’s bleacher seats and the floor for under-seat APs to be safely installed. And many schools or teams simply don’t have big IT budgets like the $20-million-plus available to Texas A&M that allowed the Kyle Field design to seek the best result possible.

But many of the new stadiums under construction, as well as existing venues that are planning for new best-of-breed networks, have already committed to under-seat Wi-Fi designs, including the Sacramento Kings’ Golden 1 Center, where Ruckus Wireless will implement its first under-seat stadium Wi-Fi network.

Steve Martin, senior vice president and general manager at Ruckus, said the Golden 1 Center design, planned to be the most dense anywhere, will “primarily be underseat,” a choice he said “helps in a lot of ways.”

Foremost is the performance, something Martin said Ruckus has been testing at the Kings’ current home, the Sleep Train Arena. “It [under seat] does give you the isolation for frequency re-use,” he said.

The under-seat design also makes sense in Golden 1 Center since the stadium’s overall design is very open, with lots of glass walls and unobstructed views.

And under-seat deployment is even making inroads into the distributed antenna system (DAS) world, with Verizon Wireless implementing more than 50 under-seat DAS antennas at Levi’s Stadium prior to Super Bowl 50. Mainly installed to cover the bottom-of-the-bowl rows, the under-seat APs helped Verizon manage a record day for DAS traffic, with 7 TB reported on its in-stadium cellular network during the game.

“To get a quality signal, we had to go under seat,” said Brian Mecum, vice president, network, for Verizon Wireless, who said that in that area of the stadium, under seat was the only way to get a quality signal close to the subscriber’s phone. Verizon, he said, helped design the under-seat DAS antenna, and is looking to deploy it in other stadiums soon.

“It’s the first of more,” he said.

END PART 2… HERE IS THE LINK TO PART 1… TO READ THE WHOLE STORY NOW, DOWNLOAD OUR REPORT!