Commentary: Super Bowl 50 and the pursuit of stadium network statistics

Dan Williams talks Wi-Fi while the Levi's Stadium new turf grows silently behind him in this 2014 photo. All photos: Paul Kapustka, MSR

Dan Williams talks Wi-Fi while the Levi’s Stadium new turf grows silently behind him in this 2014 photo. All photos: Paul Kapustka, MSR

If there’s one person who didn’t get enough recognition in the run-up to today’s Super Bowl 50 at Levi’s Stadium, it’s Dan Williams, the former director of information technology for the Niners and the man who was largely behind the venue’s celebrated network design.

From our standpoint, Williams deserves credit not just for leading the deployment of a stadium network that set the standard for the future of connected venues, but also for being among the first to openly talk about the network’s performance, sharing statistics both good and bad during his short stay at Levi’s Stadium. His honesty helped open the door wider on a potentially rich stream of information that we think could greatly assist stadium network professionals everywhere, especially if more teams followed his lead and shared stats openly and honestly.

But the pursuit of meaningful network statistics remains a challenging process on many levels, and it’s one that is ironically likely to get more confusing today as multiple parties are set to release live network statistics from Super Bowl 50 at Levi’s Stadium. Both Verizon Wireless and AT&T are planning to release live game-day performance figures for their cellular networks in and around the venue today, and we also expect to get Wi-Fi network statistics from the San Francisco 49ers, if not during the game then shortly thereafter.

Niners VP of technology Dan Williams attempts to fix my Droid 4 Wi-Fi issues (while trying not to laugh at the fact that I actually have and use a Droid 4) during the first preseason game at Levi's Stadium, July 2014

Niners VP of technology Dan Williams attempts to fix my Droid 4 Wi-Fi issues (while trying not to laugh at the fact that I actually have and use a Droid 4) during the first preseason game at Levi’s Stadium, July 2014

While we salute the change of heart (especially from Verizon) to make more information available, we’d also like to make a plea here for more independent access to venues on game day, so there can be an independent, objective voice on hand to counter the “sanitization” or simple non-reporting of less-impressive data that is almost sure to occur when interested parties are talking about themselves. But unless team, league or venue IT representatives band together and craft recognized standard methods of reporting the most-interesting data, we’re going to be left to sift through an ever-increasing mix of figures provided by those with their own agendas on what’s important and what should remain hidden. We’ll do our best on that front, but it’d be better if we had some industry-wide help.

Out of school behavior provides the most interesting info

Like many other stadium network professionals we’ve been fortunate enough to meet and spend time with — a list here that includes folks like the San Francisco Giant’s Bill Schlough, Madison Square Garden’s Katee Panter, the Dallas Cowboys’ John Winborn, and Chip Foley, formerly with the Barclays Center, among many others — Williams from the start showed a passion for making the network better, so it could better serve the fans who used it. Our first face to face meeting came during the first preseason football game at Levi’s Stadium, when I tweeted that I was having problems connecting to the Wi-Fi network. William’s response? An email asking me where I was sitting, followed quickly by an in-person visit. Now that’s customer support.

The outcome of that interaction was a revelation that the Levi’s Stadium Wi-Fi network hadn’t prepared adequately for older devices that were still using the 2.4 GHz unlicensed bands; what was cool to me was not just that I kind-of discovered it by accident, but that Williams and his team admitted it publicly, then went about fixing it. The combination of objective reporting and honest confirmation, I think, worked to the benefit of all. But not everybody sees such things in the same light.

Season opener issues: Picture of app late in the first half.

Season opener issues: Picture of app late in the first half.

For instance, during the rest of Levi’s Stadium’s first year of operation, Williams shared with us not just raw “tonnage” numbers of Wi-Fi use but also interesting stats from the app, like exact numbers of replays watched and in-seat food delivery orders. Since those numbers were kind of low — in the low thousands for food deliveries and a decline in replay use after the season opener — they weren’t necessarily flattering to the owners of the system, even if you took into account the fact that some of this (like delivering to any seat in a 68,500-seat stadium) had never been tried before. In the end, even Niners CEO Jed York admitted that some elements of the stadium hadn’t performed as expected, as the stats we’d been seeing had borne out.

After Williams left the team, the Niners took control of the network stats reporting and hugely trimmed down the list of things they were willing to report, though we would also note that Roger Hacker and the Niners communications team are still at the top of the list when it comes to consistent, repeated reporting of the network stats they feel comfortable talking about. When there are big numbers to report, all goes well because everyone loves to talk about systems that deliver for fans as promised. But I can also tell you that when things go south, so does a lot of the honest communication, an unfortunate if understandable situation.

Who will tell you when things don’t work?

For Levi’s Stadium, the darkest night for the network came on Feb. 21, 2015, at the NHL’s Coors Light Stadium Series outdoor hockey game between the San Jose Sharks and the Los Angeles Kings. In what has easily been our highest-traffic post ever, we described in detail some of the big problems that surfaced that evening, which included breakdowns of the delivery service, the Wi-Fi network, the light rail boarding process, and part of Verizon’s cellular network. What’s funny about our reporting is that I hadn’t planned on “working” that night — I had bought full-price tickets for myself and my brother, and as longtime hockey nuts we proudly wore our Blackhawks jerseys to represent our hometown team.

Screen shot of Levi's Stadium app during hockey game issues

Screen shot of Levi’s Stadium app during hockey game issues

My plan was just to relax, take CalTrain and VTA to the stadium and show off the network by ordering us food and drink to be delivered to our seats, but when some obvious and incomprehensible network breakdowns were evident, I went into reporting mode, and eventually found out mostly what went wrong. My point here is that, without independent, objective reporting live from the scene, it’s doubtful that parties with vested interests — like cellular providers or owners of the stadium network — are going to honestly and quickly talk about breakdowns or failures. Yet it’s always the lessons learned from problem times that help the most; my worry is that with more “sponsored” or in-house statistics flooding the zone, the more valuable objective data will get lost or devalued.

How can this be fixed? For starters, teams and leagues that spend an inordinate amount of time and money promoting technical features of stadiums should at the very least credential some reporters who are interested in following up to see how said deployments perform. While we’re grateful for the few forward-thinking organizations that have given us the same access that sports reporters get, more often than not our requests for media access to events are denied.

Since we can’t be at every stadium, we also rely on teams, venues and network operators to provide stats when possible, and we encourage more participants to join the growing list who do provide us with network performance figures, like the IT teams at Nebraska and Texas A&M, among others. While we have also drawn some industry criticism for our headline focus on such stats, our response is that the more data we get and the more opinions on which data matter, the better we will get over time. We also encourage any and all interested parties to attend the SEAT Conference this summer in Las Vegas, where you can be sure we will have discussions about this topic and how to make stats reporting more useful for those to whom it matters most — the professionals who install and operate these networks so that fans can stay connected at events.

MSR on the scene at Sports Authority Field at Mile High in January

MSR on the scene at Sports Authority Field at Mile High in January

AT&T: NFL fans used 55% more DAS data this year

dx1AT&T customers who visited NFL stadiums this season used 55 percent more cellular traffic this year than last, according to some year-end figures from AT&T.

In the 31 different NFL venues where there is an AT&T DAS AT&T customers used 132.8 terabytes of cellular data this NFL season, with the Dec. 14 Monday night game between the New York Jets and the Miami Dolphins topping the single-game charts with 1.6 TB of DAS data used, according to AT&T. It’s appropriate that Sun Life Stadium had the biggest data game, since Miami’s home also led the NFL for highest average DAS data used, with 1.4 TB per game. Close behind in second place for average DAS use was AT&T Stadium in Arlington, Texas, where the average hit 1.257 TB this season. Third was San Diego’s Qualcomm Stadium with 1.085 TB, and fourth was Levi’s Stadium in Santa Clara, Calif., with an average of 1.054 TB each game on the AT&T DAS.

NFL Stadium Tech Reviews — NFC East

Editor’s note: The following team-by-team capsule reports of NFL stadium technology deployments are an excerpt from our most recent Stadium Tech Report, THE PRO FOOTBALL ISSUE. To get all the capsules in one place as well as our featured reports, interviews and analysis, download your free copy of the full report today.

NFC EAST

Reporting by Paul Kapustka

Dallas fan in mobile action at AT&T Stadium. Photo: Phil Harvey, MSR

Dallas fan in mobile action at AT&T Stadium. Photo: Phil Harvey, MSR

Dallas Cowboys
AT&T Stadium
Seating Capacity: 105,121
Wi-Fi – Yes
DAS – Yes

AT&T Stadium continues to push the envelope when it comes to wireless connectivity, perhaps no surprise given the sponsor name on the building. For 2015 the venue will have up to 2,000 Wi-Fi APs at any given time (1,900 fixed and 100 flexible for deployment where needed), making it by far the stadium with the most access points.

New York Giants
MetLife Stadium
Seating Capacity: 82,500
Wi-Fi – Yes
DAS – Yes

Like with the Jets, the Giants and their fans are still benefitting from the technology installed for the Super Bowl two years ago, with superb Wi-Fi and DAS coverage throughout the facility.

Philadelphia Eagles
Lincoln Financial Field
Seating Capacity: 69,176
Wi-Fi – Yes
DAS – Yes

One of the early Extreme deployments, the “Linc” will continue to be well-connected with Wi-Fi available to all seating areas.

Washington Redskins
FedExField
Seating Capacity: 79,000
Wi-Fi – Yes
DAS – Yes

After briefly considering Chinese firm Huawei for Wi-Fi last season, the Redskins pulled a reverse and instead went for a deployment led by Cisco and Verizon, now apparently with United Airlines as a service sponsor.

Tap.in2 scores food-delivery deal for Cincinnati Bengals’ club seats; could more YinzCam deals follow?

Screenshot of Tap.in2's food ordering and delivery service embedded in the Cincinnati Bengals' team app. (Click on any photo for a larger image) Credit: Tap.in2

Screenshot of Tap.in2’s food ordering and delivery service embedded in the Cincinnati Bengals’ team app. (Click on any photo for a larger image) Credit: Tap.in2

Startup Tap.in2 has signed up the Cincinnati Bengals as its second big-league client for its mobile-app service that enables in-seat food and beverage service in stadiums, with a deal to bring app-based deliveries to 8,000 club-level seats at Paul Brown Stadium this season.

Expected to be formally announced today, the deal has actually been in place all season, according to Tap.in2 representatives. The deal follows Tap.in2’s breakout contract with the NBA’s Cleveland Cavaliers to bring similar services to the lower bowl of Quicken Loans Arena. A product of a Cleveland-area startup incubator, Tap.in2 partnered for the second time with Aramark to facilitate the delivery service, which offers a full menu of stadium food and beverage for in-seat delivery.

And while the Bengals are no longer undefeated (losing 10-6 to the Houston Texans on Monday Night Football) select fans at Paul Brown can at least enjoy in-seat concession delivery for the remaining games this year by ordering directly from the team’s stadium app. To our knowledge it’s only the second NFL team to offer app-based food and beverage delivery services, following the San Francisco 49ers and their VenueNext-powered app which supports in-seat delivery to every seat in the 68,500-seat Levi’s Stadium.

Let the food-delivery app battles begin

beng1What’s interesting about the Bengals deal is that it has Tap.in2 melding its services with an app built by sports-app giant YinzCam, which does not offer a food-delivery option in the current version of its app used by many NFL teams. However, YinzCam did just sign a big deal with the NBA to re-do 22 NBA team apps, with the option of adding concession delivery services mentioned in the press release; however, YinzCam has not yet stated publicly how it would add such services to its core stadium-app product. Could more deals with Tap.in2 be on the YinzCam horizon?

The well-funded VenueNext, meanwhile, has signed new deals with the NBA’s Orlando Magic as well as the New York Yankees and the Dallas Cowboys, to bring more VenueNext features (possibly including food delivery) to those teams’ stadium apps. While some VenueNext features have already crept into the AT&T Stadium app for this season, food ordering and delivery to seats is not yet available at that venue. VenueNext will also provide the app for Super Bowl 50, which will be held at Levi’s Stadium in February.

Though Tap.in2 has not released any actual figures about how many orders were actually taken at games this season, it does claim to have positive feedback from the fans who have used the service, and did claim that orders were being delivered in less than 5 minutes, on average. VenueNext, which did release some food-delivery numbers from Levi’s Stadium last season, is no longer making those statistics available. However the company did say that its app brought in nearly $800,000 in revenue last season, which may give you some idea why this service is hotter than a hot dog when it comes to increasing revenue inside stadiums.

Levi’s Stadium, AT&T Stadium see lots of Wi-Fi for Seahawks visits

Seahawks vs. Cowboys at AT&T Stadium, Nov. 1. Photo: Dallas Cowboys

Seahawks vs. Cowboys at AT&T Stadium, Nov. 1. Photo: Dallas Cowboys

So far this NFL season we’ve seen something that we call the Patriots effect, where games featuring the defending Super Bowl champs as visitors produce big numbers on the stadium Wi-Fi networks. There appears to be a similar trend following the Seattle Seahawks around, especially when they’re playing NFC rivals like the San Francisco 49ers and the Dallas Cowboys, as both those teams’ stadiums saw big Wi-Fi days during recent Seahawks visits.

The Niners were first to get a Seahawks effect, carding 2.2 terabytes of Wi-Fi network usage during Seattle’s 20-3 victory over San Francisco at Levi’s Stadium on Oct. 22. According to figures provided by Roger Hacker, senior manager of corporate communications for the Niners, out of the 70,799 in attendance for the Thursday-night game there 16,299 unique users on the Wi-Fi network at Levi’s Stadium, with a maximum concurrent user number of 10,306.

Screen Shot 2015-11-03 at 10.16.54 PM

Last Sunday at AT&T Stadium, the Seahawks escaped with a narrow 13-12 win in front of 91,486 fans. According to figures provided by Dallas Cowboys chief information officer John Winborn, there were 36,388 unique users on the Wi-Fi network, with a peak of 24,667 concurrent users. The total tonnage used Sunday at AT&T Stadium was 4.12 TB, perhaps proving once again that 4+ TB is becoming the “new normal” for high-fidelity networks in the largest stadiums.

Keeping the Wi-Fi hidden: AT&T Stadium perfects the art of Wi-Fi AP concealment

Wi-Fi antennas visible under the 'shroud' covering the outside of the overhang at AT&T Stadium. Photo: Dallas Cowboys (Click on any photo for a larger image)

Wi-Fi antennas visible under the ‘shroud’ covering the outside of the overhang at AT&T Stadium. Photo: Dallas Cowboys (Click on any photo for a larger image)

Since they like to do everything big in Texas, it’s no surprise that the IT team at AT&T Stadium has taken the art of Wi-Fi access point concealment to new heights.

To just above the first and second seating levels of the stadium, that is.

Even though the venue has more Wi-Fi APs than any sports stadium we’ve ever heard of, trying to find any of the 1,900 permanently installed APs is a tough task, thanks to measures like the fiberglass shrouds that circle the stadium just above the first and second seating levels. Underneath those custom-built coverings are numerous Wi-Fi APs, DAS antennas and even cameras, all contributing to the high level of connectivity inside AT&T Stadium while remaining invisible to the visiting fan’s eyes.

“The philosophy throughout the stadium is for a clean, stark look,” said John Winborn, chief information officer for the Dallas Cowboys Football Club, which is the primary tenant of the venue. “That’s a high standard, and that is a real challenge for us when it comes to Wi-Fi and DAS.”

In just about every stadium network deployment we write about, concealment and aesthetics are always one of the top concerns, especially when it comes to Wi-Fi access points and DAS antennas. For some reason, the physical appearance of an obvious piece of technology evokes strong reactions, even as other necessary structural items are ignored.

(Editor’s note: This story is an excerpt from our most recent Stadium Tech Report, the PRO FOOTBALL ISSUE, which is available for FREE DOWNLOAD right now from our site. In the report our editorial coverage includes a profile of the new Wi-Fi network at Green Bay’s Lambeau Field and team-by-team profiles of Wi-Fi and DAS deployments at all 31 NFL stadiums. Get your copy today!)

Under-seat Wi-Fi AP at AT&T Stadium. Photo: Dallas Cowboys

Under-seat Wi-Fi AP at AT&T Stadium. Photo: Dallas Cowboys

A clean, sleek look at the house that Jerry built

As one anonymous commentator at this summer’s SEAT conference noted, “stadium supervisors don’t ever care about seeing a 4-inch pipe, but leave one antenna out and they go crazy.” And whoever that stadium person is, he or she probably has a kindred soul in Dallas Cowboys owner Jerry Jones.

An unseen antenna is Jerry Jones’ favorite kind. Winborn said that AT&T Stadium embraces design in all things visible, noting that the “clean look” idea extends to advertising inside the seating bowl, where the only permanent signs are located in the end zone areas.

“We’re very conscious of the aesthetics here,” Winborn said. “Everyone here sees the benefit of what a great looking building can be. And it all starts with [Jerry] Jones.” Jones’ ideas, Winborn said, “are a major influence on everything we do.”

What that means when it comes to Wi-Fi is that while the stadium always aims to be the best-connected venue around – for this football season, AT&T Stadium will have 1,900+ permanent Wi-Fi APs and another 100 or so available for temporary placements – it also aims to hide the physical gear as much as possible. In suites and hallways there is the natural solution of putting antennas behind ceiling panels, but in the seating bowl, Winborn said, “we don’t have a lot of areas to hide them. We’ve had to become pretty clever about ways to hide APs.”

Two years ago, when the stadium’s AP count was going up from 750 to 1,250, the idea came about to design a custom fiberglass “shroud” that would circle the arena on the front of the overhangs above the first and second seating levels.

A row shot of the under-seat APs. Photo: Dallas Cowboys

A row shot of the under-seat APs. Photo: Dallas Cowboys

“We’re very conscious of the aesthetics here,” Winborn said. “Everyone here sees the benefit of what a great looking building can be.

Winborn said one of the IT staff members had contacts in the manufacturing world, which helped the Cowboys build a slighly convex design that wouldn’t be readily apparent to the untrained eye, yet be big enough to house Wi-Fi and DAS gear all the way around the bowl.

Winborn said the shroud and its underlying gear were installed during one of the recent off-seasons, taking about a couple months – and the final result was so good that Winborn says he needs to use a laser pointer to show interested parties exactly where the equipment shroud sits. Since it’s fiberglass the shroud is somewhat easy to move to allow administration and maintenance of the equipment, but the seamless flow of the structure around the bowl may just be the most elegant AP hiding strategy in the short history of stadium Wi-Fi.

But even with the shrouds there was still a need for more new placements, especially in the middle of the open seating areas. So last year the AT&T Stadium team started deploying under-seat AP enclosures, working with design teams at the AT&T Foundry program to build a custom unit that is much smaller and unobtrusive than other under-seat AP enclosures currently in use.

“We worked with the AT&T Foundry and went through [testing] about a half-dozen models,” Winborn said, before finally arriving at a design that worked well and stayed small. “It’s about the size of a small cigar box,” said Winborn of the under-seat APs, 300 of which were installed in the 100-level seating last year. Another 250 are being installed for this year up in the 300-level seating, he said.

Winborn credited the early use of under-seat APs by the IT team at AT&T Park in San Francisco as a welcome guide.

Here's the big bowl that needs to be filled with Wi-Fi. Photo: Paul Kapustka / MSR

Here’s the big bowl that needs to be filled with Wi-Fi. Photo: Paul Kapustka / MSR

“I talked to the Giants and Bill [Schlough, the Giants’ CIO] and had my concerns” about under-seat APs, Winborn said. “But after they did it and had only one complaint in 2 years, that raised my comfort level.”

Like the Giants’ under-seat APs, the ones in AT&T Stadium are designed to be as maintenance-free as possible, so that they can be steam washed and not harmed by spills or any other physical interactions. Winborn said the Cowboys have even started putting sealant and paint over the top of the under-seat APs, “so they look just like a bump.”

With 1,900 to 2,000 APs available, it might seem like the AT&T Stadium IT crew has enough APs for now, so they can relax a bit when it comes to finding new ways to hide Wi-Fi gear. But Winborn knows the next surge is probably right around the corner (including early results from this season showing 4+ terabytes of Wi-Fi use).

“Everything we are giving the fans [in Wi-Fi bandwidth] they are gobbling it up, pretty quickly,” Winborn said.

https://duwit.ukdw.ac.id/document/pengadaan/slot777/

https://mtsnupakis.sch.id/wp-content/zeusslot/

https://insankamilsidoarjo.sch.id/wp-content/slot-zeus/

https://smpbhayangkari1sby.sch.id/wp-content/slot-zeus/

https://alhikamsurabaya.sch.id/wp-content/slot-thailand/

https://mtsnupakis.sch.id/wp-content/bonus-new-member/

https://smptagsby.sch.id/wp-content/slot-bet-200/

https://lookahindonesia.com/wp-content/bonus-new-member/

https://ponpesalkhairattanjungselor.sch.id/wp-content/mahjong-slot/

https://mtsnupakis.sch.id/wp-content/slot777/

https://sdlabum.sch.id/wp-content/slot777/

https://sdlabumblitar.sch.id/wp-content/bonus-new-member/

https://sdlabumblitar.sch.id/wp-content/spaceman/

https://paudlabumblitar.sch.id/wp-content/spaceman/