Wi-Fi upgrade producing solid results for Denver Broncos at Mile High

A fan walks by a railing wireless enclosure in the upper deck of Broncos Stadium at Mile High during the Oct. 1 game against the Kansas City Chiefs. Credit all photos: Paul Kapustka, MSR (click on any picture for a larger image)

As the Denver Broncos’ Wi-Fi network upgrade nears its final steps of completion, solid coverage around the venue now known as Broncos Stadium at Mile High is producing Wi-Fi data totals averaging more than 6 terabytes per game, according to statistics from the team.

During a recent game-day visit to Mile High, Mobile Sports Report got consistent high-bandwidth readings for Wi-Fi throughout the venue, and into the parking lots as well. Multiple speed tests recorded bandwidth marks in the high double-digits of megabits per second, even at the top reaches of the stands as well as in other hard-to-cover areas, like concourses and plazas.

And even as Russ Trainor, Broncos’ senior vice president for information technology, and his networking team put the final tuning touches on an expansion that will end with somewhere near 1,500 Cisco Wi-Fi APs installed throughout the building, the football (and concert) fans who have shown up lately are already finding ways to use lots of Wi-Fi data. In the first three home games of the Broncos’ current regular season, Trainor said the Wi-Fi network has seen total single-day usage numbers of 6.4 TB, 6.3 TB and 6.2 TB, the latter coming during the exciting Monday Night Football game Oct. 1 versus the Kansas City Chiefs.

More APs coming for gate areas, concourses

“We still have a few more APs to add,” said Trainor in a quick interview during the Chiefs game, which MSR attended. And while Trainor added that the team is also planning to step up its promotion of the network, many fans are finding it already, as proven by some other high-water marks this year that include a peak of 32,837 concurrent users during the home opener on Sept. 9; peak throughput of 10.83 Gbps on Sept. 16; and the most unique connections, 42,981, on Oct. 1.

Parking lots are well-covered at Mile High

Because many of the new APs are the new Cisco 3800 Series with two radios, Trainor is confident the Broncos Stadium network is far from maxing out.

“We still have room to grow folks onto the system, and we’ll continue to advertise that network for the fans,” Trainor said.

During our visit at the Oct. 1 game, MSR was impressed the moment we got out of our car in the parking lot, when we recorded a Wi-Fi mark of 28.3 Mbps down and 56.5 Mbps up. As a Verizon customer we were automatically connected to the stadium’s Wi-Fi network, one of the perks that came with Verizon’s investments in the Wi-Fi and DAS networks at the stadium.

Inside the premium-seating United Club area, we got a Wi-Fi mark of 48.0 Mbps / 70.3 Mbps, even as fans crowded the open dining hall during pregame. We also saw some cool new food-station kiosks along one wall, each with its own connected display for menu items as well as a touchscreen payment system (a turnkey deployment from Centerplate, Tapin2, and PingHD) that eliminated the need for additional concessions staffers.

Up on the top-level concourse we saw APs every other wall section with two antennas pointing in opposite directions, coverage that produced one mark of 31.8 Mbps / 68.2 Mbps even as fans crowded the stands to get food and drink before kickoff. According to Trainor the concourse areas will get roughly a doubling of coverage with more APs next year, to support a plan to move to more digital payment methods.

A good look at the hardened, single-cable Wi-Fi APs in the walkway ramps area. According to the Broncos these use POE (power over Ethernet), cutting down on the conduit needed.

Out in the upper-level stands (Section 541, row 5) we got a Wi-Fi mark of 36.0 Mbps / 29.6 Mbps, in an area where we could see APs pointing down on the seats from the top-level light standards as well as in railing enclosures. Some areas in the upper deck are also covered by under-seat APs, which also are used in the south end zone stands where there is no overhang infrastructure.

We also got good connectivity in an often overlooked area, the walkway ramps and escalators behind the seats, where the Broncos installed some APs that use power over Ethernet and weather-hardened enclosures since those areas are more open to weather. While riding up on an escalator we not only stayed connected but got a test mark of 26.4 Mbps / 37.6 Mbps.

Keeping crowds of fans connected

In perhaps one of the biggest stress tests we could find, the Mile High Wi-Fi had no problem keeping fans connected. Just before halftime we planted ourselves on the outdoor plaza behind the south stands, and waited for fans to crowd the area during the break. With a Wi-Fi mark of 38.4 Mbps / 35.7 Mbps second five minutes into the halftime break, we were still able to easily view video highlights of the first half even as everyone around us was using their phones to check email or to connect with friends and family.

As the second-half kickoff neared, we walked into the main concourse underneath the west stands and still stayed solidly connected, with a mark of 33.0 Mbps / 59.1 Mbps in the middle of a thick crowd of fans who were either waiting for concessions or walking back to their seats.

With a high-water mark of 8.1 TB for a Taylor Swift concert earlier this spring, the new Wi-Fi network in Broncos Stadium at Mile High showed that it’s more than ready for big games or other big events. Some more photos from our visit below!

Nothing like Monday Night Football!

Fans gather on the south stands plaza during halftime

Close-up of an AP install on the back wall facing out into the south stands plaza

United Club dining area with single-stand kiosks in back

Single-stand food kiosk with its own display and self-service payment terminal (from PingHD)

AP deployment on top-level concourse

AP deployment (on post) in lower concourse area

Notre Dame’s new Wi-Fi, Mercedes-Benz Stadium first look — all in our new Stadium Tech Report!

We always get excited here at Mobile Sports Report when we have a new quarterly report out, but the stories, profiles and analysis in our Fall 2017 issue just may be our best-ever effort. With a detailed look at the new Wi-Fi network at Notre Dame Stadium, and a first look at the Atlanta Falcons’ new Mercedes-Benz Stadium, our Fall 2017 issue starts off with a doubleheader of deep information profiles and it doesn’t stop there!

In addition to Notre Dame and Mercedes-Benz Stadium, this issue also has a detailed look at the new football stadium at Colorado State University, which also has high-performing Wi-Fi and a neutral-host DAS deployment. We also take a look at the Wi-Fi renovation taking place at the Denver Broncos’ Sports Authority Field at Mile High, a network upgrade that should lift the Broncos’ home to the top of the list of NFL stadium networks. And we’re still not done!

Also in this issue is a well timed, deeply informed essay from Chuck Lukaszewski about unlicensed LTE and what it means to venues. Chuck, the top wireless guru at Aruba, a Hewlett Packard Enterprise company, digs into this developing cellular/Wi-Fi issue and delivers some heads-up knowledge that all venue tech professionals should absorb. We also have one more profle in the issue, a look at a temporary Wi-Fi network being installed at the Los Angeles Coliseum. That’s a lot of reading, so get started by downloading your free copy today!

Part of the reason we’re able to bring you so much good content is the support we get from our industry sponsors. In this issue we also have a record number of sponsors, including Mobilitie, Crown Castle, CommScope, JMA Wireless, Corning, Huber+Suhner, American Tower, Extreme Networks, Oberon, Cox Business, 5 Bars, Boingo Wireless and Aruba, a Hewlett Packard Enterprise company. The support of our sponsors allows Mobile Sports Report to not only do all the work necessary to bring you these great stories, but it also allows us to offer our reports to readers free of charge! We’d also like to welcome new readers from the Inside Towers community, who may have found their way here via our new partnership with the excellent publication Inside Towers.

Download the Fall 2017 Stadium Tech Report today!

NFL Wi-Fi update: AT&T Stadium adds 667 APs

A row shot of the under-seat APs at AT&T Stadium. Photo: Dallas Cowboys

With today’s main NFL season schedule already underway, here is a quick update on all the NFL stadium Wi-Fi updates we know about that are either finished or still under construction for the 2017 season:

— AT&T Stadium, already the location of the biggest (by number of APs) stadium network we know of, is in the process of adding another 667 APs, a project scheduled to be finished in the next few weeks. According to John Winborn, CIO for the Dallas Cowboys Football Club, the new APs are all the Cisco 3800 two-radio models, and will be installed on the stadium’s 400-level seating area.

When done, AT&T Stadium will have 2,567 APs, with plans to add another 400 in the upcoming offseason. After that, said Winborn, AT&T Stadium will “be out of locations in the stadium to place additional APs without additional spectrum opening up.” (editor’s note: This post has been updated to correct an error in reporting that previously stated that 600 new APs had been installed.)

— In Atlanta, the fans are getting ready for the first regular-season game at the new Mercedes-Benz Stadium, which will have nearly 1,800 APs total, many of those under the seats in the bowl.

— As previously reported in MSR, Denver’s Sports Authority Field at Mile High is in the process of ripping and replacing its former Wi-Fi network, installing 1,470 new Cisco 3800s in a project scheduled for completion by late October/early November.

— In Philadephia, Lincoln Financial Field is in the process of switching from an Extreme Networks deployment to a new one using Wi-Fi gear from Panasonic. More details on this project are promised later this fall.

— In Seattle, the Seahawks’ IT team is adding additional APs to improve coverage in the four corners of the upper seating sections of CenturyLink Field, according to Chip Suttles, vice president of technology for the Seahawks. Suttles said a full offseason tuning assessment with gear provider Extreme Networks also saw a new channel plan put in place, and “multiple Wi-Fi antenna orientations to improve coverage.”

— Detroit’s Ford Field also got a Wi-Fi (and DAS) upgrade over the past offseason, a revamp that will now allow all visitors, and not just Verizon Wireless customers, to use the stadium Wi-Fi.

— Chicago Bears fans will finally have free Wi-Fi this season, after being forced to pay for access previously.

— Following the announcement that the Arizona Cardinals were switching from CDW to Cox Business as the exclusive technology provider for the University of Phoenix Stadium, the UoP Stadium network was set for some additional APs and some network tuning ahead of the NFL season, according to Mark Feller, vice president of technology for the Cardinals.

Anyone else we miss? Send us the info… and the speedtests!

Denver Broncos, Verizon bring Wi-Fi blitz to Sports Authority Field at Mile High

Railing-mounted Wi-Fi enclosures in the lower seating bowl at Sports Authority Field at Mile High in Denver. Credit all photos: Paul Kapustka, MSR (click on any photo for a larger image)

Even as the team on the field seeks to regain its recent Super Bowl champion status, Denver Broncos fans will have something they can all cheer this season — vastly improved Wi-Fi networking service, which is now available to all fans and not just Verizon Wireless customers.

Even though we found good connectivity on our last visit to Sports Authority Field at Mile High two seasons ago, the caveat was that Wi-Fi was only available to Verizon customers, since the carrier couldn’t find another wireless provider who would chip in to fund the system. Fast-forward to later in 2016, when Verizon and the Broncos sought to upgrade the entire system and ended up picking a $6 million bid from Cisco to install a total of 1,470 new Wi-Fi APs, replacing the 640 APs in the old system, which started out with 500 Cisco APs in 2011 and added some more over the years.

The new network, which is scheduled to be fully completed by late October or early November this year, is already live in parts of the lower bowl at Mile High as well as in many concourse, suite and back-of-house locations. The big difference inside the hardware is the Broncos’ and Verizon’s choice of using the new Cisco 3800 APs, which can have two separate antennas in each device, basically doubling the amount of connectivity per unit. The new network will be powered by a new 10-gig backbone pipe provided by CenturyLink, replacing the 1-Gbps pipe that was previously used.

Close-up of a lower-bowl railing AP mount.

Cisco 3800s are proving to be a popular choice in venues lately, being picked for recent deployments at the San Jose Sharks’ SAP Center and at Notre Dame Stadium.

“The 3800 is a game-changer,” said Russ Trainor, the Broncos’ vice president of information technology, during a stadium tour Tuesday, citing its ability to connect more fans per device. Perhaps the most visually striking note of the upgrade is the huge amount of new railing-mounted APs in the building, with several per row not an uncommon sight in the lower bowl. Jason Moore, a senior IT engineer with the Broncos (and as Trainor calls him, a “Wi-Fi wizard”) said the enclosures are all custom designs from a local provider, with some of the fiberglass structures housing not just Wi-Fi but Verizon DAS antennas as well.

(Right now, the DAS situation at Mile High remains unchanged from our last visit, with all cellular carriers basically running their own operations.)

Going under-seat in bright orange fashion

The other new deployment method being used by the Broncos is under-seat placement, a tactic used for about 90 APs so far, half of those in the South end zone seats, where there are no overhead structures at all. Overhead AP placements are also being used in the main seating bowl, mainly to serve rows at the tops of sections.

No mistaking where the under-seat APs are at Broncos games.

Unlike other stadiums, who try to hide the under-seat APs as best they can, the Broncos have gone the opposite direction, painting many a bright Broncos orange to show up under the Broncos blue seats. “The mounting options are just about getting the APs as close to fans as we can,” Moore said. “Railing [mounts] work great. Under seat is new to us, and we’re excited to see how they work.”

In a quick empty-stadium test, MSR found Wi-Fi speeds in the south stands of 46 Mbps download and 47 Mbps up, on both the all-access and Verizon-customer SSIDs. In section 309, right at the 50-yard line, we got a Wi-Fi speed reading of 69 Mbps down, 75 Mbps up. Verizon execs at the tour said that like at other NFL stadiums where Verizon helps provide the Wi-Fi, Verizon customers will have reserved bandwidth and an autoconnect feature that links them to Wi-Fi without any sign-in needed. Trainor said the Broncos are still undecided how to approach the all-access Wi-Fi onboarding, though the team is leaning in the favor of having some kind of portal approach to gather information from fans using the service.

Of the 1,470 planned new APs (that count may change as final tuning is made, Moore said), the Broncos plan to deploy 920 of those in the seating bowl. With many of those devices having two 5 GHz antennas for each AP, the Sports Authority Field at Mile High crowds should enjoy one of the league’s top network experiences when all the work is completed.

Both railing and under-seat deployments are used to bring Wi-Fi to the south stands.

Those readers who closely track MSR stories for such stats should know that we are now working on a new chart to show not just the top numbers of APs in stadiums, but actual radios and antennas thanks to devices like the Cisco 3800 that have more than one per unit. (Any and all help with such counts is appreciated, you know where to find us!)

Like other stadiums, the network in the bowl at Sports Authority Field at Mile High will switch to only 5 GHz connections when complete. Even a few years ago, stadiums needed to still support 2.4 GHz Wi-Fi connections for fans, but the quick shift in consumer devices has shown that almost all mobile devices used these days have a 5 GHz radio.

A string of summer concerts at the stadium (which still bears the name of the now-bankupt and closed sporting-goods business as the Broncos search for a new title sponsor) kept the network deployment from being completed sooner, but Trainor and Moore said the incremental improvements are already being noticed. With both the old system and new system working simultaneously, Moore said that at last week’s college game between the University of Colorado and Colorado State University, the network saw approximately 35,000 unique users — more than the typical 25,000 unique connections during a Broncos game when only Verizon customers could use the network.

“Our goal and challenge is to connect as many fans as possible,” Moore said.

A smaller railing mount seen in the upper deck (300 level) seating section

Wi-Fi antenna mounts in the ceiling of the United Club

Wi-Fi coverage also exists for the fan-gathering area outside the stadium to the south

No Wi-Fi here, just white horses

Super Bowl 50 app use sets Levi’s Stadium records, led by video watching and drink orders

Screenshot of home page of Super Bowl 50 stadium app. (Click on any photo for a larger image)

Screenshot of home page of Super Bowl 50 stadium app. (Click on any photo for a larger image)

As part of the Wi-Fi and cellular usage records set at Super Bowl 50, fans at Levi’s Stadium also set new records for usage of the main stadium app features, including overall app adoption, viewing of action replays and Super Bowl commercials, and food and drink ordering.

According to the San Francisco 49ers networking staff, 46 percent of the 71,088 fans at the game downloaded the Super Bowl 50 stadium app, an NFL-specific app built by VenueNext, designers of the regular Levi’s Stadium app. That total is 16 percentage points higher than any recorded at a San Francisco 49ers regular-season game, according to the Niners.

One of the more unique features of the Super Bowl app was the ability for fans to use the app to order food and drinks, either for express window pickup, or for drinks only, the option for in-seat delivery. According to the Niners there were 3,284 food and beverage orders, 67 percent higher than the previous top order number ever recorded at a Levi’s Stadium game.

The Niners did not provide separate statistics for how many orders were for express pickup and how many were for in-seat delivery out of the larger total. Unlike the regular-season Levi’s Stadium app, which supports food and beverage delivery service to every seat, the Super Bowl app only offered drink delivery, per the wishes of the NFL.

Drink delivery order page on Super Bowl stadium app, including the $13 Bud Light.

Drink delivery order page on Super Bowl stadium app, including the $13 Bud Light.

According to VenueNext and the team, the average in-seat delivery time for drinks was 10 minutes. The top drink item ordered was Bud Light beer, while the top food item ordered via the app was chicken tenders, VenueNext said.

The Super Bowl 50 crowd also set Levi’s Stadium app records for video viewing, a stat helped perhaps by the availability of Super Bowl broadcast commercials, which fans at the game could watch via the app after they aired on TV. A full 55 percent of all app users either watched a video replay or Super Bowl commercials, the Niners said, 36 percent higher than the previous Levi’s Stadium record for video app views.

The app’s unique wayfinding feature, which uses the 2,000 beacons inside Levi’s Stadium to provide interactive maps, was used by 33 percent of the app users, according to the Niners. Fans could also use the app to purchase Super Bowl merchandise (which could be picked up at a concession stand or delivered to a suite), and according to the Niners all the mobile inventory was sold out before the game actually started, with an average order price of $212. Previously, the high-water average mark for app-ordered merchandise was $77 at a concert.

UPDATE: Top 4 carriers combine for 15.9 TB of cellular data use at Super Bowl 50

New Verizon Wireless under-seat DAS antenna placement at Levi's Stadium. Photo: Verizon Wireless

New Verizon Wireless under-seat DAS antenna placement at Levi’s Stadium. Photo: Verizon Wireless

UPDATE, 2/8/16, 1:50 p.m. — We now have data totals in from all four of the major U.S. cellular carriers, and at Sunday’s Super Bowl 50, fans combined to use 15.9 terabytes of data on the networks in and directly around Levi’s Stadium in Santa Clara, Calif.

Leading the way in usage was Verizon Wireless with a claim of 7 TB used by its customers; AT&T was next with 5.2 TB of claimed usage, followed by T-Mobile with a report of 2.1 TB, and Sprint with 1.6 TB. All the carriers’ numbers are well above figures from last year’s Super Bowl, where by our reporting Sprint, AT&T and Verizon had a combined 6.56 TB of cellular data consumed during the big game. (We did not have any T-Mobile reports from last year.)

For all the carriers, the data apparently includes both traffic on the in-stadium distributed antenna system (DAS) network was well as any macro deployments outside the stadium in parking lot areas. The final total was well over double the 6.56 TB of cellular traffic seen at last year’s big game in Glendale, Ariz. We are still waiting for Wi-Fi numbers from the Levi’s Stadium networking crew but it’s a good bet the 6.23 TB number from last year’s game will be eclipsed and we will have a new single-game Wi-Fi record as well so stay tuned.

Though we did hear and see some scattered reports of network connectivity issues during the Denver Broncos’ 24-10 victory over the Carolina Panthers it appears the upgrade of the DAS Group Professionals DAS install at Levi’s Stadium with its gear mainly provided by JMA Wireless stood up to the biggest-ever test of traffic. Congrats to all involved.

Thanks also to the Verizon and AT&T crews who supplied us with tweet reports and emails Sunday night, it made for some entertaining in-game stats. Some tweets embedded below.