Washington upgrades Wi-Fi backbone at FedExField

Screen Shot 2016-11-30 at 11.07.38 AMFans at FedExField now have more bandwidth to use on the stadium Wi-Fi network thanks to a recent upgrade that moved the backbone capacity to 10 gigabits per second, according to a release from the team.

Though it’s never been previously publicly announced by the Washington, D.C. NFL franchise, according to sources close to the team the reported network built by Verizon Wireless using Cisco Wi-Fi gear has been in full operation since last season, albeit under a sort of “soft launch” mode done without any promotion.

With more fans now using the network, the team saw a need to increase bandwidth from the 1 Gbps pipe that had been supplying all the Wi-Fi network needs. For many football-stadium Wi-Fi networks, 10-Gbps pipes are a popular choice, with several stadiums opting for multiple such connections. Levi’s Stadium in Santa Clara, Calif., home of the San Francisco 49ers, has four 10 Gbps pipes as part of its backbone configuration, while US Bank Stadium in Minneapolis, the new home of the Minnesota Vikings, has six 10 Gbps links.

For those who are new to stadium Wi-Fi (or who don’t know the difference between backhaul and actual end-user network speeds), the backbone bandwidth is shared by the entire network, which may have multiple tens of thousands of client devices running off any of hundreds of Wi-Fi access points. Typical user bandwidth speeds on Wi-Fi in large networks can range from single-digit megabits per second anywhere up to 20, 40 or even 60 Mbps, depending upon network configuration and density of users. Any fans attending games at FedExField who want to send us a speedtest please do!

Colorado passes on full-stadium Wi-Fi or DAS for Folsom Field

View of the west stands at Folsom Field, home of the University of Colorado football team. Credit all photos: Paul Kapustka, MSR (click on any photo for a larger image)

View of the west stands at Folsom Field, home of the University of Colorado football team. Credit all photos: Paul Kapustka, MSR (click on any photo for a larger image)

With a No. 9 ranking in all the major polls, the University of Colorado football team is experiencing a resurgence this season, which may lead to a rare CU sellout for this weekend’s final home game against Pac-12 rival Utah.

While the Buffs’ on-field performance in 2016 may have ended years of fan frustration, the 50,000+ fans expected to be in attendance at Folsom Field this Saturday may still experience another form of frustration, mainly in trying to get their mobile devices to connect to the Internet. According to school officials, there is no full-stadium, fan-facing Wi-Fi or cellular distributed antenna system (DAS) network in Folsom Field, and no plans to bring either to the venue anytime soon.

Instead, most fans at the on-campus stadium will rely on one of two nearby macro sites, one each from top wireless carriers AT&T and Verizon Wireless. According to Jeff Lipton, director of real estate for CU, the school decided to pass on bringing Wi-Fi or DAS to Folsom for a number of reasons, including the cost of the systems, the infrequent use of the facility, and potential loss of network control to prospective suppliers.

Hard to justify cost of connectivity for lightly used venue

“Nationally, stadiums are challenging” when it comes to cost justifications for installing wireless networks, Lipton said in a recent phone interview. While Lipton claimed that CU “hasn’t been sleeping on this,” saying the school has been reviewing wireless stadium options for several years, he added that CU had concluded that a cellular DAS wasn’t a good fit for Folsom, which has been the home of CU football since 1924.

Error message shown while trying to connect to ESPN's website at Folsom Field on Nov. 19.

Error message shown while trying to connect to ESPN’s website at Folsom Field on Nov. 19.

One of the main problems is, Lipton said, justifying the cost to bring connectivity to a venue that is only used a handful of times a year. In 2016, Colorado had six home games on its football schedule, the main use of 53,613-seat Folsom Field. This past summer there were two concert dates with the Dead and Company band that filled the stadium, and the stadium is also used as the finish line for the annual Bolder Boulder 10k on Memorial Day.

Though the crowds that do come to Folsom would no doubt enjoy better connectivity, right now it’s not in the cards, Lipton said, especially from a cellular DAS perspective.

“We looked at DAS for the main part of the stadium but determined it was not cost effective, and the vendors wanted a connectivity exclusive” for the rest of the campus, a deal Lipton said CU definitely did not want to agree to.

“We like to control our own [wireless] destiny on campus — we’re not going to give that up in a deal to get DAS,” Lipton said. And, Lipton said that “I’m not sure that [technically] in the long term, DAS is the solution” for stadium networks.

The right way to Wi-Fi

What’s more interesting to CU is finding some way or waiting for new technology to emerge to make owning and operating a Wi-Fi network inside Folsom something that makes sense. When it comes to Wi-Fi, Lipton said that CU has been aggressively installing it in many of the 12 billion square feet of building space it manages at the Boulder, Colo., campus.

Around Folsom, there is free public Wi-Fi available at the new Champions Center (an indoor practice field and offices building located adjacent to the east side of Folsom Field) as well as in the attached parking structure. There is also some free Wi-Fi available for suites and club spaces in the newer structures on the east side of Folsom Field, but nothing for the balance of bowl seating in the stadium.

Folsom's east side structure, which does have some Wi-Fi inside suites and club areas.

Folsom’s east side structure, which does have some Wi-Fi inside suites and club areas.

“We have pretty ubiquitous Wi-Fi throughout campus, and we installed it and run it,” said Lipton. In terms of bringing Wi-Fi into the Folsom Field bowl — as well as to the stands at the Coors Event Center, the school’s 11,00-seat basketball facility — that idea is still being studied by an internal working group, Lipton said.

“We recognize that long term, there are some real revenue opportunities [around Wi-Fi networks] that could pay for this later on,” Lipton said. “But it’s not there yet.”

Any Wi-Fi network that does end up getting built inside Folsom would also have to surmount the non-trivial challenge of bringing wireless networks to a facility with parts that are nearly 100 years old. Part of the bowl also sits in the ground, bringing another degree of difficulty to the idea of getting cables underneath the stands (for possible under-seat or railing-based antenna options). But for newer parts of the stadium, including the north end zone structures and the new east side, bringing connectivity to the stands outside wouldn’t be as difficult.

At the 11,064-capacity Coors Event Center, Lipton said there is some CU Wi-Fi inside the building, but it was not designed for full-stadium crowd access down into the seating bowl.

Unable to send texts, or get Internet access

Though Lipton claims that the two Folsom macro sites — one atop the roof on the stadium’s west side building and another on a building across the street — are working “much better” this season, an MSR visit to Folsom for the Nov. 19 home game against Washington State saw almost zero connectivity, on both the cellular and Wi-Fi front.

A look at the newer north and northeast structures at Folsom Field from the east stands.

A look at the newer north and northeast structures at Folsom Field from the east stands.

Though our tests were sporadic, with only one phone in one part of the stadium, our not being able to send a text message with a photo of the stunning Colorado Rockies backdrop was probably something many others experienced inside Folsom last week, where 48,658 fans saw CU beat WSU 38-24. On the Verizon network, we were almost always looking at a “1x” number for connectivity, which pretty much guaranteed no signal all day long.

Even inside the east building’s 5th-floor club area, where we detected the CU campus Wi-Fi network, our phone couldn’t connect, briefly showing a link but then dropping it as soon as we tried to do anything. There was no visible promotion of the CU Wi-Fi, or any instruction about whether fans should use one of two visible SSIDs, one with a “guest” label and one without. Back out in the stands, we tried to get to the ESPN website to see other college scores, but again our device failed to connect.

While Lipton admitted that “traffic on [football] game days can overwhelm” the macro sites, he still thinks any advanced connectivity has to make fiscal sense. As the one who says he signs contracts for such deployments at CU, Lipton said “there’s an art to every deal.” For Folsom Field connectivity, however, that deal hasn’t yet been done.

Seahawks see 4.1 TB of Wi-Fi usage for Monday Night game vs. Bills

Seahawks fans at CenturyLink Field during Nov. 7 game. Credit: Seahawks.com

Seahawks fans at CenturyLink Field during Nov. 7 game. Credit: Seahawks.com

With the Seahawks starting to gain momentum for another playoff push, the Wi-Fi network at their home of CenturyLink Field is getting a good workout as well, with 4.10 terabytes of data used during Seattle’s 31-25 win over the Buffalo Bills on Nov. 7.

The Monday Night Football game, an exciting, close contest, also saw a total of 35,808 unique clients and a peak concurrent user number of 23,281 on the Extreme Networks-powered Wi-Fi network at the stadium, according to Seahawks IT officials. The game set an attendance record for CenturyLink Seahawks games, with 69,084 in the seats, breaking the old record of 69,080 from last season.

According to information from all the games so far this season (see below), the CenturyLink Field Wi-Fi has been seeing strong and growing usage, with unique numbers starting out in the high 20,000-user level and edging up each week. Upload totals also seem to be growing, with maybe the team’s improved record as the impetus.

image001

Verizon: Series fans used 5.3 TB of cellular data at Wrigley Field, 15.4 TB at Progressive Field

Cubs victory celebration in Chicago's Grant Park. Credit: KIICHIRO SATO/AP from Cubs.com.

Cubs victory celebration in Chicago’s Grant Park. Credit: KIICHIRO SATO/AP from Cubs.com.

We now have the Verizon Wireless stats for cellular usage by fans at Chicago’s Wrigley Field during the recent World Series, and according to Verizon fans on its cellular network in the stadium used a total of 5.3 terabytes of data for the three games at the Friendly Confines. Verizon also said that for the four games at the Cleveland Indians’ Progressive Field, Verizon saw a combined 15.4 TB of wireless data on its cellular network.

Combined with AT&T’s previously announced total of 3 TB on its network over the three games gives us a total of 8.3 TB of DAS usage for the top two wireless carriers for the three Chicago Cubs home games during the Fall Classic. For the Cleveland games, the 15.4 TB on Verizon and 2 TB on AT&T comes to 14.4 TB combined.

Outside the games, Verizon added that Chicagoans used another 6.5 TB of data along the World Series celebration parade route last Friday. According to a published report, AT&T’s network in Chicago saw 10.5 TB of traffic during the parade; we haven’t been able to confirm that number with AT&T directly but with 5 million fans estimated taking place in the celebration, 17 TB of combined data usage sounds about right.

The MSR Interview: San Francisco Giants CIO Bill Schlough

AT&T Park CIO Bill Schlough shows off his World Series bling.

AT&T Park CIO Bill Schlough shows off his World Series bling.

Who better to talk about stadium Wi-Fi than the guy who was there when it all started? Our guest for our first MSR Interview (part of our Stadium Tech Report Podcast series) is San Francisco Giants senior vice president and chief information officer Bill Schlough, who goes old-school talking about stadium Wi-Fi back in 2004… and brings it to the current day with stats from the most recent season at AT&T Park. Plus, his thoughts on game-day apps and why great connectivity is the real winner. Listen in now!

Hear Bill talk about:

— New Wi-Fi records set… during the Warriors’ playoff run

— Why going under-seat with Wi-Fi was a necessary thing to do

— How the Giants are experimenting with virtual reality

— Why he thinks great connectivity matters most (even more than stadium-app features)

Some story links that offer some history about AT&T Park’s networks from MSR:

S.F. Giants add more Wi-Fi, ‘virtual reality experience’ to AT&T Park for 2016 season

SF Giants fans used 78.2 TB of Wi-Fi data at AT&T Park during 2015 season

Stadium Tech Report: World Series set new wireless records at AT&T Park

Stadium Tech Report: San Francisco’s AT&T Park lives up to its wireless reputation

Giants: NLCS stadium Wi-Fi usage at AT&T Park quadrupled since 2012

SUBSCRIBE TO THE PODCAST:

Here is the link to the podcast on iTunes!

Stadium Tech Report: Carolina Panthers take ownership of DAS, Wi-Fi at Bank of America Stadium

James Hammond, director  of IT for the Panthers, poses next to an under-seat Wi-Fi AP. Credit all photos: Carolina Panthers

James Hammond, director of IT for the Panthers, poses next to an under-seat Wi-Fi AP. Credit all photos: Carolina Panthers

“The fan is the most valuable member of our team,” Jerry Richardson, owner of the Carolina Panthers, is fond of saying.

And it’s become the virtual mission statement for the Charlotte, N.C.-based National Football League franchise. So even though its home field, the Bank of America Stadium, was built relatively recently (1996), technology has come a long way in two decades. And as the Panthers began a four-phase renovation in 2014, they did it with fans’ MVP status in mind, according to James Hammond, director of IT for the Panthers. “It was time for some changes,” he said.

While Carolina was among the first NFL stadiums to install fan-facing Wi-Fi and enhanced cellular networks, the previous DAS and Wi-Fi systems weren’t keeping up with demand and that was starting to adversely impact the Panthers fan experience, Hammond said.

“We chose to perform a rip-and-replace on both DAS and Wi-Fi and take ownership in-house,” Hammond explained. Because the Panthers own and operate BofA Stadium, making those moves was a lot easier than if they were tenants.

Time for an upgrade

Editor’s note: This profile is part of our latest STADIUM TECHNOLOGY REPORT, which includes more stadium profiles as well as looks at Wi-Fi at the Mall of America, and analytics software being used by the Cleveland Browns. DOWNLOAD YOUR FREE COPY of the report today!

The first “fan-centric improvements,” as Hammond called them, came in 2014 in the form of escalators, video big boards and a distributed audio system. As part of the second phase of upgrades, the Panthers then used the 2015 offseason to renovate the club-level suites and tore out the old DAS system while they were at it. And after a careful evaluation of different DAS solutions, they shortlisted two vendors: CommScope and Corning.

CommScope ultimately got the nod; the Panthers then had to decide between the vendor’s ION-B and ION-U DAS systems. “We went with the ION-U, which was quite new and cutting edge at that point, since it had NEMA-rated remotes,” Hammond said. Other systems lacked that kind of weatherproofing and would require additional enclosures – and expense.

CommScope's ION-U powers the new DAS at Bank of America Stadium.

CommScope’s ION-U powers the new DAS at Bank of America Stadium.

“We started over with all new fiber and coax. We did the decommissioning and construction in 90 days, which was pretty quick for a ground-up project,” he said. Beam Wireless Inc. did the design, integration and optimization and is handling the ongoing maintenance of the DAS system; Optical Telecom installed the DAS gear. BofA Stadium now has 256 DAS remotes and more than 600 DAS indoor and outdoor antennas.

AT&T, Verizon and Sprint are the participating DAS carriers; T-Mobile is weighing whether to join the mix during the 2017 off-season.

The Panthers have divided BofA Stadium into 48 DAS zones: 16 zones for the upper bowl, 16 in the lower bowl, and another 16 for concourses, suites, clubs, and offices. Not all zones are used exclusively; carriers choose simulcast patterns that place multiple zones into sectors, and can change them as capacity requirements dictate, Hammond told Mobile Sports Report.

“With some minor design changes to the interior areas, we can accommodate nearly 70 zones,” he explained. “At present the most sectors in use by a carrier is 32. This means the carrier simulcasts across a mix of our 48 zones in order to match them up to 32 carrier sectors.”

Once the new DAS was built and the first couple of events were analyzed, carriers began asking for more frequencies and additional DAS sectors to continue meeting ever-growing demand. In response to the new carrier requests, the first round of DAS upgrades were implemented in the spring of 2016, Hammond said. During the 2015 season, DAS bandwidth was running around 2 GB during games. Hammond said, “With these latest DAS upgrades, we expect the bandwidth numbers to be even higher.”

A DAS remote in a NEMA-rated enclosure.

A DAS remote in a NEMA-rated enclosure.

The impact of the new DAS system was felt immediately upon its debut in July 2015. “It was a much better experience for fans who noticed the improved cellular experience,” Hammond said. Another unexpected benefit: The upgraded DAS helped mitigate bottlenecks with the old Wi-Fi system, which Hammond characterized as “under-designed.”

Going under seat for Wi-Fi upgrade

Unfortunately, there wasn’t time to address any Wi-Fi upgrades before the 2015 football season began, but the Panthers issued an RFP for new Wi-Fi in August 2015 in preparation for Phase 3 renovations that would also include security upgrades and renovations to the upper concourse.

Interested vendors needed to ensure high bandwidth rates as well as high take-rates that allowed three different ISPs (Time Warner Cable/Charter, Level 3 Communications and Windstream) to deliver in excess of 10 GB, though Hammond said they’re starting at a 7-GB threshold.

The Wi-Fi award went to Aruba, now HP Enterprise, in December 2015, and construction began in January 2016 after the last postseason game, when the Panthers beat the Arizona Cardinals to win the NFC championship and a trip to Super Bowl 50.

Similar to Levi’s Stadium and the Dodgers Stadium, the Panthers chose underseat AP enclosures; BofA Stadium sports 770 AP enclosures in the upper and lower bowls out of a total of 1,225 APs, all to ensure maximum coverage and minimal dead spots. The Panthers selected AmpThink to do the Wi-Fi integration and construction; the turnkey contractor also designed and fabricated a custom enclosure for the APs.

Indoor access point inside the stadium.

Indoor access point inside the stadium.

One other innovation in the Panthers’ Wi-Fi installation is that the underseat enclosure is mounted to the riser — the vertical part of the step — but looks like it’s on the tread, the horizontal part, which is intended as a waterproofing measure. “The riser is easier to seal and isn’t affected by pressure washers, which you’re doing constantly with an outdoor stadium,” Hammond said. “And by running pipe through the riser, you don’t have gravity working against you,” which helps keep out water, he explained.

Panthers fans access the stadium Wi-Fi through a portal page after accepting the team’s terms and conditions. From there, they are whitelisted and can automatically join the Wi-Fi network for the rest of the season. Hammond said a fan’s email is requested but not required by the portal page, and there’s a small incentive offered to encourage fans’ email subscriptions.

The new Wi-Fi system got a workout with a soccer game at BofA Stadium at the end of July 2016, then with a Panthers’ Fan Fest the following week. “All the indicators were good, and fan feedback about the system was excellent,” Hammond said. But he cautioned that the two events were not “full bowl” events with smaller attendance numbers (~50,000) than a regular season football game (75,000+). “We will continue to optimize and tune settings as we learn more during events with higher attendance,” Hammond said.

Total budget so far for the technology upgrades totals about $16 million; the DAS build-out was just under $10 million; Wi-Fi was a little more than $6 million, which included additional wired infrastructure, according to the team.

Beacons coming next

And the Panthers aren’t done making technology improvements to their stadium. Phase 4 looks to add Bluetooth beacons and do some refinement of the Panthers app. “My goal during the upcoming season is to look at options for location-aware services,” Hammond said. Some APs have beacons built in; other may need to be added to get the granularity the Panthers want for location awareness.

Hammond also wants to give fans more things to do with the Panthers app and also optimize it for push notifications, even with something as basic and useful as restroom and concessions location information. “As we learn more about fans individually, we can direct them to things of particular interest to them,” he added.

“So far, we are very pleased with the performance of the Wi-Fi and DAS systems,” Hammond said, noting the Panthers will continue to tune frequencies, add zones and increase bandwidth where needed. It’s the sort of attention that smart sporting franchises pay to their most valued team members.

car1