November 27, 2015

IBM formally launches sports consulting practice to bring tech to stadiums

Texas A&M student at recent Aggies football game. Photo: Paul Kapustka, MSR (click on any photo for a larger image)

Texas A&M student at recent Aggies football game. Photo: Paul Kapustka, MSR (click on any photo for a larger image)

IBM formally cemented its entrance to the sports-stadium tech deployment market with the announcement of a sports and fan experience consulting practice, and a “global consortium” of tech and service suppliers who may help IBM in its future stadium and entertainment venue deployments.

For industry watchers, the Nov. 19 debut of the IBM “Sports, Entertainment and Fan Experience” consulting practice was not a surprise, since its leader, Jim Rushton, had already appeared at tech conferences this past summer, talking about IBM’s plans to deploy a fiber-based Wi-Fi and DAS network at the new Mercedes-Benz Stadium being built for the Atlanta Falcons. IBM was also publicly behind a similar network build over the last two years at Texas A&M’s Kyle Field. For both networks, IBM is using Corning optical gear.

Still, the formal creation of the IBM practice (you can read all about it at the new IBM sports website) means that the 800-pound gorilla is now firmly inside the competitive ring of the stadium-tech marketplace, a landscape that currently has multiple players, many of which have multiple stadium deployments under their belts. However, IBM’s vast experience in big-time sports technology deployments — Big Blue is behind such endeavors as the truly wonderful online experience of The Masters, as well as technical underpinnings of three of tennis’ Grand Slam events (Wimbledon, the U.S. Open and the Australian Open) — as well as its considerable tech and monetary resources probably makes it a No. 1 contender for all of the biggest projects as well as possibly smaller ones as well.

Artist's rendering of planned overhead view of new Atlanta NFL stadium

Artist’s rendering of planned overhead view of new Atlanta NFL stadium

Rushton, who spoke with Mobile Sports Report earlier this year in one of his first public appearances as an IBMer, said in a phone interview this week that IBM’s fiber-to-the-fan network model isn’t just for large-scale deployments like the one at 105,000-seat Kyle Field or the Falcons’ new $1.4 billion nest, which will seat 71,000 for football and up to 83,000 for other events after it opens in 2017.

“That type of system [the optical network] is scalable,” Rushton said, and even in smaller venues he said it could potentially save customers 30 percent or more compared to the cost of a traditional copper-based cabled network. The flip side to that equation is that purchasers have fewer gear suppliers to choose from on the fiber-based side of things, and according to several industry sources it’s still sometimes a problem to find enough technical staffers with optical-equipment expertise.

How much of the market is left?

The other question facing IBM’s new consulting practice is the size of the market left for stadium tech deployments, an answer we try to parse each year in our State of the Stadium survey. While this year’s survey and our subsequent quarterly reports found a high number of U.S. professional stadiums with Wi-Fi and DAS networks already deployed, there are still large numbers of college venues as well as international stadiums and other large public venues like concert halls, race tracks and other areas that are still without basic connectivity.

Full house at Kyle Field. Photo: Paul Kapustka, MSR

Full house at Kyle Field. Photo: Paul Kapustka, MSR

With its new “global consortium” of companies that supply different parts and services of the connected-stadium experience, IBM could be an attractive choice to a customer that doesn’t have its own technical expertise, providing a soup-to-nuts package that could conceivably handle tasks like in-stadium IPTV, DAS and Wi-Fi, construction and stadium design, and backbone bandwidth solutions.

However, IBM will be going up against vendors who have led deployments on their own, and league-led “consortium” type arrangements like MLBAM’s project that brought Wi-Fi to almost all the Major League Baseball stadiums, and the NFL’s list of preferred suppliers like Extreme Networks for Wi-Fi and YinzCam for apps. Also in the mix are third-party integrators like CDW, Mobilitie, 5 Bars, Boingo Wireless and others who are already active in the stadium-technology deployment space. And don’t forget HP, which bought Wi-Fi gear supplier Aruba Networks earlier this year.

Certainly, we expect to hear more from IBM soon, and perhaps right now it’s best to close by repeating what we heard from Jared Miller, chief technology officer for Falcons owner Arthur Blank’s namesake AMB Sports and Entertainment (AMBSE) group, when we asked earlier this year why the Falcons picked IBM to build the technology in the new Atlanta stadium:

Remote optical cabinet and Wi-Fi AP at Kyle Field. Photo: Paul Kapustka, MSR

Remote optical cabinet and Wi-Fi AP at Kyle Field. Photo: Paul Kapustka, MSR

“IBM is unique with its span of technology footprint,” Miller said. He also cited IBM’s ability to not just deploy technology but to also help determine what the technology could be used for, with analytics and application design.

“They’ve looked at the [stadium] opportunity in a different manner, thinking about what we could do with the network once it’s built,” Miller said.

From the IBM press release, here is the IBM list of companies in its new “global consortium,” which IBM said is not binding, meaning that none of the companies listed is guaranteed any business yet, and others not on the list may end up in IBM deployments, like Kyle Field, which uses Aruba gear for the Wi-Fi:

Founding members of the consortium, include:

· Construction and Design: AECOM, HOK, Whiting Turner

· Infrastructure Technology/Carriers: Alcatel/Lucent, Anixter, Commscope, Corning, Juniper Networks, Ruckus Wireless, Schneider Electric, Smarter Risk, Tellabs, Ucopia, Zebra Technologies, YinzCam (IPTV), Zayo, Zhone

· Communications Solutions Providers: Level 3, Verizon Enterprise Solutions, AT&T

· Fan Experience Consulting & Data Management Integration: IBM

Arizona Cardinals’ University of Phoenix Stadium beefs up Wi-Fi and DAS ahead of College Football Playoff championship game

University of Phoenix Stadium before Super Bowl XLIX. Photo: Paul Kapustka, MSR (click on any photo for a larger image)

University of Phoenix Stadium before Super Bowl XLIX. Photo: Paul Kapustka, MSR (click on any photo for a larger image)

After just hosting a Super Bowl, one with record wireless traffic numbers, you might not think that the University of Phoenix Stadium in Glendale, Ariz., needed to upgrade its Wi-Fi and DAS networks. But with many more big events on the way soon, including hosting this season’s College Football Playoff championship game, the UoP Stadium isn’t sitting still, but instead is fine-tuning and expanding its networks to ensure fans stay connected as well as possible.

According to Mark Feller, vice president of technology for the Arizona Cardinals, more Wi-Fi has been added to the stadium networks for this football season, including lawn areas just outside the stadium and the Pat Tillman Plaza area on the north side of the stadium. For the Super Bowl last year the venue had extensive DAS coverage outside from a Crown Castle deployment, but in an email message Feller said adding Wi-Fi to the mix was always part of the plan. Here’s Mark:

“Our plan from the start was to have Wi-Fi outdoors for our fans to use and we are rolling it out as time allows. We have such good weather that there are thousands of people tailgating on game days. In addition, the Cardinals Mobile App (from Yinzcam) provides live Stadium Feeds, Replays, and the Red Zone Channel so our fans can keep up with the early games while they are outside.”

Outside UoP Stadium, where the architecture allows for DAS antenna placement under the fascia as well as behind speaker covers.

Outside UoP Stadium, where the architecture allows for DAS antenna placement under the fascia as well as behind speaker covers.

Inside the stadium, Feller said there are now Gimbal beacons deployed for “selective messaging” alerts that are tied to the stadium app. The team also added a separate Verizon Wireless SSID to its Wi-Fi mix, giving Verizon customers reserved bandwidth as well as the ability to autoconnect. The Wi-Fi network uses Cisco gear and is managed and supported by CDW. At the Cardinals’ most recent home game, a 26-18 win over the Baltimore Ravens on Oct. 26, the Wi-Fi network carried 1.445 terabytes of data, with 22,502 unique connections, according to numbers provided by Feller. Out of the 63,500-seat stadium a maximum number of 19,559 concurrent users was seen that day, with the top sites connected to by fans being Apple, Facebook, Google, iCloud, Yahoo, Instagram, Twitter and ESPN, according to Feller.

Getting ready for the playoff championship

For both the biggest college game of the year (scheduled for Jan. 11, 2016) which like last year should be a big network event, as well as a host of other “big events,” like a U.S. Women’s soccer team game vs. China on Dec. 13 and the Fiesta Bowl on New Year’s Day, Feller said the UoP stadium team is continuing to expand the Crown Castle DAS as well, with more sectors in the stadium’s Club and Loft sections, as well as more coverage outside on the lawns. Portable Wi-Fi is also an option, Feller said, as the stadium adds temporary seating to expand for the big game of the collegiate season:

“Having the Super Bowl here did give us some ideas about increasing density in some areas where we put temporary seating. We tested some different WiFi portable enclosure systems that we could put up and take down quickly and figured out how to get cabling to them quickly as well. That will help us get set up for the CFP Championship.”

Kansas City Royals score with jump in postseason stadium Wi-Fi and DAS traffic

Royals fans at Kauffman Stadium enjoying the postseason. Credit all photos: Kansas City Royals (click on any photo for a larger image)

Royals fans at Kauffman Stadium enjoying the postseason. Credit all photos: Kansas City Royals (click on any photo for a larger image)

If you need a reason to justify Wi-Fi network installs or improvements in your stadium, here’s an optimistic rationale: If your team makes it to the playoffs and the championship, you can expect a big surge in postseason wireless traffic.

That idea was proven again this fall by the Kansas City Royals, who racked up big postseason Wi-Fi and DAS traffic numbers at Kauffman Stadium during their march to the 2015 World Series championship, including a 3.066 terabyte night on the Wi-Fi network for Game 1 of the World Series. That’s a 1 TB jump from last season, when Kansas City saw 2+ TB of Wi-Fi traffic during Game 7 of the 2014 World Series.

According to numbers provided by Brian Himstedt, senior director of information systems for the Royals, the Kauffman Stadium Wi-Fi network saw an average of 1.9 TB of aggregate throughput for the eight home games Kansas City hosted in the playoffs.

Fans cheering the Royals at Kauffman Stadium

Fans cheering the Royals at Kauffman Stadium

The average peak user count over those games was 11,850, with a high peak of 13,900 during Game 2 of the World Series. The stadium’s capacity for the postseason games, Himstedt said, was 40,500.

The postseason Wi-Fi traffic, Himstedt said, was approximately 34 percent upload and 66 percent download. During the regular season, the Kauffman Wi-Fi network had upload/download averages of 22 percent and 78 percent respectively, meaning that during the playoffs fans were probably more busy sharing information than obtaining it.

Overall, the postseason Wi-Fi numbers were much larger than the Royals’ regular-season stats, Himstedt said. Here are some of the regular-season stats during a summer that saw the network serve more than 180,500 unique clients on the Wi-Fi network:

– Average throughput per game: 625 GB
– High throughput, single game: 1.05 TB (Sept. 26)
– Average peak concurrent users per game: 5,150
– High peak concurrent users, single game: 7,500 (Opening Day, April 6)

The average attendance of Kauffman Stadium during the regular season was 33,900, Himstedt said.

Sprint DAS numbers also jump

And while the DAS at Kauffman Stadium is still awaiting full participation by all of the top wireless carriers, hometown favorite Sprint was active on the new system deployed by Advanced RF Technologies before the start of the season, and according to Sprint there were huge increases in DAS traffic compared to 2014.

Here are some DAS numbers from Sprint about the playoff traffic at Kauffman Stadium:

– Total tonnage for the 2015 eight game post-season was 2.6 terabytes

– Average tonnage per post-season game increased 4,000% in 2015 compared to 2014

– Sprint fans talked on their phones a total of 178,954 minutes in the post-season

– LTE connection rates for the 2015 post-season improved by approximately 40% compared to 2014

According to Sprint, the DAS supported all the frequencies used by Sprint devices, including the 1.9GHz, 2.5GHz and 800MHz bands.

Keeping the Wi-Fi hidden: AT&T Stadium perfects the art of Wi-Fi AP concealment

Wi-Fi antennas visible under the 'shroud' covering the outside of the overhang at AT&T Stadium. Photo: Dallas Cowboys (Click on any photo for a larger image)

Wi-Fi antennas visible under the ‘shroud’ covering the outside of the overhang at AT&T Stadium. Photo: Dallas Cowboys (Click on any photo for a larger image)

Since they like to do everything big in Texas, it’s no surprise that the IT team at AT&T Stadium has taken the art of Wi-Fi access point concealment to new heights.

To just above the first and second seating levels of the stadium, that is.

Even though the venue has more Wi-Fi APs than any sports stadium we’ve ever heard of, trying to find any of the 1,900 permanently installed APs is a tough task, thanks to measures like the fiberglass shrouds that circle the stadium just above the first and second seating levels. Underneath those custom-built coverings are numerous Wi-Fi APs, DAS antennas and even cameras, all contributing to the high level of connectivity inside AT&T Stadium while remaining invisible to the visiting fan’s eyes.

“The philosophy throughout the stadium is for a clean, stark look,” said John Winborn, chief information officer for the Dallas Cowboys Football Club, which is the primary tenant of the venue. “That’s a high standard, and that is a real challenge for us when it comes to Wi-Fi and DAS.”

In just about every stadium network deployment we write about, concealment and aesthetics are always one of the top concerns, especially when it comes to Wi-Fi access points and DAS antennas. For some reason, the physical appearance of an obvious piece of technology evokes strong reactions, even as other necessary structural items are ignored.

(Editor’s note: This story is an excerpt from our most recent Stadium Tech Report, the PRO FOOTBALL ISSUE, which is available for FREE DOWNLOAD right now from our site. In the report our editorial coverage includes a profile of the new Wi-Fi network at Green Bay’s Lambeau Field and team-by-team profiles of Wi-Fi and DAS deployments at all 31 NFL stadiums. Get your copy today!)

Under-seat Wi-Fi AP at AT&T Stadium. Photo: Dallas Cowboys

Under-seat Wi-Fi AP at AT&T Stadium. Photo: Dallas Cowboys

A clean, sleek look at the house that Jerry built

As one anonymous commentator at this summer’s SEAT conference noted, “stadium supervisors don’t ever care about seeing a 4-inch pipe, but leave one antenna out and they go crazy.” And whoever that stadium person is, he or she probably has a kindred soul in Dallas Cowboys owner Jerry Jones.

An unseen antenna is Jerry Jones’ favorite kind. Winborn said that AT&T Stadium embraces design in all things visible, noting that the “clean look” idea extends to advertising inside the seating bowl, where the only permanent signs are located in the end zone areas.

“We’re very conscious of the aesthetics here,” Winborn said. “Everyone here sees the benefit of what a great looking building can be. And it all starts with [Jerry] Jones.” Jones’ ideas, Winborn said, “are a major influence on everything we do.”

What that means when it comes to Wi-Fi is that while the stadium always aims to be the best-connected venue around – for this football season, AT&T Stadium will have 1,900+ permanent Wi-Fi APs and another 100 or so available for temporary placements – it also aims to hide the physical gear as much as possible. In suites and hallways there is the natural solution of putting antennas behind ceiling panels, but in the seating bowl, Winborn said, “we don’t have a lot of areas to hide them. We’ve had to become pretty clever about ways to hide APs.”

Two years ago, when the stadium’s AP count was going up from 750 to 1,250, the idea came about to design a custom fiberglass “shroud” that would circle the arena on the front of the overhangs above the first and second seating levels.

A row shot of the under-seat APs. Photo: Dallas Cowboys

A row shot of the under-seat APs. Photo: Dallas Cowboys

“We’re very conscious of the aesthetics here,” Winborn said. “Everyone here sees the benefit of what a great looking building can be.

Winborn said one of the IT staff members had contacts in the manufacturing world, which helped the Cowboys build a slighly convex design that wouldn’t be readily apparent to the untrained eye, yet be big enough to house Wi-Fi and DAS gear all the way around the bowl.

Winborn said the shroud and its underlying gear were installed during one of the recent off-seasons, taking about a couple months – and the final result was so good that Winborn says he needs to use a laser pointer to show interested parties exactly where the equipment shroud sits. Since it’s fiberglass the shroud is somewhat easy to move to allow administration and maintenance of the equipment, but the seamless flow of the structure around the bowl may just be the most elegant AP hiding strategy in the short history of stadium Wi-Fi.

But even with the shrouds there was still a need for more new placements, especially in the middle of the open seating areas. So last year the AT&T Stadium team started deploying under-seat AP enclosures, working with design teams at the AT&T Foundry program to build a custom unit that is much smaller and unobtrusive than other under-seat AP enclosures currently in use.

“We worked with the AT&T Foundry and went through [testing] about a half-dozen models,” Winborn said, before finally arriving at a design that worked well and stayed small. “It’s about the size of a small cigar box,” said Winborn of the under-seat APs, 300 of which were installed in the 100-level seating last year. Another 250 are being installed for this year up in the 300-level seating, he said.

Winborn credited the early use of under-seat APs by the IT team at AT&T Park in San Francisco as a welcome guide.

Here's the big bowl that needs to be filled with Wi-Fi. Photo: Paul Kapustka / MSR

Here’s the big bowl that needs to be filled with Wi-Fi. Photo: Paul Kapustka / MSR

“I talked to the Giants and Bill [Schlough, the Giants’ CIO] and had my concerns” about under-seat APs, Winborn said. “But after they did it and had only one complaint in 2 years, that raised my comfort level.”

Like the Giants’ under-seat APs, the ones in AT&T Stadium are designed to be as maintenance-free as possible, so that they can be steam washed and not harmed by spills or any other physical interactions. Winborn said the Cowboys have even started putting sealant and paint over the top of the under-seat APs, “so they look just like a bump.”

With 1,900 to 2,000 APs available, it might seem like the AT&T Stadium IT crew has enough APs for now, so they can relax a bit when it comes to finding new ways to hide Wi-Fi gear. But Winborn knows the next surge is probably right around the corner (including early results from this season showing 4+ terabytes of Wi-Fi use).

“Everything we are giving the fans [in Wi-Fi bandwidth] they are gobbling it up, pretty quickly,” Winborn said.

Cowboys fans at AT&T Stadium use 4.295 TB of Wi-Fi during Sunday’s game

Dallas fans cheering on the Cowboys Sunday. Photo:

Dallas fans cheering on the Cowboys Sunday. Photo:

It’s no fun to try to take on Tom Brady and the New England Patriots when your star quarterback and receiver are sidelined with injuries. But watching an undermanned team didn’t keep Dallas Cowboys fans from using a whole lot of Wi-Fi at AT&T Stadium Sunday, as statistics from AT&T show that 37,173 unique users consumed 4.295 terabytes of Wi-Fi data during the contest.

UPDATE, 10/20: We just heard from the AT&T network folks who told us that an additional 1.517 TB of data was used on the AT&T DAS network at the same game. So that’s nearly 6 TB of wireless data (and probably more with the other carrier DAS traffic added in) for a regular-season game… !

This is the second time this season that the defending Super Bowl champs helped run up a big wireless score when visiting an enemy arena. While we don’t have DAS stats yet for Sunday’s game (a 30-6 New England victory) it’s a good guess that 4+ TB is the “new normal” for big-ish regular season games at pro or college venues with high-fidelity Wi-Fi networks. Maybe it’s just a “the Patriots are in town” thing but we are betting we’ll see a lot more 4+ TB games around the country this football season.

Also according to AT&T, there was a peak concurrent user number of 25,551 connections Sunday, when combined with the 37,173 unique users is a pretty healthy user percentage out of the 93,054 in attendance. If you want to read a good story about how the IT team at AT&T Stadium got creative about hiding some of the almost 2,000 Wi-Fi APs they use inside the venue, download our latest report which has a feature all about those techniques.

That's sunlight, and not visible Wi-Fi beams, at AT&T Stadium. Photo:

That’s sunlight, and not visible Wi-Fi beams, at AT&T Stadium. Photo:

Husker Wi-Fi: Nebraska fans use 4.2 TB of Wi-Fi data during Sept. 12 home game

It looks like we have an early leader in the (unofficial) college football Wi-Fi usage race, as the University of Nebraska folks are claiming that fans at Memorial Stadium in Lincoln, Neb., used 4.2 terabytes of Wi-Fi data during the Huskers’ Sept. 12 victory over South Alabama.

Thanks to Chad Chisea, IT operations manager for the Huskers and Dan Floyd, Nebraska’s director of IT for athletics, we’ve got some stats and tweets to share — of the 4.2 TB, approximately 3.0 TB was downloaded data and 1.2 was uploaded, according to network stats sent to us via email. But if you look at the embedded tweet below, the numbers that really jump out at us are the 34,439 unique connected devices and the 28,290 peak connections at a single time — those are numbers that rival anything we’ve seen in NFL stadiums, and are dwarfed only by Super Bowl or college playoff championship game numbers.

With 89,822 in attendance to watch Nebraska whup up on South Alabama 48-9, it’s perhaps no surprise that there are pro-type numbers being put up on the Wi-Fi scoreboard. With a top deployment from Cisco and CDW put in last year, the Memorial Stadium Wi-Fi should be on par with any other large football stadium, and so far the numbers from Nebraska look to be proof of that idea. The Huskers also seem to have a good handle on promoting the Wi-Fi network, as witnessed by the two tweets below that direct fans to the network and let them know they also have game-day help available.

We’re looking forward to getting some hard stats from other top college venues — so far we’ve heard anecdotal evidence that the fiber-based network at Texas A&M’s Kyle Field is rocking, but no numbers yet — so send them our way, and let’s see how the stadium networks stack up. Right now it’s Big Red in the lead, but if DAS numbers from AT&T are any indication, there is lots more data being used this year in stadiums so let’s start adding up the scores.