Report excerpt: AT&T’s Donovan talks stadium DAS

Editor’s note: The following excerpt from our exclusive interview with AT&T senior executive vice president John Donovan comes from our Stadium Tech Report for Q2 2014, which includes a wealth of information, research and analysis about the stadium tech marketplace. With a focus on Major League Baseball technology deployments, the report is available free for download so get your copy today. Enjoy the excerpt that follows.

AT&T senior executive vice president John Donovan

AT&T senior executive vice president John Donovan

John Donovan: The ‘network chief’ talks about AT&T’s successful stadium strategy

In a strategy borne by necessity, AT&T has become far and away the leader in deploying distributed antenna system (DAS) technology in stadiums and other large venues across the country. In a recent phone interview, Mobile Sports Report spoke with the man behind the plan, AT&T senior executive vice president John Donovan, who told us the hows and whys behind AT&T’s DAS strategy, and how AT&T is continuing to innovate to improve DAS performance. Donovan also offered some interesting insights about large-venue wireless consumption patterns, based on AT&T’s thorough and varied operator experience, which is likely second to none.

With 670 DAS systems deployed in large venues, including 150 in the past year, AT&T has no plans to slow down. In fact, Donovan said AT&T will deploy another 250 DAS systems this year, adding to its impressive totals for presence in large public venues. According to Donovan, AT&T DAS systems are currently active in 75 percent of the “big 4” professional sports venues in the U.S. (football, baseball, basketball and hockey), a list thaat includes 90 percent of NFL stadiums. AT&T also has DAS deployments in a large number of top college stadiums, making Donovan confident that his company is far and away the top provider of enhanced cellular services in stadiums.

“We are absolutely crushing the competition on performance in venues,” said Donovan, who 5 years ago spearheaded a move inside AT&T to create a team that specifically targeted large public venues. Now, the fruits of that team’s labors are paying off.

Building the DAS group

The several-hundred strong group, which Donovan said included employees with experience in tower rental operations, building right-of-way negotiations, and “young, aggressive wireless” technicians, had a mandate, he said, to target all the premium venues in the U.S., and get them a world-class wireless architecture.

“The objective was, to wire them all, really,” Donovan said. Armed with a large budget (“in the early days, it was ulimited,” Donovan said), the group started ranking every large public venue, calculating stats like “seat minutes,” a value of how often a seat in a stadium would be filled.

“The Staples Center [in Los Angeles] blew everyone away — one year they had 367 events,” said Donovan.

The main reason why AT&T had to improve cellular connectivity at large venues had to do mainly with the company’s legacy as the initial, exclusive carrier for the Apple iPhone. Though Verizon Wireless and other carriers eventually got access to the iPhone in 2011, AT&T’s early lead meant that many iPhone owners were still AT&T customers — and according to Donovan, the kind of people who bought iPhones were also the kind of people who went to sporting events and concerts.

“In the early days, we were the only ones with the problem [of congestion in arenas],” Donovan said. “If you take the demographic of an event, and map it to the demographic of an iPhone buyer, you get a big overlap.” While AT&T may have only had 30 percent of the overall wireless market share, in some arenas Donovan said AT&T’s “internal” market share could represent as many as 75 percent of the fans in attendance.

In those early days of a few years ago, with many of AT&T’s iPhones still using older 2G and 3G technology, making them work in crowded arenas was a challenge, Donovan said.

“We were really forced to innovate around architectures and manage RF [radio frequency],” Donovan said. “We got really good at design.”

Fast-forward to 2014, and the team is in a much different space, innovating ahead of the curve instead of scrambling to respond to pressures. One example of the new thinking is the debut of some large, spherical antennas that AT&T used at the Coachella music festival in April.

“We had this huge thing that looked like a human-sized bowling ball with 12 [antenna] sectors in it,” Donovan said. “We’ve got a 20-sector version coming out next year.” AT&T in the last couple years has also debuted antennas that allow the carrier to focus signals into smaller geographical space, to better target the packed crowds in arenas and large venues.

“We’ve gotten a lot better at design,” Donovan said.

To read the rest of the interview, download your free copy of our Stadium Tech Report for Q2 2014.

Stadium Tech Report: Miami Marlins rely on ExteNet DAS to keep wireless traffic flowing

Marlins Park. Credit all photos: Miami Marlins.

Marlins Park. Credit all photos: Miami Marlins.

If you know anything about Marlins Park, maybe it’s the stadium’s unique retractable roof or the spectacular art that catches your eye. But there’s also something you can’t see that is equally exciting, at least when it comes to the in-stadium connectivity experience: A neutral-host distributed antenna system (DAS) that has more than kept pace with the rapid, continual increase in fan cellular activity.

“When it came to DAS, we were ahead of the game,” said David Enriquez, senior director of information technology for the Miami Marlins, in a recent phone interview. Well before the 37,000-seat stadium opened in 2012, Enriquez said the Marlins’ IT team was researching and planning for enhanced cellular connectivity – even before “DAS” became a hot industry acronym.

“We planned for a DAS even before they were in vogue,” said Enriquez. “We saw it as a necessary evil.”

With the iPhone and all its cataclysmic changes already in motion, Enriquez said the Marlins wanted to avoid what had happened recently at another arena that opened in the Sunshine state without good connectivity.

“What we didn’t want to see was something like what happened in Orlando, when they opened the arena [in 2010], it had bad coverage, and they were crucified in the press for bad [cellular] service,” Enriquez said. “We said, what we’d love to have is the complete opposite of that.”

David Enriquez

David Enriquez

At the opening of Marlins Park, the connectivity inside the walls was better than most, with a full-park Wi-Fi network using gear from Meru Networks and a neutral-host DAS deployed by integrator ExteNet Systems. And though Wi-Fi often gets the headlines when there is talk about stadium networks, in many facilities like Marlins Park, the DAS is an equal workhorse, since many fans still either don’t know how or don’t take the time to switch their devices over to Wi-Fi.

DAS is the workhorse

According to Enriquez, on an average night at the ballpark the Wi-Fi network will handle 40 percent of the wireless traffic, with the DAS taking care of the other 60 percent. That may be because of lack of knowledge, or perhaps satisfaction with the signal the DAS is giving them, Enriquez said.

“Early on, most people, honestly, did not know how to change [their phone] to Wi-Fi,” Enriquez said. Most fans, he added, weren’t typically streaming lots of video — they may, he said, have used the MLB At Bat app to look at a replay or two, but that could all be handled by DAS. “That trend is changing though and we are seeing much more video traffic, especially with the younger generation of guests,” Enriquez said.

Marlins Park outside

Marlins Park outside

“The truth is, many users may not take the time to switch [to Wi-Fi],” Enriquez said. “If they’re getting 4 to 5 bars on their cellular signal, they’re happy.”

Though the Marlins and ExteNet now have five major carriers on their DAS – AT&T, Verizon Wireless, Sprint, T-Mobile and MetroPCS (now part of T-Mobile), Enriquez said there was a bit of the chicken and egg problem at the start.

“Early on, nobody wanted to be the first on (the neutral DAS),” Enriquez said. “ We [the stadium] were just another node. Now, 3 years later, we are a central node in the Miami area and all the carriers are here. We’re a very central location.”

Staying in neutral

Enriquez, who has considerable experience in the large-venue IT world, said that having a neutral host for the DAS eliminates any potential concerns about favoritism between service providers. Even though costs to the team or stadium may be lower if they allow a carrier to take over DAS deployment, Enriquez said that for the Marlins a neutral host was worth the extra price.

“We didn’t want an advantage to be held by one carrier,” Enriquez said. Even if a carrier says it will act as a neutral host, when one carrier owns the deployment, others can “find it hard to believe there will be an equal time slice” when it comes to antenna access.

“We just wanted to avoid that, and make it irrelevant [as a concern],” Enriquez said.

The choice of bringing in an integrator like ExteNet, he said, provides an additional streamlining of operations, as there is now a single point for vendors to interact with to work out technology and deployment issues.

“We wanted to deal with one vendor – I didn’t want to be the middleman between the carriers and the Marlins,” Enriquez said. In that regard, he said, ExteNet has been “wonderful” as a neutral host. “They deal with all the carrier issues that I have no desire to deal with,” Enriquez said.

Less space needed for DAS upgrades

And even as fan cellular bandwidth use continues to grow – requiring carriers to constantly upgrade their systems – Enriquez said that DAS infrastructure is benefiting from improved technology to the point where even as carriers upgrade, their head end footprint is shrinking.

AT&T, for instance, has upgraded its DAS presence in Marlins Park four times over the past 2 years, Enriquez said, to the point where the carrier now has coverage for all four frequency bands. “They [AT&T] have done quite a bit to expand their coverage,” Enriquez said.

Still, the Marlins Park DAS head end hasn’t had to find new space beyond its original 1,500-square foot enclosure.

“Every time someone comes in to replace gear, we have a smaller [DAS] footprint,” Enriquez said. “It’s not going to eat you out of house and home anymore.”

Like other stadium IT directors, Enriquez is still surprised by the amount of wireless traffic generated by the fans who come to the games. “It’s incredible to see the need [for bandwidth” grow,” he said. “But people continue to give our network a thumbs up, we see that in our guest comments all the time. I just don’t know what we would do without the DAS.”

Stadium Tech Report: Atlanta Hawks’ Philips Arena signs Boingo for DAS, Wi-Fi

Philips Arena. Credit: Atlanta Hawks.

Philips Arena. Credit: Atlanta Hawks.

As the central event stadium of its kind in the region, Atlanta’s Philips Arena is well known for the entertainment options it offers, from professional sports to concerts, ice-show extravaganzas and even the circus. What it didn’t want to continue to be known for was its inability to provide service when fans at the stadium tried to connect to the Internet.

“We wanted to avoid the spinning [timeout] wheel of death,” said Andrew Steinberg, senior vice president and chief revenue officer for the Atlanta Hawks and for Philips Arena, in a recent phone interview. Home to the NBA’s Hawks, Philips stood out as one of just six NBA arenas without fan-facing Wi-Fi services, and with no cellular DAS.

That will change soon, thanks to a deal Philips signed with Boingo Wireless to provide free fan Wi-Fi and a neutral host DAS for the facility by the start of the next NBA season.

Andrew Steinberg

Andrew Steinberg

According to Steinberg, the area’s well-connected citizens, who probably trail only California’s Silicon Valley in devices per person, weren’t happy when they tried to access their social networks at Philips.

“We know people are coming to arena have been very challenged, just to tweet or post a picture to Instagram,” Steinberg said. “We needed to correct that.”

The process to do so actually started a couple years ago, when the Philips IT team put out a request for proposals that was very detailed in nature. “We wanted a robust backbone and infrastructure, and we were very thorough,” said Steinberg about the review process. “We wanted to find the best solution. And Boingo [Wireless] had the best fit for our plans.”

Neutral DAS a necessity

Philips Arena

Philips Arena

One of the requirements for Philips was that its DAS host needed to be a neutral third party, and not a single carrier. “I much prefer a neutral host DAS,” said Steinberg. “I’ve seen it done different ways. What I prefer is a best in class solution from a partner who can further engage the carriers.”

Boingo, Steinberg said, “delivered on that promise” and will have all the major carriers online when the DAS deployment is ready to go for the next NBA season.

Doug Lodder, vice president of business development at Boingo Wireless, said Steinberg and the Hawks were “very thorough” with their request for technology. “They took the time and did due diligence – they were way out in front of most venues [in their proposals],” Lodder said.

According to Steinberg, the infrastructure at Philips – which opened in 1999 – presented no special challenges to deployment. There was even plenty of space on site for the DAS head end, which is sometimes a challenge for stadium retrofits.

If there is a challenge, it could come from Boingo’s crews having to work around the extensive activity schedule at Philips, which includes concerts, games, and even the occasional “mudder” race with mud pits and obstacles. According to Steinberg, the arena averages about 130 events a year. But Lodder said Boingo is used to such deployment scheduling. “We can do three shifts in a row if needs be,” Lodder said.

Build it, then you can use it

Even though Philips hasn’t had high-end connectivity, that doesn’t mean Steinberg and his team haven’t been looking at what a good network will allow them to do.

“One of the foremost challenges is to increase revenue while improving the fan experience,” said Steinberg, “while also keeping the fans engaged with the game or event.”

From activities like seat upgrades to future apps, Steinberg said the Philips team knew they needed a good network, as soon as possible.

“The fan engagement potential and opportunities are large, but they require the necessity of connectivity,” said Steinberg. “We have a robust tech roadmap, and having connectivity like this was something we felt was paramount.”

Especially so in an area like Atlanta, which has more than its share of people whose lives already revolve around connectivity. Steinberg said that expectation shouldn’t end just because those people wanted to attend an event at Philips.

“We want people to experience a seamless transition from their office to our venue, and not have a disconnect,” Steinberg said.

AT&T scores Wi-Fi and IPTV deal at Wisconsin, DAS deal for Arkansas hoops

Camp Randall Stadium, University of Wisconsin. Credit: David Stluka/UW

Camp Randall Stadium, University of Wisconsin. Credit: David Stluka/UW

AT&T’s march into the college stadium marketplace continues apace, with announcements today of a Wi-Fi and IPTV deal for the University of Wisconsin’s Camp Randall Stadium, and a DAS deployment at the University of Arkansas’s hoops home, the Bud Walton Arena.

While AT&T has been extremely active pushing primarily DAS deployments into college and pro stadiums, the IPTV deal for the 80,321-seat Camp Randall Stadium is an interesting twist (and one we’ll try to find out more about in an interview later on). For right now all we have to work with is this press release, which says AT&T will deploy an in-stadium video solution that brings “innovative video and digital content distribution solution on nearly 700 high definition screens.” To us this sounds like it might be a Cisco SportsVision deployment, but we’ll double-check since it’s the first time we’ve heard of a video deployment with an AT&T branding behind it.

On the Wi-Fi side, it seems AT&T is doing its usual job of high quality engineering, with 750 access points planned, according to the press release.

At Arkansas, where AT&T had previously deployed DAS in the football arena, it will now also do so in the 19,200-seat Bud Walton Arena. No word yet if this is a neutral host deployment that AT&T will allow other carriers onto, or if it is an AT&T-only deal.

More photos below!

Head end gear at the Bud Walton Arena. Credit: AT&T.

Head end gear at the Bud Walton Arena. Credit: AT&T.

Bud Walton Arena, University of Arkansas. Credit: University of Arkansas website.

Bud Walton Arena, University of Arkansas. Credit: University of Arkansas website.

Stadium Tech Report: Upgrades keep San Francisco Giants and AT&T Park at front of stadium DAS and Wi-Fi league

Outside AT&T Park. All photos, Paul Kapustka, Mobile Sports Report. (Click on any photo for larger image)

Outside AT&T Park. All photos, Paul Kapustka, Mobile Sports Report. (Click on any photo for larger image)

What’s it like when the best-connected park in Major League Baseball loses its cellular mojo for a month? This winter the San Francisco Giants found out how fun it isn’t to revisit the days of “no signal,” when a DAS upgrade meant about 30 days of little to no connectivity inside AT&T Park.

“It was painful,” said Bill Schlough, senior vice president and chief information officer for the Giants, during a recent in-person interview at AT&T Park. Though no big sporting events took place during the Feburary-to-March overhaul of the main AT&T distributed antenna system (DAS) head end, Schlough said during that time many of the roughly 200 to 300 employees who work at AT&T Park every day were forced to find daylight to make a call, just like the bad old days before DAS.

“We never really knew how much we rely on DAS [for internal operations], but having it down really drove it home,” said Schlough. The good news on the DAS front was that once the upgrade was complete, the Giants had a lot more space in their previously cramped head-end headquarters. According to Schlough, the new back-end equipment for AT&T’s DAS operations takes up less than 50 percent of the previous gear footprint, room that is likely to be filled with gear from yet another carrier slated to join the AT&T neutral-host DAS later this season.

Painful, but worth it.

Second major upgrade in 5 years of DAS

Giants CIO Bill Schlough (left) talks with workers in the park's main DAS head end facility.

Giants CIO Bill Schlough (left) talks with workers in the park’s main DAS head end facility.

If you’re not familiar with a neutral DAS like the one at AT&T Park, it’s an implementation where there is one set of antennas and internal wiring, and then a “head end” where each carrier puts its cellular-specific networking gear, equipment that identifies and authorizes callers and then connects those calls or messages to fiber links back out to the Internet and beyond. As the lead provider of DAS and as the namesake sponsor of the park it makes sense that AT&T has the biggest DAS requirement on site. Verizon, which has been on the AT&T Park DAS for two years now, actually houses most of its head end gear in a separate facility nearby, and links to the AT&T Park system via fiber.

Part of this year’s DAS renovations include a new room specifically being built for Sprint’s DAS equipment, a sort of re-arrange-the-house construction project that saw the ballpark wall off half its painting services workshop to make space for Sprint’s gear. During our visit we saw workers putting up the racks that will hold the Sprint head end gear, as thick fiber cables snaked in the doorway.

Additional carrier(s) would likely be placed in the same room as AT&T and Verizon, on floor space that used to hold AT&T racks before those were un-drilled from the concrete floor and new racks were installed during the February-March overhaul. According to Schlough, the DAS upgrade (which required minimal tweaks to the previously installed DAS antennas) was the second major rip-and-replace action in the 5 years the DAS has been live at AT&T Park.

DAS performance improves over time; Wi-Fi is good too

White box at bottom is one of the under-the-seat Wi-Fi access points at AT&T Park.

White box at bottom is one of the under-the-seat Wi-Fi access points at AT&T Park.

Though Wi-Fi services in stadiums gets a lot of technology headlines, in many big arenas the DAS is an equal workhorse, connecting people who either don’t know how to or prefer not to connect to Wi-Fi services. Through the first 18 games of the 2014 season, Schlough said AT&T Park was seeing average AT&T traffic loads on the DAS of 150 Megabytes on the download side (fans requesting data) and 50 MB on the upload side (fans sending data). Figures for the Wi-Fi network (which is free to all customers) for the same span of games was an average of 400 MB download, 200 MB upload per game.

Schlough said performance stats for the AT&T portion of the DAS have improved vastly since the distributed antenna system was first put in.

“Just four or five years ago, 97 percent [connection rate] was actually relatively respectable,” Schlough said. Now, Schlough said network connect rates regularly hover in the “four nines” region, with a recent report showing a success rate of 99.9925 percent of all calls or texts going through.

The Wi-Fi network at AT&T Park, the first in any major sporting arena and still among the world’s most expansive, has more than 1,200 access points, many of which are now located beneath the seats. According to Schlough this coming offseason will likely represent the final phase of a stadium-wide deployment effort for the new, under-seat access points, which are installed symmetrically under the seats that are out in the open air.

Giants senior VP and CIO Bill Schlough, at the office

Giants senior VP and CIO Bill Schlough, at the office

Since AT&T Park doesn’t have many railings alongside the seats “in the bowl” or those in the upper decks, the under-the-seat APs were the only choice to extend Wi-Fi connectivity, he said. Though the box-like antennas do take away some under-seat storage area from approximately every 40th seat, Schlough said there haven’t been many complaints from fans about the gear.

What he has seen, however, are many compliments about the network services, especially from fellow professionals in the sports IT world.

“I get friends in the business who come here and send me texts with Speedtests attached, showing how great the Wi-Fi is,” said Schlough. My own ad hoc testing before our interview (albeit during non-game hours) showed speeds of greater than 40 Mbps for Wi-Fi just outside the park near McCovey Cove, and speeds of 25+ Mbps just outside the main gate. Schlough also showed us some of the new iBeacon antennas, which are being tested at MLB parks this summer to provide near-field communication marketing opportunities, like automatically checking fans in to the official At Bat app when they pass by a beacon. It’s just another way the best-connected park in baseball seeks to continue to improve the fan experience.

According to Schlough, the connectivity at AT&T Park doesn’t hurt when it comes to ticket sales.

“People do come here more frequently, I think, because they know there will be good connectivity,” said Schlough. “There’s no compromise. I do think that’s part of why we’re currently riding the third longest sellout streak in MLB history.”

MORE PHOTOS BELOW — CLICK ON IMAGES TO SEE LARGER VERSION

Can you find the iBeacon in the bowels of AT&T Park? It's the small grey box to the left of the other antenna.

Can you find the iBeacon in the bowels of AT&T Park? It’s the small grey box to the left of the other antenna.

Sprint's new DAS room at AT&T Park.

Sprint’s new DAS room at AT&T Park.

A close-up of the under-seat AP. Each AP requires holes drilled through concrete to provide wiring access. APs are weather-sealed, according to the Giants.

A close-up of the under-seat AP. Each AP requires holes drilled through concrete to provide wiring access. APs are weather-sealed, according to the Giants.

Bill Schlough's "old phones" collection. How many of these can you identify?

Bill Schlough’s “old phones” collection. How many of these can you identify?

Artemis Networks demos 20 iPads running HD video via one 5 MHz LTE channel

Artemis Networks CEO Steve Perlman shows off a "wall" of iPads, all simultaneously running video off one 5 MHz LTE channel via an Artemis pCell network. Credit: Artemis Networks.

Artemis Networks CEO Steve Perlman shows off a “wall” of iPads, all simultaneously running video off one 5 MHz LTE channel via an Artemis pCell network. Credit: Artemis Networks.

We won’t be there to confirm it, but Artemis Networks is scheduled to demonstrate its latest trick today at the Code Conference, using its revolutionary pCell network gear to stream video simultaneously to 20 Apple iPads via a single 5 MHz LTE channel.

In case you’re not a networking or cellular nerd, the ability to send that much data to a cluster of nearby devices is basically un-doable with currently deployed LTE technologies. What makes the trick possible is Artemis’s new method of combining signal interference to provide a direct connection to each client device, something that has worked well in demos but as of yet has no commercial deployment.

Still, with the track record of entrepreneur CEO Steve Perlman, best known for developing QuickTime and WebTV, it’s a good bet that Artemis will deliver something, sometime soon, to a paying customer. Artemis, which quickly added Wi-Fi and stadium networking to its target lists, says its technology addresses the big problem facing networks in large public venues, according to Perlman, who spoke with MSR in a phone interview earlier this week.

“The biggest strain on a [stadium] system is HD video,” said Perlman. And though many stadiums are adding Wi-Fi networks, almost all right now have some kind of enhanced cellular infrastructure or a cellular DAS, which these days includes LTE. A cluster of LTE devices close together trying to run video, Perlman said, “is exactly the problem [stadium networks] are facing today.”

Artemis’s pCell technology currently works only with LTE networks, though the company is working on developing a version for Wi-Fi as well. Part of the lure of the technology is that it works with off-the-shelf client devices; at the demonstration Thursday Perlman was scheduled to add the 20th iPad to his “wall of iPads” by unpacking it right at the event, to show how nothing needs to be done to client devices to add them to a pCell network.