Report excerpt: At Bat app driving the MLB digital experience bus

Editor’s note: The following excerpt from our MLB technology deployment analysis comes from our Stadium Tech Report for Q2 2014, which includes a wealth of information, research and analysis about the stadium tech marketplace. With a focus on Major League Baseball technology deployments, the report is available free for download so get your copy today. Enjoy the excerpt that follows.

At Bat driving the application bus

If there is one other thing that defines MLB’s digital advantage, it’s the league-wide requirement to use the MLB.com’s At Bat app as the only in-stadium app offered by every team. Though there is some grumbling heard from time to time from teams who want to innovate at a faster pace, for these early days of digital in-stadium experience having At Bat as a base is most likely a huge bonus, since it makes it easier for fans to learn how to find and use the features, no matter which stadium they are in.

Though we here at MSR are more in favor of an eventual open infrastructure – say, a package of MLB-approved APIs that third-party developers could use to bolster the At Bat ecosystem – in these days when fans are still learning how to connect to Wi-Fi and are still getting familiar with the idea of using their phones to purchase in-game seat upgrades or to order concessions, it’s probably not a bad idea to limit choices.

The interesting thing to watch may be to see if, in a few years, MLB has metrics to back up its all-for-one strategy, or whether the MLB digital team decides (like Apple and the iPhone) that opening up the platform could lead to more innovation. The good news for fans is, with better connectivity and more apps, going to games should be easier and more fun as time goes on.

Giants CIO Bill Schlough (left) talks with workers in the park's main DAS head end facility.

Giants CIO Bill Schlough (left) talks with workers in the park’s main DAS head end facility.

DAS upgrades are good news

Maybe the best news on the DAS front is what seems to be (finally) some benefit from the always-improving pace of technology – accord- ing to several teams we’ve talked to recently, a pleasant surprise that comes about during DAS upgrades is the fact that head end equipment footprints are actually decreasing, meaning that the space crunch often caused by DAS may be easing somewhat.

Of course, some of those space savings may be eaten right back up by additional carriers joining in, or by existing carriers adding more coverage support. A continued issue that we will keep watching is whether or not more teams and stadium owner/operators choose neutral third- party hosts for their DAS, or whether they trust that a single carrier will be able to balance the needs of all. In our interview with AT&T’s John Donovan for this issue he said that he doesn’t think any carriers want to use DAS deployments as a strategic advantage over others; we will track your stories and what happens in the wiring closet to see if his opinion reflects reality.

To read the rest of our analysis, download your free copy of our Stadium Tech Report for Q2 2014.

Report excerpt: AT&T’s Donovan talks stadium DAS

Editor’s note: The following excerpt from our exclusive interview with AT&T senior executive vice president John Donovan comes from our Stadium Tech Report for Q2 2014, which includes a wealth of information, research and analysis about the stadium tech marketplace. With a focus on Major League Baseball technology deployments, the report is available free for download so get your copy today. Enjoy the excerpt that follows.

AT&T senior executive vice president John Donovan

AT&T senior executive vice president John Donovan

John Donovan: The ‘network chief’ talks about AT&T’s successful stadium strategy

In a strategy borne by necessity, AT&T has become far and away the leader in deploying distributed antenna system (DAS) technology in stadiums and other large venues across the country. In a recent phone interview, Mobile Sports Report spoke with the man behind the plan, AT&T senior executive vice president John Donovan, who told us the hows and whys behind AT&T’s DAS strategy, and how AT&T is continuing to innovate to improve DAS performance. Donovan also offered some interesting insights about large-venue wireless consumption patterns, based on AT&T’s thorough and varied operator experience, which is likely second to none.

With 670 DAS systems deployed in large venues, including 150 in the past year, AT&T has no plans to slow down. In fact, Donovan said AT&T will deploy another 250 DAS systems this year, adding to its impressive totals for presence in large public venues. According to Donovan, AT&T DAS systems are currently active in 75 percent of the “big 4” professional sports venues in the U.S. (football, baseball, basketball and hockey), a list thaat includes 90 percent of NFL stadiums. AT&T also has DAS deployments in a large number of top college stadiums, making Donovan confident that his company is far and away the top provider of enhanced cellular services in stadiums.

“We are absolutely crushing the competition on performance in venues,” said Donovan, who 5 years ago spearheaded a move inside AT&T to create a team that specifically targeted large public venues. Now, the fruits of that team’s labors are paying off.

Building the DAS group

The several-hundred strong group, which Donovan said included employees with experience in tower rental operations, building right-of-way negotiations, and “young, aggressive wireless” technicians, had a mandate, he said, to target all the premium venues in the U.S., and get them a world-class wireless architecture.

“The objective was, to wire them all, really,” Donovan said. Armed with a large budget (“in the early days, it was ulimited,” Donovan said), the group started ranking every large public venue, calculating stats like “seat minutes,” a value of how often a seat in a stadium would be filled.

“The Staples Center [in Los Angeles] blew everyone away — one year they had 367 events,” said Donovan.

The main reason why AT&T had to improve cellular connectivity at large venues had to do mainly with the company’s legacy as the initial, exclusive carrier for the Apple iPhone. Though Verizon Wireless and other carriers eventually got access to the iPhone in 2011, AT&T’s early lead meant that many iPhone owners were still AT&T customers — and according to Donovan, the kind of people who bought iPhones were also the kind of people who went to sporting events and concerts.

“In the early days, we were the only ones with the problem [of congestion in arenas],” Donovan said. “If you take the demographic of an event, and map it to the demographic of an iPhone buyer, you get a big overlap.” While AT&T may have only had 30 percent of the overall wireless market share, in some arenas Donovan said AT&T’s “internal” market share could represent as many as 75 percent of the fans in attendance.

In those early days of a few years ago, with many of AT&T’s iPhones still using older 2G and 3G technology, making them work in crowded arenas was a challenge, Donovan said.

“We were really forced to innovate around architectures and manage RF [radio frequency],” Donovan said. “We got really good at design.”

Fast-forward to 2014, and the team is in a much different space, innovating ahead of the curve instead of scrambling to respond to pressures. One example of the new thinking is the debut of some large, spherical antennas that AT&T used at the Coachella music festival in April.

“We had this huge thing that looked like a human-sized bowling ball with 12 [antenna] sectors in it,” Donovan said. “We’ve got a 20-sector version coming out next year.” AT&T in the last couple years has also debuted antennas that allow the carrier to focus signals into smaller geographical space, to better target the packed crowds in arenas and large venues.

“We’ve gotten a lot better at design,” Donovan said.

To read the rest of the interview, download your free copy of our Stadium Tech Report for Q2 2014.

AT&T: Hoops fans use more data than hockey fans

In a somewhat-not-surprising statistical revelation, AT&T said that basketball fans used more wireless data on its network than hockey fans at the respective arenas during both leagues’ recent championship series.

Using measurements of only AT&T customer traffic from the AT&T digital antenna system (DAS) deployments in arenas in Miami, San Antonio, Los Angeles and New York, AT&T said that hoops fans at the NBA Finals had both higher average data consumption rates and peak data rates than their NHL-watching counterparts. And when it came to home-fan data use, Miami’s American Airlines Arena hit the highest mark, with an average of 177 gigabytes of data used at the two games played in South Beach.

Though the San Antonio Spurs won the NBA title, fans at AT&T Center in Texas used an average of 138 GB of data. Miami’s arena also generated the highest peak data total of 223 GB of data. Of course maybe most of that was Miami fans using OpenTable to make early dinner reservations as the Spurs started blowing the Heat off the court.

On the frozen side of things, Los Angeles won both the real title and the data title, with fans in the Staples Center using an average of 98 GB of data during the three games there during the Stanley Cup Final. The average data usage in New York at Madison Square Garden was 83 GB of data on the AT&T network.

In defense of hockey fans, it’s really no surprise that they used less data since hockey games, especially playoff games, are mostly action and excitement, and not a million time outs. Plus, we all know that had the Chicago Blackhawks been rightfully in the Final to defend their title from last year, Da Hawks fans would have pushed everyone to shame with video renditions of the Chelsea Dagger. Next year.

Stadium Tech Report: Miami Marlins rely on ExteNet DAS to keep wireless traffic flowing

Marlins Park. Credit all photos: Miami Marlins.

Marlins Park. Credit all photos: Miami Marlins.

If you know anything about Marlins Park, maybe it’s the stadium’s unique retractable roof or the spectacular art that catches your eye. But there’s also something you can’t see that is equally exciting, at least when it comes to the in-stadium connectivity experience: A neutral-host distributed antenna system (DAS) that has more than kept pace with the rapid, continual increase in fan cellular activity.

“When it came to DAS, we were ahead of the game,” said David Enriquez, senior director of information technology for the Miami Marlins, in a recent phone interview. Well before the 37,000-seat stadium opened in 2012, Enriquez said the Marlins’ IT team was researching and planning for enhanced cellular connectivity – even before “DAS” became a hot industry acronym.

“We planned for a DAS even before they were in vogue,” said Enriquez. “We saw it as a necessary evil.”

With the iPhone and all its cataclysmic changes already in motion, Enriquez said the Marlins wanted to avoid what had happened recently at another arena that opened in the Sunshine state without good connectivity.

“What we didn’t want to see was something like what happened in Orlando, when they opened the arena [in 2010], it had bad coverage, and they were crucified in the press for bad [cellular] service,” Enriquez said. “We said, what we’d love to have is the complete opposite of that.”

David Enriquez

David Enriquez

At the opening of Marlins Park, the connectivity inside the walls was better than most, with a full-park Wi-Fi network using gear from Meru Networks and a neutral-host DAS deployed by integrator ExteNet Systems. And though Wi-Fi often gets the headlines when there is talk about stadium networks, in many facilities like Marlins Park, the DAS is an equal workhorse, since many fans still either don’t know how or don’t take the time to switch their devices over to Wi-Fi.

DAS is the workhorse

According to Enriquez, on an average night at the ballpark the Wi-Fi network will handle 40 percent of the wireless traffic, with the DAS taking care of the other 60 percent. That may be because of lack of knowledge, or perhaps satisfaction with the signal the DAS is giving them, Enriquez said.

“Early on, most people, honestly, did not know how to change [their phone] to Wi-Fi,” Enriquez said. Most fans, he added, weren’t typically streaming lots of video — they may, he said, have used the MLB At Bat app to look at a replay or two, but that could all be handled by DAS. “That trend is changing though and we are seeing much more video traffic, especially with the younger generation of guests,” Enriquez said.

Marlins Park outside

Marlins Park outside

“The truth is, many users may not take the time to switch [to Wi-Fi],” Enriquez said. “If they’re getting 4 to 5 bars on their cellular signal, they’re happy.”

Though the Marlins and ExteNet now have five major carriers on their DAS – AT&T, Verizon Wireless, Sprint, T-Mobile and MetroPCS (now part of T-Mobile), Enriquez said there was a bit of the chicken and egg problem at the start.

“Early on, nobody wanted to be the first on (the neutral DAS),” Enriquez said. “ We [the stadium] were just another node. Now, 3 years later, we are a central node in the Miami area and all the carriers are here. We’re a very central location.”

Staying in neutral

Enriquez, who has considerable experience in the large-venue IT world, said that having a neutral host for the DAS eliminates any potential concerns about favoritism between service providers. Even though costs to the team or stadium may be lower if they allow a carrier to take over DAS deployment, Enriquez said that for the Marlins a neutral host was worth the extra price.

“We didn’t want an advantage to be held by one carrier,” Enriquez said. Even if a carrier says it will act as a neutral host, when one carrier owns the deployment, others can “find it hard to believe there will be an equal time slice” when it comes to antenna access.

“We just wanted to avoid that, and make it irrelevant [as a concern],” Enriquez said.

The choice of bringing in an integrator like ExteNet, he said, provides an additional streamlining of operations, as there is now a single point for vendors to interact with to work out technology and deployment issues.

“We wanted to deal with one vendor – I didn’t want to be the middleman between the carriers and the Marlins,” Enriquez said. In that regard, he said, ExteNet has been “wonderful” as a neutral host. “They deal with all the carrier issues that I have no desire to deal with,” Enriquez said.

Less space needed for DAS upgrades

And even as fan cellular bandwidth use continues to grow – requiring carriers to constantly upgrade their systems – Enriquez said that DAS infrastructure is benefiting from improved technology to the point where even as carriers upgrade, their head end footprint is shrinking.

AT&T, for instance, has upgraded its DAS presence in Marlins Park four times over the past 2 years, Enriquez said, to the point where the carrier now has coverage for all four frequency bands. “They [AT&T] have done quite a bit to expand their coverage,” Enriquez said.

Still, the Marlins Park DAS head end hasn’t had to find new space beyond its original 1,500-square foot enclosure.

“Every time someone comes in to replace gear, we have a smaller [DAS] footprint,” Enriquez said. “It’s not going to eat you out of house and home anymore.”

Like other stadium IT directors, Enriquez is still surprised by the amount of wireless traffic generated by the fans who come to the games. “It’s incredible to see the need [for bandwidth” grow,” he said. “But people continue to give our network a thumbs up, we see that in our guest comments all the time. I just don’t know what we would do without the DAS.”

AT&T scores Wi-Fi and IPTV deal at Wisconsin, DAS deal for Arkansas hoops

Camp Randall Stadium, University of Wisconsin. Credit: David Stluka/UW

Camp Randall Stadium, University of Wisconsin. Credit: David Stluka/UW

AT&T’s march into the college stadium marketplace continues apace, with announcements today of a Wi-Fi and IPTV deal for the University of Wisconsin’s Camp Randall Stadium, and a DAS deployment at the University of Arkansas’s hoops home, the Bud Walton Arena.

While AT&T has been extremely active pushing primarily DAS deployments into college and pro stadiums, the IPTV deal for the 80,321-seat Camp Randall Stadium is an interesting twist (and one we’ll try to find out more about in an interview later on). For right now all we have to work with is this press release, which says AT&T will deploy an in-stadium video solution that brings “innovative video and digital content distribution solution on nearly 700 high definition screens.” To us this sounds like it might be a Cisco SportsVision deployment, but we’ll double-check since it’s the first time we’ve heard of a video deployment with an AT&T branding behind it.

On the Wi-Fi side, it seems AT&T is doing its usual job of high quality engineering, with 750 access points planned, according to the press release.

At Arkansas, where AT&T had previously deployed DAS in the football arena, it will now also do so in the 19,200-seat Bud Walton Arena. No word yet if this is a neutral host deployment that AT&T will allow other carriers onto, or if it is an AT&T-only deal.

More photos below!

Head end gear at the Bud Walton Arena. Credit: AT&T.

Head end gear at the Bud Walton Arena. Credit: AT&T.

Bud Walton Arena, University of Arkansas. Credit: University of Arkansas website.

Bud Walton Arena, University of Arkansas. Credit: University of Arkansas website.

Stadium Tech Report: Upgrades keep San Francisco Giants and AT&T Park at front of stadium DAS and Wi-Fi league

Outside AT&T Park. All photos, Paul Kapustka, Mobile Sports Report. (Click on any photo for larger image)

Outside AT&T Park. All photos, Paul Kapustka, Mobile Sports Report. (Click on any photo for larger image)

What’s it like when the best-connected park in Major League Baseball loses its cellular mojo for a month? This winter the San Francisco Giants found out how fun it isn’t to revisit the days of “no signal,” when a DAS upgrade meant about 30 days of little to no connectivity inside AT&T Park.

“It was painful,” said Bill Schlough, senior vice president and chief information officer for the Giants, during a recent in-person interview at AT&T Park. Though no big sporting events took place during the Feburary-to-March overhaul of the main AT&T distributed antenna system (DAS) head end, Schlough said during that time many of the roughly 200 to 300 employees who work at AT&T Park every day were forced to find daylight to make a call, just like the bad old days before DAS.

“We never really knew how much we rely on DAS [for internal operations], but having it down really drove it home,” said Schlough. The good news on the DAS front was that once the upgrade was complete, the Giants had a lot more space in their previously cramped head-end headquarters. According to Schlough, the new back-end equipment for AT&T’s DAS operations takes up less than 50 percent of the previous gear footprint, room that is likely to be filled with gear from yet another carrier slated to join the AT&T neutral-host DAS later this season.

Painful, but worth it.

Second major upgrade in 5 years of DAS

Giants CIO Bill Schlough (left) talks with workers in the park's main DAS head end facility.

Giants CIO Bill Schlough (left) talks with workers in the park’s main DAS head end facility.

If you’re not familiar with a neutral DAS like the one at AT&T Park, it’s an implementation where there is one set of antennas and internal wiring, and then a “head end” where each carrier puts its cellular-specific networking gear, equipment that identifies and authorizes callers and then connects those calls or messages to fiber links back out to the Internet and beyond. As the lead provider of DAS and as the namesake sponsor of the park it makes sense that AT&T has the biggest DAS requirement on site. Verizon, which has been on the AT&T Park DAS for two years now, actually houses most of its head end gear in a separate facility nearby, and links to the AT&T Park system via fiber.

Part of this year’s DAS renovations include a new room specifically being built for Sprint’s DAS equipment, a sort of re-arrange-the-house construction project that saw the ballpark wall off half its painting services workshop to make space for Sprint’s gear. During our visit we saw workers putting up the racks that will hold the Sprint head end gear, as thick fiber cables snaked in the doorway.

Additional carrier(s) would likely be placed in the same room as AT&T and Verizon, on floor space that used to hold AT&T racks before those were un-drilled from the concrete floor and new racks were installed during the February-March overhaul. According to Schlough, the DAS upgrade (which required minimal tweaks to the previously installed DAS antennas) was the second major rip-and-replace action in the 5 years the DAS has been live at AT&T Park.

DAS performance improves over time; Wi-Fi is good too

White box at bottom is one of the under-the-seat Wi-Fi access points at AT&T Park.

White box at bottom is one of the under-the-seat Wi-Fi access points at AT&T Park.

Though Wi-Fi services in stadiums gets a lot of technology headlines, in many big arenas the DAS is an equal workhorse, connecting people who either don’t know how to or prefer not to connect to Wi-Fi services. Through the first 18 games of the 2014 season, Schlough said AT&T Park was seeing average AT&T traffic loads on the DAS of 150 Megabytes on the download side (fans requesting data) and 50 MB on the upload side (fans sending data). Figures for the Wi-Fi network (which is free to all customers) for the same span of games was an average of 400 MB download, 200 MB upload per game.

Schlough said performance stats for the AT&T portion of the DAS have improved vastly since the distributed antenna system was first put in.

“Just four or five years ago, 97 percent [connection rate] was actually relatively respectable,” Schlough said. Now, Schlough said network connect rates regularly hover in the “four nines” region, with a recent report showing a success rate of 99.9925 percent of all calls or texts going through.

The Wi-Fi network at AT&T Park, the first in any major sporting arena and still among the world’s most expansive, has more than 1,200 access points, many of which are now located beneath the seats. According to Schlough this coming offseason will likely represent the final phase of a stadium-wide deployment effort for the new, under-seat access points, which are installed symmetrically under the seats that are out in the open air.

Giants senior VP and CIO Bill Schlough, at the office

Giants senior VP and CIO Bill Schlough, at the office

Since AT&T Park doesn’t have many railings alongside the seats “in the bowl” or those in the upper decks, the under-the-seat APs were the only choice to extend Wi-Fi connectivity, he said. Though the box-like antennas do take away some under-seat storage area from approximately every 40th seat, Schlough said there haven’t been many complaints from fans about the gear.

What he has seen, however, are many compliments about the network services, especially from fellow professionals in the sports IT world.

“I get friends in the business who come here and send me texts with Speedtests attached, showing how great the Wi-Fi is,” said Schlough. My own ad hoc testing before our interview (albeit during non-game hours) showed speeds of greater than 40 Mbps for Wi-Fi just outside the park near McCovey Cove, and speeds of 25+ Mbps just outside the main gate. Schlough also showed us some of the new iBeacon antennas, which are being tested at MLB parks this summer to provide near-field communication marketing opportunities, like automatically checking fans in to the official At Bat app when they pass by a beacon. It’s just another way the best-connected park in baseball seeks to continue to improve the fan experience.

According to Schlough, the connectivity at AT&T Park doesn’t hurt when it comes to ticket sales.

“People do come here more frequently, I think, because they know there will be good connectivity,” said Schlough. “There’s no compromise. I do think that’s part of why we’re currently riding the third longest sellout streak in MLB history.”

MORE PHOTOS BELOW — CLICK ON IMAGES TO SEE LARGER VERSION

Can you find the iBeacon in the bowels of AT&T Park? It's the small grey box to the left of the other antenna.

Can you find the iBeacon in the bowels of AT&T Park? It’s the small grey box to the left of the other antenna.

Sprint's new DAS room at AT&T Park.

Sprint’s new DAS room at AT&T Park.

A close-up of the under-seat AP. Each AP requires holes drilled through concrete to provide wiring access. APs are weather-sealed, according to the Giants.

A close-up of the under-seat AP. Each AP requires holes drilled through concrete to provide wiring access. APs are weather-sealed, according to the Giants.

Bill Schlough's "old phones" collection. How many of these can you identify?

Bill Schlough’s “old phones” collection. How many of these can you identify?