MSR releases ‘State of the Stadium’ technology survey for 2014

SOS14_thumbWe cover a lot of news about stadium Wi-Fi and DAS deployments, but how many venues out there in the real world actually have such systems deployed? How are their fans using it? And what about other types of stadium-specific technology, like digital signage, social media and CRM systems? How are teams, schools and stadium owner/operators really using technology to improve the fan experience while also improving their business bottom line?

For the second straight year, we asked those questions and more and bring you the answers in our State of the Stadium Technology Survey for 2014, available for free download. Like last year’s inaugural survey, this year’s takes a look at deployments of Wi-Fi and DAS, of digital signage and CRM, and social media campaigns. It is the only numerical survey of this market that we know of, and with more than 70 respondents across all U.S. pro leagues, major colleges and other venues like race tracks and golf courses, we think it represents a pretty good snapshot of where technology deployment stands right now.

What did we find out this year? Mainly that on the wireless side, DAS deployments are far outpacing Wi-Fi, with 71.4 percent of respondents claiming a full-facility DAS, while just 35.7 percent said their facility had full fan-facing Wi-Fi. Our analysis of this situation is pretty clear as to why — with carriers willing to foot the bill for DAS, it’s no surprise that advanced cellular is ahead of Wi-Fi and in some cases is the only advanced wireless system in a stadium.

There’s more good stuff in the survey that we’ll break out over time, but for now why not just download the report and read through it yourself. We’d like to thank report sponsor SOLiD for enabling the free distribution of the survey results, and we’d also like to thank our partner the SEAT Conference, who once again helped us find willing participants to share their deployment statistics. SEAT conference attendees, of course, have had access to the survey since the most recent SEAT event in July; if you want early access to our survey results next year, just sign up for SEAT 2015 in San Francisco — which is already shaping up as an incredible event.

Stadium Tech Report: DAS Group Professionals makes a name for itself with Levi’s Stadium DAS

DAS antennas hanging from a Levi's Stadium overhang. Credit, all Levi's photos: Paul Kapsutka, MSR

DAS antennas hanging from a Levi’s Stadium overhang. Credit, all Levi’s photos: Paul Kapsutka, MSR

If you look around at the walls, ceilings and overhangs at Levi’s Stadium, it’s hard to miss the small square boxes with the off-white color and a “DGP” logo in one corner. While the wires hanging out the back of each box make it an easy guess that the equipment has something to do with wireless networks, even many industry insiders may not know the company behind the boxes and the three-letter acronym.

Meet DAS Group Professionals, the Bay Area firm in charge of deploying a distributed antenna system (DAS) to make sure your cell phone gets a good signal at San Francisco 49ers games and any other event inside the 68,500-seat Levi’s Stadium. And while you might not be familiar with DGP, rest assured the company is extremely familiar with cellular deployments for large public venues, having installed similar DAS networks for airports, casinos and hotels, and even for San Francisco’s BART train system. Of course, most of that work was done when the company was called Forza Telecom, before changing its name to DGP earlier this year, another reason why “DGP” may not have rung any bells.

“It may appear like we just fell out of the sky, but we’ve actually built quite a few [DAS] systems,” said Steve Dutto, president of DGP, in a recent phone interview. “We’ve got years of experience.”

DAS antennas above a food stand

DAS antennas above a food stand

DAS: The Rodney Dangerfield of stadium connectivity

One thing that keeps firms like DGP in the shadows is the relative obscurity of DAS itself. While most people generally understand how cell phones work — you turn on your phone, and it connects to an antenna somewhere on a tower or rooftop — in crowded public facilities like stadiums, traditional cellular networks with towers several miles apart can’t handle the concentrated capacity. To provide connectivity for areas with large crowds, the latest tactic is to deploy a DAS, a network of lots of smaller antennas. Originally deployed in places like office buildings, hotels and convention centers, DAS is rapidly gaining favor in stadiums and arenas, helping to alleviate the “no signal” problem that has cropped up in many venues the past few years.

And while stadium Wi-Fi gets lots of headlines whenever it gets deployed — probably thanks again to the widespread understanding of how Wi-Fi works — there are already far more DAS deployments in stadiums than Wi-Fi, mainly because cellular carriers will pay almost all the associated costs of a DAS buildout to make sure their customers get a signal. According to our most recent 2014 State of the Stadium survey, 71.4 percent of our respondents said they had a full DAS at their facility, while only 35.7 percent had fan-facing Wi-Fi.

Steve Dutto, president, DGP

Steve Dutto, president, DGP

How does a DAS work? Usually, either a major cellular carrier or a third-party “neutral” host like DGP will build out the antenna infrastructure, which includes many small antennas and then cables to bring the connections back to a wiring room or data center. There, cellular carriers install their own cell-tower back-end gear to authenticate customers and to provide a connection to the company networks and the Internet. Since it’s in the cellular carriers’ interest to keep their customers connected (and using billable minutes and data), carriers will often pay the full cost of a DAS infrastructure by building and running it themselves. In the case like Levi’s, where DGP is the “neutral host,” DGP builds the infrastructure and then charges cellular carriers to use it.

Such deals are rarely publicized, and Dutto would not comment on how much each carrier was being charged to use the Levi’s DAS — though industry gossip has the figure somewhere around $5 million per carrier per year. And just like fight club, for many deployments the first rule of DAS is that you don’t talk about DAS, because no cellular carrier ever wants to admit that its network might need help. So just like Rodney Dangerfield, DAS often doesn’t get a lot of public respect. But at Levi’s Stadium and many other sports and entertainment venues, DAS is a booming business that would be sorely missed if it wasn’t there.

The ‘dream and the nightmare’ of building the Levi’s DAS

Now that DGP’s 700-plus antenna DAS deployment is up and running at Levi’s, Dutto can breathe a small bit easier. While the network is good business for the company and an obvious prominent calling card for the future, the aggressive deployment timeframe probably isn’t something Dutto is eager to repeat.

“Levi’s Stadium was both a dream and a nightmare [for DGP],” Dutto said, due in part both to the Niners’ aggressive performance expectations and the rapid buildout schedule. Of course, DGP was somewhat used to working quickly with the Niners — when the company put a DAS in Candlestick Park back in 2012 to solve that stadium’s legendary lack of connectivity, Dutto said it was deployed “in about 90 days.”

The success of the Candlestick deployment, Dutto said, led to the Niners offering the Levi’s DAS gig to DGP. With it, however, came the need to match the team’s out-front statements about how the stadium was going to be the best ever in terms of wireless connectivity. And with many people not knowing or not bothering to switch their phones to Wi-Fi, the Levi’s DAS, like most stadium DAS deployments, would probably handle most of the wireless connections.

Door sign for head end equipment room at Levi's Stadium (there are many of these)

Door sign for head end equipment room at Levi’s Stadium (there are many of these)

“We knew that it needed to be significantly better than anywhere else, right at the launch,” Dutto said. While the late addition of an early August soccer game at Levi’s pushed deployment schedules ahead even faster, Dutto said in the end it helped DGP overall.

“I wasn’t a big fan of getting ready for that date [the Aug. 2 soccer game was Levi’s first event] but it was a blessing in disguise,” Dutto said. “The trouble with a network is that you can’t really test it until everyone shows up. We got some good data from that event.”

After the “daily and nightly” discussions with the Niners’ tech team about antenna placements and other matters, the DAS network performed well when it mattered, during the Niners’ two preseason games on Aug. 17 and 24. According to a traffic report from the stadium tech team, the DAS network carried a combined 1.02 terabytes of wireless traffic for the two preseason games, which is on par with activity seen at big events in the past, like Super Bowls. According to Dutto DGP’s internal tests showed that few, if any, calls were dropped or didn’t connect.

“We’re at 98 percent [network performance] already, without [the network] being fully optimized,” Dutto said.

RF challenges and too many iPhone 4 customers

While many of the stadium’s Wi-Fi antennas are well hidden — including the ones in boxes under seats — the DAS antennas are a bit more prominent, especially if you are near where the first overhang comes close to the seats.

“Unfortunately, with Levi’s exposed steel-beam construction, if you want the DAS to work, you’re going to see it,” Dutto said. “It’s hard to be stealthy in there.”

And in many cases there isn’t just one but instead two DGP antennas side by side, which reflects the company’s decision to actually build two parallel DAS systems to better accomodate more wireless carriers. According to Dutto, AT&T and T-Mobile are based on one system, while Verizon Wireless, Sprint and public safety communications are handled by the other one. Currently, all the carriers are live on the DAS except for Sprint, which is still in the process of installing its back-end equipment.

DAS antenna in "Faithful Mile" area

DAS antenna in “Faithful Mile” area

One of the biggest challenges for DGP, Dutto said, comes from outside the stadium, and not inside. Like other open-air stadiums located in city cores, Levi’s Stadium faces significant interference from cellular antennas on nearby office building rooftops, as well as from the Santa Clara Convention Center right across the street.

“Less than a half mile from Levi’s you can see seven different rooftop cell sites,” said Dutto, who said the flat, open terrain around the stadium increases the ability for those signals to interfere with the stadium DAS deployment. Target Field in Minneapolis had some similar problems with cell antennas on nearby office buildings.

“We’ve done a lot of work with the carriers to adjust their macro networks around the stadium,” Dutto said. “We’ll do more of that as we go, and expect it to get better.”

And while Levi’s Wi-Fi network has shown itself to be incredibly robust, Dutto said that cellular connectivity over the DAS might be even faster than Wi-Fi in many instances, especially if fans have later-model phones with 4G LTE.

Even though Dutto said DGP’s testing recorded download speeds of up to 200 Mbps — and 65 Mbps sustained — he acknowledged that many Levi’s patrons might never see those kinds of numbers unless they snap up some of the new iPhone 6 models introduced by Apple this week. According to network stats collected by DGP during the preseason games, a lot of fans may be ready for an upgrade.

“There’s a lot more iPhone 4 users out there than we thought,” Dutto said.

Bengals tap Extreme for Wi-Fi, TE Connectivity for DAS at Paul Brown Stadium

Screen Shot 2014-09-10 at 10.42.52 AMEven though the NFL season is barely underway, there’s one organization that already has a multi-win streak going: Extreme Networks said it will provide a Wi-Fi deployment for the Cincinnati Bengals at Paul Brown Stadium, the fifth NFL team deal for Extreme and the third announced this year.

The Wi-Fi network at Paul Brown Stadium is scheduled to go live this Sunday for the Bengals’ home opener against the Atlanta Falcons. According to the team and Extreme, the Bengals will also utilize the Extreme Wi-Fi coaches strategy, where knowledgeable network types roam the stands to help fans connect. Prior to this year, Extreme won Wi-Fi deals for the New England Patriots’ Gillette Stadium and the Philadelphia Eagles’ Lincoln Financial Field. In addition to Cincinnati, this year Extreme has announced Wi-Fi projects for the Tennessee Titans’ LP Field, and a joint deal with SignalShare to bring Wi-Fi to EverBank Field, home of the Jacksonville Jaguars.

And while we haven’t seen a press release yet we also know that TE Connectivity is in charge of installing a DAS at Paul Brown Stadium, meaning that cellular reception should improve as well.

We haven’t yet spoken to the folks at Paul Brown Stadium, but here are some prepared quotes from the press release to tell you how happy they are to be joining the NFL Wi-Fi fraternity:

“Our fans are of the utmost importance to our organization, and as technology continues to provide new possibilities, it was crucial that we identify a way to provide fans a truly differentiated in-person game day experience,” said Michael Kayes, Bengals director of technology. “The partnership with Extreme Networks provides the foundation to do just that, affording fans the connectivity they’ve grown to expect. We’re also excited to be able to offer our fans the opportunity to get field-level action through NFL RedZone.”

Here’s a Twitter pic of the press conference today:

CBS to stream entire 2014 SEC football schedule online and via app

If you’re one of those college football fans who simply can’t get enough SEC, CBS has you covered this season even if you’re not near a TV set. Starting with this weekend’s Georgia vs. South Carolina game, CBS said it will stream its entire schedule of SEC football games live online for free, and also through its Android and iOS apps. The online version of the broadcast, dubbed SEC Live, will also feature extras like an “all-22” overhead camera angle, and a special online postgame show.

The CBS slate of 16 SEC games is about one game a weekend, with a couple doubleheaders thrown in as well as the SEC championship game on Dec. 6. While the online streaming is not only convenient for those who want to watch games while away from their couch, it in many ways offers more features than the regular broadcast, with the multiple camera angles and other goodies like Twitter stream integration, player stats, highlights and polls. (Let’s see if CBS can fix the problem that plagues other online efforts, namely Twitter feeds that get ahead of the live video stream.)

The all-22 camera is one that adds a lot to football watching, especially for true football geeks who want to see plays unfold with coach-like access. ESPN had an all-22 camera among its options during the most recent BCS championship megacast. The inclusion of the multiple camera angles by CBS is good news, a signal that broadcasters are responding to fans’ wishes and using technology to answer the need. The entire package — especially the free nature of it, not requiring any cable contract qualification for access — is just another example of the savvy digital chops at CBS.

According to CBS, the SEC Live package each week will also include “a pregame show, a halftime show, and an original 5th Quarter postgame featuring analysis from CBS Sports’ lead college football analyst Gary Danielson.” We here at MSR are big fans of the postgame show with Danielson, which is sometimes conducted in a motorhome with a Skype-like feel. Good stuff.

Verizon, NFL fumble opening-day live video for NFL Mobile app — for the 2nd year in a row

Screen shot of the message NFL Mobile users are getting used to seeing on opening day. Credit: Paul Kapustka, MSR.

Screen shot of the message NFL Mobile users are getting used to seeing on opening day. Credit: Paul Kapustka, MSR.

It’s not a streak you want to continue, especially if you are the NFL, Verizon Wireless, or one of the frustrated customers who weren’t able to see live video using Verizon’s NFL Mobile application during Sunday’s season-opening games. But just like last year, when activation servers for a new version of the app got overwhelmed, issues arose again on Sunday, making live video — one of the premium features of NFL Mobile that makes the app and service so attractive — unavailable for most of the day, including the large slate of early games.

Verizon executives who spoke with MSR Monday declined to comment on the NFL Mobile issues, instead referring all questions to their partner, the NFL media team. Verizon wireless support was also apparently offline on Sunday, leaving the task of responding to the many irate messages and tweets to the small but valiant NFL Mobile Support team.

Alex Riethmiller, vice president of communications for NFL Media, said the issue once again was due to high traffic overwhelming the authentication servers, which (putting it simply) identify whether or not a customer is a Verizon subscriber, and whether or not that customer has access to the premium content. Some of the additional traffic may have been due to two changes Verizon made in NFL Mobile subscriptions from last season; one was to give free premium access (which lets you see live action) to all customers on a More Everything data plan, and the other was to charge all of those More Everything viewers another $1.99 a month if they wanted to also watch the NFL’s RedZone channel, one of the more attractive features of the app (since RedZone gives you lots of live action, even on games that might otherwise be blacked or not viewable for other reasons). Verizon customers who are not More Everything customers must pay $5 a month for access to the NFL Mobile premium content.

Theoretically, you should be able to order RedZone from this "settings" page of the app. But it didn't work when we tried to click the button.

Theoretically, you should be able to order RedZone from this “settings” page of the app. But it didn’t work when we tried to click the button.

For many NFL Mobile customers, yours truly included, any live video at all was not available for long stretches of Sunday, and after a brief 2-minute “preview” period the RedZone content was also unavailable, and the app was not allowing it to be purchased. As of Monday afternoon, the MSR official testing lab still couldn’t get the RedZone box to work — clicking it kept taking us to a page explaining the charges but not offering a way to add it to our bill.

“NFL Mobile requires a number of backend systems to ensure only Verizon customers can get live NFL video,” said Riethmiller, in a prepared statement. “We experienced issues with one of the systems that validates customers, and it took longer than anticipated to resolve because of tremendous demand. We are confident we have addressed the issue going forward.”

Kristin Rooney, Verizon Wireless director of sponsorships and branded entertainment, would not comment on the NFL Mobile outages in a phone interview Monday morning. However, Rooney was happy to talk about the new features in NFL Mobile this season, including the ability to watch local games live, even if they are out of the traditional prime-time window NFL Mobile has access to. In addition to the local games, NFL Mobile premium customers are supposed to be able to watch live games on Sunday nights, Monday nights and Thursday nights, and also have access to RedZone broadcasts during the day. That is, when the servers are working.

BUY... MORE... SERVERS

BUY… MORE… SERVERS

Another unclear option is what happens to RedZone access when NFL Mobile customers visit NFL stadiums — while Verizon blocks access to RedZone at some stadiums, at others it allows it, usually through a partnership with a team’s stadium app. Rooney, however, did not have a list of stadiums where NFL Mobile customers could watch RedZone broadcasts during a game; most famously, the service was blocked at last year’s Super Bowl in MetLife Stadium.

Rooney also declined to say just how many customers Verizon has for its NFL Mobile app, though the Google Play store says the app has between 10 and 50 million installs. However, since a non-premium version of the app is available to customers with contracts from other carriers, it’s unclear how many “premium” members were overwhelming Verizon’s servers. It’s worth noting that Verizon is no stranger to large numbers of single-day activations, as its systems support more than 100 million wireless subscribers, nation-wide.

In a related note, NFL Mobile wasn’t the only streaming product having issues Sunday. According to a report at TVPredictions.com, the DirecTV streaming service for NFL games also suffered outages Sunday, also for the second straight year.

College DAS Update: AT&T and Verizon partner for DAS at Oregon; Minnesota taps AT&T for DAS at TCF Bank Stadium

Talk about an unlikely partnership: AT&T and Verizon Wireless announced that they have co-installed a DAS (distributed antenna system) deployment at the University of Oregon’s Autzen Stadium, a 200-antenna upgrade that should make cellular access a lot better for Ducks fans this football season.

There are few business competitions as balanced and as fierce as the one between the top two U.S. cellular carriers — so why are they doing the Kumbaya thing in Eugene? We haven’t heard any confirmation yet, but our guess is that it was somewhat of a forced partnership, due to an legacy deal between Oregon and Verizon, and the new Pac-12 infrastructure deal AT&T forged last year. (If either carrier would like to call in to confirm or deny, our lines are open.)

If you’re a follower of telecom business you might want to bookmark the Oregon press release announcing the deal and deployment, since it may be the first and last time you see Verizon and AT&T together in a single press release (that isn’t about keeping the government out of telecom regulation, that is). You can read between the lines when you see that AT&T is quoted first in the press release, Verizon second. Good news is, Ducks fans will have better cellular no matter which of the two major carriers they are a customer of.

We haven’t heard yet from AT&T about whether or not the conference is fully covered with DAS yet — according to this news report from last year there were only four Pac-12 schools whose DAS wasn’t up to snuff, with Autzen clearly being one of them. We are trying to get AT&T and Pac-12 folks on the line for a general update, so stay tuned.

AT&T bringing DAS to Minnesota’s TCF Bank Stadium

Moving to the other side of the Rose Bowl rivalry, AT&T announced it had installed a DAS at the University of Minnesota’s TCF Bank Stadium, a network of 339 antennas that will also be used by Minnesota Vikings fans since the NFL team is also playing there this year and next while their new stadium gets built.

According to AT&T, fans at TCF Bank Stadium used 106 GB of data on the new network during the Gophers’ Aug. 28 home opener against Eastern Illinois.