Stadium Tech Report: DAS Group Professionals makes a name for itself with Levi’s Stadium DAS

DAS antennas hanging from a Levi's Stadium overhang. Credit, all Levi's photos: Paul Kapsutka, MSR

DAS antennas hanging from a Levi’s Stadium overhang. Credit, all Levi’s photos: Paul Kapsutka, MSR

If you look around at the walls, ceilings and overhangs at Levi’s Stadium, it’s hard to miss the small square boxes with the off-white color and a “DGP” logo in one corner. While the wires hanging out the back of each box make it an easy guess that the equipment has something to do with wireless networks, even many industry insiders may not know the company behind the boxes and the three-letter acronym.

Meet DAS Group Professionals, the Bay Area firm in charge of deploying a distributed antenna system (DAS) to make sure your cell phone gets a good signal at San Francisco 49ers games and any other event inside the 68,500-seat Levi’s Stadium. And while you might not be familiar with DGP, rest assured the company is extremely familiar with cellular deployments for large public venues, having installed similar DAS networks for airports, casinos and hotels, and even for San Francisco’s BART train system. Of course, most of that work was done when the company was called Forza Telecom, before changing its name to DGP earlier this year, another reason why “DGP” may not have rung any bells.

“It may appear like we just fell out of the sky, but we’ve actually built quite a few [DAS] systems,” said Steve Dutto, president of DGP, in a recent phone interview. “We’ve got years of experience.”

DAS antennas above a food stand

DAS antennas above a food stand

DAS: The Rodney Dangerfield of stadium connectivity

One thing that keeps firms like DGP in the shadows is the relative obscurity of DAS itself. While most people generally understand how cell phones work — you turn on your phone, and it connects to an antenna somewhere on a tower or rooftop — in crowded public facilities like stadiums, traditional cellular networks with towers several miles apart can’t handle the concentrated capacity. To provide connectivity for areas with large crowds, the latest tactic is to deploy a DAS, a network of lots of smaller antennas. Originally deployed in places like office buildings, hotels and convention centers, DAS is rapidly gaining favor in stadiums and arenas, helping to alleviate the “no signal” problem that has cropped up in many venues the past few years.

And while stadium Wi-Fi gets lots of headlines whenever it gets deployed — probably thanks again to the widespread understanding of how Wi-Fi works — there are already far more DAS deployments in stadiums than Wi-Fi, mainly because cellular carriers will pay almost all the associated costs of a DAS buildout to make sure their customers get a signal. According to our most recent 2014 State of the Stadium survey, 71.4 percent of our respondents said they had a full DAS at their facility, while only 35.7 percent had fan-facing Wi-Fi.

Steve Dutto, president, DGP

Steve Dutto, president, DGP

How does a DAS work? Usually, either a major cellular carrier or a third-party “neutral” host like DGP will build out the antenna infrastructure, which includes many small antennas and then cables to bring the connections back to a wiring room or data center. There, cellular carriers install their own cell-tower back-end gear to authenticate customers and to provide a connection to the company networks and the Internet. Since it’s in the cellular carriers’ interest to keep their customers connected (and using billable minutes and data), carriers will often pay the full cost of a DAS infrastructure by building and running it themselves. In the case like Levi’s, where DGP is the “neutral host,” DGP builds the infrastructure and then charges cellular carriers to use it.

Such deals are rarely publicized, and Dutto would not comment on how much each carrier was being charged to use the Levi’s DAS — though industry gossip has the figure somewhere around $5 million per carrier per year. And just like fight club, for many deployments the first rule of DAS is that you don’t talk about DAS, because no cellular carrier ever wants to admit that its network might need help. So just like Rodney Dangerfield, DAS often doesn’t get a lot of public respect. But at Levi’s Stadium and many other sports and entertainment venues, DAS is a booming business that would be sorely missed if it wasn’t there.

The ‘dream and the nightmare’ of building the Levi’s DAS

Now that DGP’s 700-plus antenna DAS deployment is up and running at Levi’s, Dutto can breathe a small bit easier. While the network is good business for the company and an obvious prominent calling card for the future, the aggressive deployment timeframe probably isn’t something Dutto is eager to repeat.

“Levi’s Stadium was both a dream and a nightmare [for DGP],” Dutto said, due in part both to the Niners’ aggressive performance expectations and the rapid buildout schedule. Of course, DGP was somewhat used to working quickly with the Niners — when the company put a DAS in Candlestick Park back in 2012 to solve that stadium’s legendary lack of connectivity, Dutto said it was deployed “in about 90 days.”

The success of the Candlestick deployment, Dutto said, led to the Niners offering the Levi’s DAS gig to DGP. With it, however, came the need to match the team’s out-front statements about how the stadium was going to be the best ever in terms of wireless connectivity. And with many people not knowing or not bothering to switch their phones to Wi-Fi, the Levi’s DAS, like most stadium DAS deployments, would probably handle most of the wireless connections.

Door sign for head end equipment room at Levi's Stadium (there are many of these)

Door sign for head end equipment room at Levi’s Stadium (there are many of these)

“We knew that it needed to be significantly better than anywhere else, right at the launch,” Dutto said. While the late addition of an early August soccer game at Levi’s pushed deployment schedules ahead even faster, Dutto said in the end it helped DGP overall.

“I wasn’t a big fan of getting ready for that date [the Aug. 2 soccer game was Levi’s first event] but it was a blessing in disguise,” Dutto said. “The trouble with a network is that you can’t really test it until everyone shows up. We got some good data from that event.”

After the “daily and nightly” discussions with the Niners’ tech team about antenna placements and other matters, the DAS network performed well when it mattered, during the Niners’ two preseason games on Aug. 17 and 24. According to a traffic report from the stadium tech team, the DAS network carried a combined 1.02 terabytes of wireless traffic for the two preseason games, which is on par with activity seen at big events in the past, like Super Bowls. According to Dutto DGP’s internal tests showed that few, if any, calls were dropped or didn’t connect.

“We’re at 98 percent [network performance] already, without [the network] being fully optimized,” Dutto said.

RF challenges and too many iPhone 4 customers

While many of the stadium’s Wi-Fi antennas are well hidden — including the ones in boxes under seats — the DAS antennas are a bit more prominent, especially if you are near where the first overhang comes close to the seats.

“Unfortunately, with Levi’s exposed steel-beam construction, if you want the DAS to work, you’re going to see it,” Dutto said. “It’s hard to be stealthy in there.”

And in many cases there isn’t just one but instead two DGP antennas side by side, which reflects the company’s decision to actually build two parallel DAS systems to better accomodate more wireless carriers. According to Dutto, AT&T and T-Mobile are based on one system, while Verizon Wireless, Sprint and public safety communications are handled by the other one. Currently, all the carriers are live on the DAS except for Sprint, which is still in the process of installing its back-end equipment.

DAS antenna in "Faithful Mile" area

DAS antenna in “Faithful Mile” area

One of the biggest challenges for DGP, Dutto said, comes from outside the stadium, and not inside. Like other open-air stadiums located in city cores, Levi’s Stadium faces significant interference from cellular antennas on nearby office building rooftops, as well as from the Santa Clara Convention Center right across the street.

“Less than a half mile from Levi’s you can see seven different rooftop cell sites,” said Dutto, who said the flat, open terrain around the stadium increases the ability for those signals to interfere with the stadium DAS deployment. Target Field in Minneapolis had some similar problems with cell antennas on nearby office buildings.

“We’ve done a lot of work with the carriers to adjust their macro networks around the stadium,” Dutto said. “We’ll do more of that as we go, and expect it to get better.”

And while Levi’s Wi-Fi network has shown itself to be incredibly robust, Dutto said that cellular connectivity over the DAS might be even faster than Wi-Fi in many instances, especially if fans have later-model phones with 4G LTE.

Even though Dutto said DGP’s testing recorded download speeds of up to 200 Mbps — and 65 Mbps sustained — he acknowledged that many Levi’s patrons might never see those kinds of numbers unless they snap up some of the new iPhone 6 models introduced by Apple this week. According to network stats collected by DGP during the preseason games, a lot of fans may be ready for an upgrade.

“There’s a lot more iPhone 4 users out there than we thought,” Dutto said.

Team stadium apps vs. Twitter: Which one will win?

Screen shot of the home page for the Niners' Gameday Live app

Screen shot of the home page for the Niners’ Gameday Live app

Will team stadium apps be able to hold off the challenge from independent apps like Twitter? This matchup came to mind Sunday when the Mobile Sports Report team convened for a get-together at Candlestick Park, the on-the-way-out home of the San Francisco 49ers.

Since Candlestick is going to be all blowed up after this season, it’s probably not fair to single out the Niners’ app and network for poor performance this year. I mean, why build a Wi-Fi network in a place that’s going to be torn down? I will say that the new DAS seems to be working well, since I had no problems getting a cell signal all day. But when I tried to watch live video via the Niners app, it told me I had to be on stadium Wi-Fi to watch video.

But the Wi-Fi network wouldn’t connect. After long minutes and several attempts. Finally I gave up. I tried my Verizon NFLMobile app, which lets me watch RedZone on Sundays. But no! Verizon NFLMobile, which monitors your location via GPS, won’t let you watch live video or RedZone while in an NFL stadium. The only person around us with live video of anything NFL on his phone was a guy who gets the Sunday Ticket service from DirecTV. Tell me, if you’re a fan, you’re not frustrated with the idiotic hurdles the NFL puts in front of its best content to satisfy its rights deals. Guys, you’ve had several years to figure this out. It’s the biggest C’mon Man I can think of. LET US WATCH LIVE VIDEO! MAKE IT EASY!

Again to be clear: This isn’t an app review, or a formal survey. But just looking at all the phone use in the stands, I didn’t see anyone else on the Niners team app. I saw a lot of people on Facebook, on Instagram, on Twitter. Or just sending picture and text messages. What is the common thread for those apps? They are simple to use, they are fast, they have great and easy interfaces on a mobile phone. They are already filled with the people who I want to follow or communicate with. With any one of those apps, you are doing something within one or two clicks.

Fan Zone page of Niners stadium app.

Fan Zone page of Niners stadium app.

With the team apps, that’s just not the case. The Niners app — which looks like a lot of other team apps, since it’s built by stadium app market leader YinzCam — is incredibly dense, with lots of very small type. Which, while it looks OK in a screenshot like the ones here, is almost impossible to see in the harsh outdoor light of a stadium. Opening it up for the first time at the Niners game, I was underwhelmed by the overload of information and choices available. And then when the live video didn’t work… I mean, really, what else is there in the team app that could be different, or make me want to go there?

Stats? Yardages? That stuff isn’t crucial to people sitting in the stands. Where the team app could really make a difference is if it gave detailed information on what just happened in front of my eyes — you know, the kind of stuff that is instantly delivered to people at home watching games on their couch. Someone is hurt? Injured? You’re up there in the stands, you have no idea of what happened or why there are people standing around on the field. I couldn’t find an audio feed of the TV broadcast on the team app — why not have that available? Or at least the radio simulcast? What about that last play? Was it a fumble? How did Vernon Davis get a concussion? In the stands, you have one chance to see what happens. And in many cases, no way of knowing what the outcome was, especially since most teams (Niners included) only show replays of “positive” events for the home team. Again: treating fans like idiots or children is no way to make the stadium a better experience.

My simple thought, as I switched back to Twitter — where, by following some of the beat writers who cover the Niners, I was able to get almost-instant info from their press box tweets — is that the team apps seem designed to be sold to the teams and the leagues, and not with the fan in mind. I have no desire to go to the Niners’ app to find other people on Twitter to interact with or follow. If public sports websites are any guide, anything open to the public is already overrun by ignorant trolls. I’ll stick with my own Twitter feed, thanks. And now that Twitter is adding in NFL highlights, I probably have a better chance of seeing live video there than via the team apps. How are team apps, with their rights restrictions, clunky design and team-sanitized information, going to keep up with fast-moving folks like Twitter, especially now with tools like Vine or Instagram video? Anyone want to bet that we start seeing more fan replay videos on Twitter before we get good, easy to get official team replays?

Maybe these apps are working better in other stadiums, where the networks are better. My guess is, even at those places there is slow uptake. If teams really want to use technology to make the stadium a better experience than the couch, they’ve got to do more to make connecting easier. The network hookup needs to be drop dead simple. If I don’t have Wi-Fi turned on, the app should figure out how to do that itself. (Or ask when it’s first opened up, not after I’ve gone three clicks in to find the “live video” button.) Activities should be one or two clicks, not a laundry list of choices and treed menus. Though there is a lot of down time at games, it’s not that long. Apps should work faster than a play clock… if you can’t get there in 45 seconds, it’s a fail.

Safe to say, we are going to cover app development AND uptake as part of our stadium technology focus. I think right now it is the weak link in the whole connected stadium equation. One scene on the way out of the Niners game made me realize just how far behind the apps are; instead of staying in their seats to watch the crucial possible last-minute drive, many San Francisco fans were outside on the concourse… watching the TV coverage on the high-def screens above the concession stands. Because on TV, they know, they will get multiple replay angles and explanations. These fans weren’t bad fans for leaving their seats. They were, actually, just trying to find the best game-viewing experience. They should be the people interviewed next about what should be in a team app. Because what’s there now obviously isn’t reaching them. Or keeping them in their seats.

Wednesday Wi-Fi Whispers: DAS, but no Wi-Fi, for Niners at Candlestick

There’s already buzz building in Silicon Valley for the new Niners stadium being constructed in Santa Clara, as the team is already out front saying the facility will be an example of how to do stadium technology right.

Unfortunately for Niners fans, the next two home seasons will still be played in San Francisco’s Candlestick Park, which has historically been one of the worst places to try to get a cellular signal. Though a new Distributed Antenna System (DAS) deployment should help matters some this season, there is no stadium-wide Wi-Fi in the cards, a bit of a bummer since the team’s new game-day app features lots of video — which you need Wi-Fi to watch.

With an edict from the commish Roger Goodell to put Wi-Fi into every stadium, teams across the league are moving quickly to figure out how to get that done (see the second part of this post about Carolina’s new spiffy network). Caught in the middle of this deployment strategy is Candlestick, which has to be one of the worst geographic locations for wireless traffic. Not only is the stadium hidden by a small hill directly to the west (which can block signals from nearby cell towers), it is surrounded on its three other sides by the San Francisco Bay — in case you weren’t aware, wide open spaces of water also play havoc with wireless signals, and you don’t see too many antenna towers floating around.

The historically terrible cellular situation at Candlestick was brought even more to light by last year’s “blackout” game, a Monday night tilt against the Steelers that saw the stadium lose power not once but twice. Though we didn’t hear any reports of fan panic (no shaking) we did hear from a lot of folks about how nobody knew what was going on because nobody could get a cell signal to check Twitter.

To help alleivate the problem the Niners and the top three wireless carriers — Verizon Wireless, AT&T Mobility and Sprint Nextel — collaborated on the installment of a DAS system at Candlestick, essentially a bunch of small cellular antennas mounted inside the stadium to make it easier for fans to connect. Apparently there is also a day-of-game Wi-Fi network in service at Niners games, though we haven’t been able to find any press material telling us where the service can be accessed. While we wait for the Niners’ reps to get back to us we will take a wild guess and post that it is a parking-lot or concession-area only network, and not something you can use at your seat.

So, Niners fans — even though there is a spiffy new game-day app, you probably aren’t going to get much use out of the video component at Candlestick. And since it doesn’t make sense to drop a few million bucks on a Wi-Fi network that will only be used less than a couple dozen times before the team moves south, unless the Niners can figure out how to bring in a portable Wi-Fi network the Candlestick fans are probably stuck with the DAS deployment as their best connection. Though DAS deployments are better than nothing, they simply don’t have the bandwidth that a robust Wi-Fi network can bring to the table.

Carolina Gets Stadium-Wide Wi-Fi, Courtesy of AT&T

In stark contrast to the situation at Candlestick is the news from the Carolina Panthers, who will have a powerful new Wi-Fi network at Bank of America stadium in Charlotte this year, courtesy of Ma Bell.

You can read the press release and from it what jumps out at us is the 460 Wi-Fi access points, a huge number that should keep everyone there connected. According to the release the Wi-Fi access is free and easy for AT&T customers, with users of other carrier systems having to connect via a “simple login.” Anyone out there in Panther land sample the new network yet? If so give us some SpeedTest results in the comments.

NFL’s Mobile Device Stadium Strategy Slowly Coming into View

There is no official announcement we have seen but if you peruse any NFL team web page you will see a bunch of little widgets popping up saying things like “Watch 49ers games online” with a link to the new preseason and rewind tablet apps that Greg Quick wrote about last week. There are also several teams, like the Niners, who apparently have some kind of GameDay Live-branded app — if this reminds you at all of and its AtBat app strategy, it’s not a coincidence. You don’t need a press release to see what is happening, albeit a bit slowly — the NFL, like baseball, is moving to a single app for live mobile-device action, and it will cost you a bunch of extra dollars to watch it.

I think the fly in the ointment right now is the NFL’s current exclusive deal with Verizon for the NFL Mobile app, but I think that contract is up soon and I would be surprised if the NFL renews it. More likely we will see an strategy emerge, where you purchase mobile-device access on a monthly or season-long basis. For the current year the NFL will take baby steps as it tries to help teams get networks put into stadiums. But I bet by next year there is a cohesive digital device content strategy that will cost fans a few more bucks. Might be worth it though, to get other games and RedZone while you are tailgating or waiting through halftime.