Stadium Tech Report: Los Angeles Dodgers hit it out of the park with Cisco, Aruba Wi-Fi

Dodgers Stadium, the SoCal baseball shrine. All photos: Terry Sweeney, MSR (click on any photo for a larger image)

Dodgers Stadium, the SoCal baseball shrine. All photos: Terry Sweeney, MSR (click on any photo for a larger image)

Growing up in the Los Angeles suburb of Norwalk, Ralph Esquibel recalled playing outdoors while inside the Dodger game was on the radio. “I knew from the kinds of noises coming out of the house how the game was going,” he laughed. Esquibel, now vice president of IT for the Los Angeles Dodgers, may have wished for some similar indicators or guideposts as he began the wireless retrofitting of Major League Baseball’s third oldest stadium (after Boston’s Fenway Park and Chicago’s Wrigley Field) in early 2011.

Esquibel faced multiple challenges with Dodger Stadium. First, there was all that concrete to push signals through or around. There was the size of the Chavez Ravine venue and its far-flung parking lots, spanning more than 350 acres. The stadium also has few overhangs, a favorite place to attach Wi-Fi access points or distributed antenna system (DAS) gear. Then there’s Dodger Stadium’s capacity — 56,000 seats – the largest in the league and almost 30 percent larger than the average MLB stadium (42,790).

Esquibel’s biggest hurdle? ” Trying to achieve the network that we wanted but also maintain an appropriate budget for the solution,” he said. While Esquibel would not specify what the Dodgers spent, he did allow that it was “an 8-figure project.”

Coverage challenges in the best seats

Initially, the best seats in the house presented a coverage challenge; field and club level seats along the third- and first-base lines and the dugout lack any overhangs. So while phones in those sections could receive a short, directional beam sent from across the outfield, the upstream signal couldn’t get back to the AP across the field, said Esquibel.

Ralph Esquibel, VP of IT for the Dodgers, with the new Wi-Fi relief pitcher mobile.

Ralph Esquibel, VP of IT for the Dodgers, with the new Wi-Fi relief pitcher mobile.

“We wanted to guarantee a premium experience, regardless of the seat,” said Esquibel, who joined the Dodgers 6 years ago after working in IT at Toyota and Honda. So by using what he calls “a hybrid approach,” Wi-Fi APs and antennas are installed overhead where possible, but also under seats and in staircase handrails that divide the stadium’s steep aisles.

All told, nearly 1,000 APs from Cisco and Aruba Networks blanket Dodger stadium, its concession areas and parking lots. Horizon Communications helped the Dodgers with design and installation of the Wi-Fi and DAS.

The under-seat APs/Wi-Fi antennas on the club level are housed in NEMA enclosures about every 15 seats, set eight rows apart. Esquibel was concerned about losing real estate under those seats; he also didn’t want to create any potential trip hazard for fans. In addition, the Dodgers use Cat 6A cabling, whose thickness and rigidity couldn’t run up a stepped incline. Consequently, they drilled through concrete to snake the cabling through from the clubhouse underneath. “There’s no visible conduit leading into the enclosure,” Esquibel explained. The profile and footprint of the enclosure still leaves space for fans to place belongings.

Handrail Wi-Fi enclosure

Handrail Wi-Fi enclosure

It’s the same modus operandi for the enclosures housed in the stair rails, except there are two APs in larger enclosures at the top of each staircase on the reserve level and upper deck, then a single AP per enclosure as the stairways descend. Some 290 APs offer coverage on the reserve level, which by itself has a greater capacity than nearby Staples Center (18,118 seats), Esquibel told Mobile Sports Report. After 2 years of use, there have been no issues with the AP enclosures. “We power-wash the seats and stands after games and [the enclosures] are very resilient against the sun, water and wind,” Esquibel said.

He also acknowledged some early challenges with Wi-Fi. Part of the issue was working with Cisco’s CleanAir technology, which is supposed to minimize RF interference, if not eliminate it altogether. If an AP starts broadcasting over a frequency in use by another AP, for example, CleanAir helps it find another frequency. It took a few months to fully tune the network; some directional antennas needed a 10-degree adjustment, Esquibel said. Another challenge was having APs from more than one vendor. “If your network is 100 percent Cisco and all leveraging the same controllers, [CleanAir] will work perfectly,” Esquibel said. “If you have a mixed environment that pushes Wi-Fi in certain locations, it becomes a problem — there’s competition for frequencies.”

Coordinating the APs

A third-party leveraging a non-public frequency would switch channels, for example, causing the APs for public use to also switch channels. “What we had was a lot of bouncing back and forth,” Esquibel said, which affected performance. “So we assigned channels and frequencies for each AP, which still requires a lot of coordination.”

Under-seat Wi-Fi enclosure

Under-seat Wi-Fi enclosure

Since 2013, the stadium has been carved into 24 DAS sectors. AT&T, T-Mobile USA and Verizon Wireless are the carriers presently using the DAS; Ericsson makes the DAS antennas. Stubborn Sprint relies on a tower adjacent to the stadium.

Dodger fans average anywhere from 500-655 megabytes of data use per game, according to Esquibel. During a busy game, the wireless networking accommodates 16,000 concurrent users; a slower event clocks in at 4,000-8,000. To test upload speed, Esquibel will push a 50MB video to Facebook. When there’s lots of available bandwidth, he gets 60 Mbps performance; on the low end, it’s closer to 4 Mbps. Esquibel said users are mostly streaming and posting videos and photos to social media; Dodger Stadium is the second most Instagrammed site in southern California, after Disneyland, Esquibel added.

The Dodgers have their own version of Ballpark, the in-stadium MLB app, which offers video replay and highlights; in-seat ordering of food and drink in certain areas; and stadium mapping. Check-ins on Ballpark are handled through a network of 44 iBeacons, which takes advantage of Bluetooth Low Energy (BTLE) technology. Between Ballpark and social media activity, Dodger fans have run up as much as 700 MB data usage during games — and the network is ready if more demand is needed.

“We don’t do any rate limiting, so if we consume all our bandwidth we get a free upgrade, thanks to a clause in our agreement with our ISP, AT&T,” Esquibel explained.

To ensure a family-friendly and wholesome environment, the Dodgers use Palo Alto Networks 5020 firewalls for content filtering. “As we developed our SLAs, it was one of the first issues to pop up — no sexual content, no malware/phishing, and no illegal drug sites,” he said.

What’s on his wish list for the future? “I’d like geo-fencing within the Wi-Fi network so if I see someone enter a club, I can say hi or welcome them, notify them of specials, or flag points of interest around the stadium,” Esquibel said, like the World Series trophy case or giveaway locations for promotional items. Alongside all the other applications, wireless can be used as guideposts for fans and visitors to Dodger Stadium.

Extreme, DGP part of Wi-Fi and DAS upgrade coming to Florida State’s Doak Campbell Stadium

Screen shot 2015-09-03 at 10.47.36 PMExtreme Networks and DAS Group Professionals have been selected by IMG to build a new Wi-Fi and DAS network respectively at Florida State University’s Doak Campbell Stadium, the 82,300-seat home of the Seminoles football team, according to the school’s website.

Extreme, which has mainly made its stadium name in pro football venues, has one other college deployment that we know of but nothing on the scale of regular national-championship caliber FSU. DGP, which was relatively unknown until its Levi’s Stadium deployment last year, is now racking up big deals in a hurry, following its selection (also alongside Extreme) by the Chicago Cubs for new communications infrastructure at Wrigley Field.

Since the press release said the deals are only so far at the “evaluation” stage that means that Wi-Fi and DAS aren’t available yet. More details on this deal as we hear them, in what is shaping up to be a very busy season of new stadium deployments. This also appears to be the first fruit of the Extreme/IMG relationship announced last winter.

Verizon makes live NFL action via NFL Mobile free to all customers; RedZone still $1.99 a month

Screen shot of NFL Mobile app showing possible live games to watch.

Screen shot of NFL Mobile app showing possible live games to watch.

If you’re a Verizon Wireless customer, watching live NFL action on your smartphone just got a lot easier and perhaps cheaper. For the 2016 pro football season, Verizon will give all its customers free access to the schedule of live NFL games available via the NFL Mobile app, eliminating a $5 fee that used to be charged to non-premium plan users.

Though every game isn’t available, the NFL Mobile live schedule is pretty good, as it includes Sunday Night Football games, Monday Night games, Thursday night games and Sunday local games and also preseason games like tonight’s action. Also free is non-live action programming from the NFL Network, but if you want the prize catch — the NFL Network’s RedZone Channel — you still need to pay an extra $1.99 a month, and it’s a good idea to register NOW as Verizon has a history of having technical difficulty when people call on opening day.

(Don’t say we didn’t warn you!)

And, as always, though the games are free the data usage is not, so if you are on cellular games will count against your service plan; our best suggestion is to find free Wi-Fi.

If you are a savvy NFL watcher, you know that the RedZone Channel is the “secret” extra window to live action, especially on the West coast where we would often see entire games shown on RedZone simply because there weren’t other games to switch back and forth to. If you are a customer of any other cellular service, you are mostly out of luck in getting live football on your phone, since Verizon has an exclusive $1 billion deal with the NFL to be the sole carrier of live action on phone-type devices for another 2 years.

What’s still a muddle is whether or not you will be able to watch live action via NFL Mobile when you are at an NFL stadium, something we are working on trying to find the answers to. Neither the league nor Verizon has ever offered up anything like a guide to whether or not you can use the app to watch other games while at a game, even though with fantasy football it’s a good bet that many NFL ticketholders would like to have access to games other than the one they are attending. You can blame local and stadium broadcasting rights for most of the places that block NFL Mobile live games, along with substandard wireless services, but our hope is that the situation will improve as more teams put in Wi-Fi and upgrade their DAS.

You may also see a message prompting you to sign up for a new $99.99 (or so) package called Game Pass, which is a new all-inclusive video subscription deal from the league that includes live game audio and a lot of other features, but no live game action. Look for more info on that package and a revamped NFL Mobile app next week, and let’s all cross our fingers and burn incense to hope that Verizon added a server or two so many of us aren’t frustrated when the footballs fly.

Wi-Fi for the Frozen Tundra: Extreme, Verizon bring Wi-Fi to Green Bay Packers’ Lambeau Field

Lambeau Field, home of the Green Bay Packers, now has Wi-Fi for fans. All photos: Green Bay Packers (click on any photo for a larger image)

Lambeau Field, home of the Green Bay Packers, now has Wi-Fi for fans. All photos: Green Bay Packers (click on any photo for a larger image)

Lambeau Field, the Green Bay Packers’ historic home, now has full fan-facing Wi-Fi services thanks to a deployment led by the Packers, Extreme Networks and Verizon Wireless.

Much like the deployment last year at the Seattle Seahawks’ CenturyLink Field, the Lambeau network will be ready for this season’s games and will feature separate Wi-Fi SSIDs for Verizon customers and for all other subscribers, according to the Packers and Extreme. The network, which was installed earlier this year, has approximately 1,000 access points in and around the venue, many on handrail enclosures to provide service to the large bowl seating areas where there are no adjacent overhangs.

Lambeau bench seating with railing-mounted Wi-Fi APs

Lambeau bench seating with railing-mounted Wi-Fi APs

According to the Packers, the network was live in a “testing” mode for some pre-football season events this summer, including a Kenny Chesney concert and the Brett Favre Packers Hall of Fame induction ceremony. Wayne Wichlacz, director of IT for the Packers, said the network wasn’t advertised at those events but was still found and used by fans in attendance.

Like at other Extreme deployments, the Packers will put together a group of “Wi-Fi coaches,” network-savvy people who will roam the stands on game days to help fans connect. According to the Packers they will partner with and help train local high school children to be the “coaches,” a unique twist.

Green Bay is the second NFL franchise to announce a new network built by Extreme for the upcoming season, following the news of an Extreme network being installed at the Baltimore Ravens’ M&T Bank Stadium. By our unofficial count this is the eighth NFL stadium to get an Extreme Wi-Fi deployment.

Wi-Fi APs visible on press box structure

Wi-Fi APs visible on press box structure

Verizon, which does not comment publicly on its Wi-Fi deployments, has also backed Wi-Fi networks for NFL stadiums in Seattle and Detroit, as well as at the Staples Center in Los Angeles. If the Green Bay network is like Seattle’s, Verizon customers can set their devices to automatically connect to the Wi-Fi network when it is detected. There will be no extra charge for non-Verizon users at Lambeau, and again if the network works like Seattle’s there won’t be any difference in performance between Verizon and non-Verizon customers on the Wi-Fi network. Verizon also built the DAS at Lambeau, which was already operational prior to this season. It’s not known if other carriers are on the Verizon DAS or not.

The deployment at Lambeau was no doubt a special challenge, given the historic nature of the venue and the lack of overhang space for APs for much of the bowl seating. Look for a more detailed profile of the network deployment in our upcoming Stadium Tech Report next month!

Taylor Swift shows card 3+ TB each, and push Levi’s Stadium Wi-Fi past half-million user mark

Taylor Swift at Levi's Stadium. All photos: Levi's Stadium (click on any picture for a larger image)

Taylor Swift at Levi’s Stadium. All photos: Levi’s Stadium (click on any picture for a larger image)

Because her humungous stage kept Levi’s Stadium from filling to capacity, Taylor Swift’s Aug. 14 and 15 shows didn’t come close to setting a Wi-Fi capacity record at the venue, as some had predicted. But with almost 20,000 unique users the first night and almost 24,000 the second, the “1989” tour events pushed the cumulative Levi’s Stadium Wi-Fi user number past the half-million mark, just more than a year after the venue officially opened.

Thanks to Levi’s Stadium owners and operators, the San Francisco 49ers, we have some very interesting statistics about Wi-Fi use at large outdoor stadiums. Now that 509,524 unique users have logged in to the Levi’s Stadium Wi-Fi network during its numerous football and other events, the networking team has some interesting observations, including the fact that concertgoers use more bandwidth than football fans.

Over the 28 events hosted at the stadium in the 54 weeks (not counting this past weekend’s preseason game with the Dallas Cowboys) the Levi’s Stadium Wi-Fi network has carried a total of 56.88 terabytes of data, according to figures sent to us by Roger Hacker, senior manager, corporate communications for the Niners. One of the more interesting tidbits was the sustained connectivity during concert events; from Hacker, these direct notes:

Concerts generate up to 65% more load per fan than sporting events.

o For 2014 NFL season, the average fan on Wi-Fi used 100MB.

o For One Direction and Taylor concerts, the average fan consumed 164MB.

Concerts are generating higher sustained loads on the network than sporting events.

o For One Direction, the network was over 2 Gbps for 1 hour and 15 minutes continuously.

o For Taylor the network was over 2 Gbps for a total of 1 hour and 10 minutes continuously.

At many venues with Wi-Fi, the so-called “take rate” or the number of users actually logging in to the network is one key piece of data about how well the system is working. The idea is, the better it works, the more people log on. At the Taylor Swift concerts, the take rates were among the highest we’ve seen: For the first show with attendance of 50,393, there were 19,963 unique users on the Wi-Fi network; for the second show the numbers were 23,885 out of 52,479 in attendance.

Breaking 3 TB twice in a row

Let it be said, that even though the Swift concerts didn’t beat the all-time Levi’s Stadium Wi-Fi mark of 4.5 TB set by WrestleMania 31 back in March, each of the shows passed the 3 TB total mark, with 3.31 TB used on Aug. 14 and 3.807 TB used on Aug. 15. And remember, that’s with just a little more than 50,000 fans in the house, far fewer than the 76,976 who crammed in for the wrassling on March 31.

But even at other events that might not be as high on the “gotta do it” list, fans at Levi’s used the Wi-Fi in high numbers, as another of Hacker’s notes spells out:

On average, nearly 40% of people attending an event at Levi’s use the Wi-Fi.

For the first year, the average attendance at all events was 46,400 based on the VenueNext valid scans. (including smaller events as well as bigger events)

For the first year, the average unique wifi users per event was 18,200 (or 39% of attendance)

On-field Aruba Networks Wi-Fi AP at Levi's Stadium

On-field Aruba Networks Wi-Fi AP at Levi’s Stadium

Part of what might have made the Swift audience bigger users of Wi-Fi was a new, improved temporary Wi-Fi network for the field seats, an innovation crafted by Wi-Fi gear supplier Aruba Networks and the Levi’s Stadium networking team. By putting temporary Wi-Fi APs on both the walls surrounding the field level, on railings in the temporary seats and even underneath the temporary flooring, the stadium provides service to the premium-seat customers, a challenge for NFL stadiums who must keep regular Sunday-game networks from bleeding into the field area (because that’s where the league operates its own Wi-Fi network for game operations). For what it’s worth, we also heard from DAS supplier DAS Group Professionals who said they were going to have temporary DAS antennas on the field as well for the Swift concerts. No DAS stats yet, however.

Earlier this summer at the annual SEAT Conference, we had a very interesting panel discussion about which stats really matter when it comes to stadium Wi-Fi usage — though I will always remain a sucker for top-line totals, I do agree that there should probably be separate categories for events like the Swift concerts, the WrestleMania events, and regular football games. What is catching my attention more lately are numbers like the sustained connectivity and the unique and concurrent numbers of users, since in the end the bottom line on stadium Wi-Fi network performance has to be how many people are using it. A year in, it looks like the Levi’s Stadium network has passed a crucial test, in that people are both finding it and using it, even if they are not regular attendees or season-ticket holders. Those facts say a lot about how well the Levi’s Stadium system works, and should be numbers stadium tech professionals look to when assessing their own deployments.

More Swift shots below. Enjoy.

ts3

ts4

ts5

Boingo, Aruba behind new Wi-Fi deployment at Kansas State’s football stadium

The WIldcats take the field at Bill Snyder Family Stadium. All photos: Kansas State website. (click on any photo for a larger image)

The WIldcats take the field at Bill Snyder Family Stadium. All photos: Kansas State website. (click on any photo for a larger image)

So far we haven’t seen any press releases but a web-posted letter to Kansas State supporters from Athletic Director John Currie confirms that Boingo Wireless and Aruba Networks are part of a new Wi-Fi and DAS deployment at the Bill Snyder Family Stadium that will have at least 300 antennas of some kind bringing service to the 50,000-seat venue.

Bill Snyder, still the head coach at the stadium that bears his name, will lead the Wildcats into the 2015 football season knowing that fans in Manhattan — Manhattan, Kansas — will now have sufficient bandwidth to stay connected while they cheer on the team. While the letter from AD Currie wasn’t clear on details and specifics — at least not to the MSR level of clarity — it did say that “When you arrive for Fan Appreciation Day this weekend [last weekend] you’ll likely notice a few of the 300-plus new Wi-Fi and Distributed Antenna System (DAS) fixtures being installed around Bill Snyder Family Stadium as we work to address connectivity for cell phones and other personal electronic devices throughout the stadium.” So we’re not sure if that means 300 APs for Wi-Fi and then a DAS, or if that means 300 antenna systems for both. We’ve got messages out to everyone involved, so watch for a follow-up with some more details as they are available.

Screenshot of map on new K-State app.

Screenshot of map on new K-State app.

The letter from Currie went on to say that the network will be in a testing mode the first couple games, and then the school and its new communication partners will move on to bring similar connectivity to Bramlage Coliseum, the K-State hoops arena, before the basketball season starts.

New app as well

K-State also has a new gameday football app, built by the Boulder, Colo.-based Sportslabs, the first big program we’ve heard of that is using Sportslabs. We haven’t yet seen the app in action but according to the K-State website plug it seems like it will have most of the usual bells and whistles, but not anything fancier like instant replays or food ordering. Stay tuned as we get more info.

The K-State deal is a return to the public eye for Boingo, which had a spate of venue announcements a year or so ago and then pulled back out of the spotlight until coming back a bit recently with an NBA announcement. Boingo using Aruba gear is also a new twist for us, but not surprising as HP’s recent acquistion Aruba has been steadily winning new stadium deals and continues to innovate at already-deployed venues (watch for an upcoming report about the temporary on-field Wi-Fi network that Aruba and the San Francisco 49ers deploy for concerts and other events at Levi’s Stadium).

Are you ready for some football?

Are you ready for some football?