Stadium Tech Report: San Francisco’s AT&T Park lives up to its wireless reputation

Another tough day at the office for MSR. Credit all photos: Paul Kapustka, MSR. (Click on any picture for a larger image)

Another tough day at the office for MSR. Credit all photos: Paul Kapustka, MSR. (Click on any picture for a larger image)

Many times here at Mobile Sports Report we’ve referred to AT&T Park as the “most connected” or “best connected” stadium in baseball, if not for all sports anywhere. But even after multiple network tours and numerous reports from stats given to us by Giants CIO Bill Schlough and his staff I realized that one thing I’ve never done is to roam around the park on a game day, checking to see if its renowned Wi-Fi and DAS networks perform as promised.

Now after attending a recent day game as the guest of AT&T, I can tell you that the stadium that first put in fan-facing Wi-Fi for every seat is still at the forefront of ballpark connectivity, with Wi-Fi and DAS performance that knocks the ball out of the park almost every time. I say “almost” because during my walkaround tour I was able to find one place in the park that had almost no connectivity at all — but I will also bet you that as soon as this story is published, Schlough and his team will likely be out there the next day installing some kind of coverage since he and they have an almost unmatched enthusiasm for making their fan-facing network the best it can be. That, plus a strong partnership with AT&T, gives fans at Giants games perhaps the best stadium network anywhere, with performance so good for so long that it has almost become somewhat of an afterthought, one of the best kind of compliments a network staff can receive.

Nothing beats a strong team

Dynasty is a word fans like to use around Third and King Street in San Francisco, especially after the Giants won their third World Series title in 5 years last fall. Schlough, who loves baseball and the Giants as much as he loves networks, used the press-day gathering to show off his most recent Series ring, a chunk of gold and diamonds that probably gives you a wrist-curl workout when you put it on.

AT&T Park CIO Bill Schlough shows off his World Series bling.

AT&T Park CIO Bill Schlough shows off his World Series bling.

(Point of reference: Schlough offered the ring to me to try on, but as a dyed-blue Cubs fan I refused. “Waiting for one with ‘Cubs’ on it,” I told him. As we say in Chicago, “if it takes forever.”)

You might have heard some of the early Wi-Fi stories from AT&T Park before, but they’re fun to repeat. Then known as SBC Park, after the sponsoring “Baby Bell” that would later revive the family name of AT&T, the Wi-Fi network that debuted in 2004 attracted an average of about 94 fans a game, Schlough said, diehard geeks who would have to put up with people mocking them for bringing laptops to a ballgame. Remember, the iPhone was still 3 years away from existing, and you had to stick a PC card in a laptop to connect to Wi-Fi.

Fast forward to 2015, and now for regular season games the Giants see an average of just more than 13,000 people connecting to the Wi-Fi network, a number that has basically leveled off over the past 3 years, Schlough said. What hasn’t leveled off, however, is data use — even from last season, when fans used an average of 591 gigabytes of data per game, so far this season they’re averaging 915 GB per game. That’s why this season Schlough and team will be busy adding another 400+ Cisco Wi-Fi access points to the park, a total that should hit 1,700 by the time October rolls around.

“We’ll be working hand in hand with AT&T trying to stay one step ahead of demand,” Schlough said.

AT&T Park cabling runs are SRO

AT&T Park cabling runs are SRO

Scott Mair, AT&T’s senior vice president of technology, was on hand to help lead network tours before the game and to talk about how AT&T was using a new device called the EchoBOT to help gauge network performance in the park. Though it’s a bit of inside baseball, EchoBOT — which is basically an off-the-shelf cell phone that sits in a ruggedized box — is the kind of thing that can come out of the smart-person pool at a technology giant like AT&T. Invented in-house, the EchoBOT basically gives network operators an on-the-spot way to determine not just how the network is working, but what the actual user experience is like from the end-user point of view.

With 18 EchBOTs scattered throughout AT&T Park, Schlough and AT&T can get a much more granular view of how the stadium’s network is performing, just another way of using the resources of one of the world’s biggest telecom companies helps the fan experience at AT&T Park.

The one place without Wi-Fi

With our network tours concluded and some crispy chicken fingers inhaled in the comfy confines of the AT&T suite just above the third-base line, it was time to go to work to see if the AT&T Park network could deliver as promised. Since I’ve sat in seats at the park many times and had great connectivity there, I spent my time during the afternoon game seeking out what I thought might be some of the hardest places to bring connectivity. The first, in the second-level concourse, looked like it might be a tough antenna spot, with narrow halls and lots of concrete. But bam, on my Verizon iPhone 6 Plus I got a Wi-Fi signal of 31 Mbps down, 21 Mbps up; on my companion loaner device from AT&T, an LG Optimus G Pro, I got a 4G LTE connection of 26.15 Mbps down, 18.02 up. (For all remaining measurements I’ll just use the down/up convention to save time; I was using the standard Speedtest.net app from Ookla for all measurements.)

So yeah, you can connect while you’re in line to get a hot dog.

Here's the only place we couldn't find decent Wi-Fi. #firstworldproblem

Here’s the only place we couldn’t find decent Wi-Fi. #firstworldproblem

Strolling through the concourse toward right field, I saw several Wi-Fi advertisement signs, letting fans know they should definitely connect to the network. That’s a sign of deployment confidence, unlike many parks that install Wi-Fi but don’t really promote it, perhaps in order to keep user numbers down. After walking outside to get a view of McCovey Cove and the kayaks waiting for home-run balls to clear the park’s fences for a “splash landing,” I found a spot with almost zero connectivity — in the standing-room-only area backed up against the wall overlooking the bay.

With a Wi-Fi reading of 0.93/2.23 on the Verizon device and a 0.94/3.24 on Verizon 4G the SRO perch on the promenade was easily the poorest connection I found all day. But looking around, it’s kind of a silly place to be looking at your phone since from one direction a well-hit ball might be landing on your head and in the other direction there’s great views of the San Franicsco Bay. But still it does go to show that even in the most-connected stadium perhaps on earth, it’s not easy to get a signal everywhere. With no roof overhead and no railings close by, there simply isn’t a place to put an antenna out there. (But I bet Schlough and team will soon come up with a solution.)

Concourses covered, and upper deck too

I kept wandering around the outfield concourse and found decent connectivity at the centerfield Coors Light bar, 9.32/17.31 for Wi-Fi on the Verizon device and 14.21/43.00 for Wi-Fi on the AT&T device, as well as outside the Giants’ social media cafe in left-center, 14.32/32.20 for Wi-Fi on the Verizon device and 13.61/34.70 for Wi-Fi on the AT&T phone. Unless you’re a Wi-Fi geek like me you probably won’t ever see the APs since they are painted to match the structures they’re attached to. But I could see multiple APs hanging off the centerfield scoreboard structure, a piece of architecture that helps to deliver such solid connectivity to the open outside areas.

You can see Wi-FI APs -- if you know where to look.

You can see Wi-FI APs — if you know where to look.

Taking a break in a standing area behind the left field foul pole I got a smoking result for Wi-Fi on the Verizon phone, 22.46 Mbps on the download side and an amazing 52.05 on the upload. (I think it’s important to note that some of the best signals were on a device from an AT&T competitor, a sign that the facility does a great job of ensuring that any customer will get a good signal, no matter where you purchased your phone or service plan.) Climbing the stairwells to the view deck I still got a good signal on the concourse behind the seats in upper left field — 7.24/8.75 on the Verizon phone.

Since it was getting windy and cold (summer in SF!) I ducked back inside and found an empty seat in section 332, near the upper left field corner of the park. There I got a Verizon Wi-Fi mark of 17.52/24.33, and a Verizon 4G LTE mark of 5.62/6.81, again showing that the AT&T Park DAS is also delivering solid performance for customers on other carriers. The AT&T phone at that location saw 12.46/18.79 on Wi-Fi and 12.68/17.06 on 4G LTE. According to Schlough, the AT&T neutral-host DAS, which uses CommScope ION equipment, is so good that many fans don’t even bother to switch their phones to Wi-Fi. The upper deck, or view level, is scheduled to get many of the APs slated for installation this summer, in the under-the-seat enclosures that bring the network right into the seating areas.

Conclusion: Like the Giants, AT&T Park is tough to beat

Here at MSR we get the question a lot — “what’s the park with the best network?” — and I would have to say that like its tenants, AT&T Park is tough to beat. Schlough and the impressive IT team down at AT&T Stadium have a friendly rivalry, and you can’t have the most-connected discussion without mentioning Levi’s Stadium. But the park that did Wi-Fi first continues to improve year in and year out, never resting on its historic laurels. That’s a “dynasty” that is perhaps as impressive as the one built by the team on the field.

(More AT&T Park visit pictures below)

A little hard to see, but if you look closely you can see the Giants showing fan social media posts on the big screen.

A little hard to see, but if you look closely you can see the Giants showing fan social media posts on the big screen.

These signs are up all over the park

These signs are up all over the park

EchoBOT enclosures (white) next to a Wi-FI AP

EchoBOT enclosures (white) next to a Wi-FI AP

View from the Coors Light concourse walk-up bar in center field

View from the Coors Light concourse walk-up bar in center field

Good connectivity here in left field. Maybe the buttons on the hat improved reception?

Good connectivity here in left field. Maybe the buttons on the hat improved reception?

MLB app ads greet you as you walk up the stairs at AT&T Park

MLB app ads greet you as you walk up the stairs at AT&T Park

Nice place for a ballpark, don't you think?

Nice place for a ballpark, don’t you think?

For Giants fans only: I swear that thing weighs about 3 pounds

For Giants fans only: I swear that thing weighs about 3 pounds

MSR editor Paul Kapustka, your man on the Wi-Fi scene.

MSR editor Paul Kapustka, your man on the Wi-Fi scene.

Bowman: MLB won’t stop fans from using Meerkat or Periscope at games — for now

Bob Bowman, president of business and media for Major League Baseball. Credit: Paul Kapustka, MSR

Bob Bowman, president of business and media for Major League Baseball. Credit: Paul Kapustka, MSR

At a Major League Baseball game and feeling the need to livestream some live game action via Periscope or Meerkat? Go right ahead, because the powers that be at MLB aren’t going to stop you — at least not yet.

While the nascent livestreaming services — which basically allow users to broadcast live video of what their phone cameras can see — potentially create conflicts with both broadcast rights and available network bandwidth, they aren’t yet a problem at MLB ballparks, according to Bob Bowman, president of business and media for Major League Baseball and the CEO of MLB’s advanced media operations.

Bowman, who was at Harding Park Golf Course Tuesday to announce a joint deal between MLBAM and the PGA, spoke briefly with MSR to address the livestreaming question, which surfaced earlier this month when fans started using Periscope and Meerkat to “broadcast” live video from MLB games. Though showing live video “without the express written consent” of MLB games is “strictly prohibited” (as anyone who’s ever watched a MLB game broadcast knows), Bowman said Tuesday that he and MLB don’t see livestreaming as a problem that needs to be addressed by policing fans or blocking the services.

Screen Shot 2015-03-29 at 11.20.55 PM“I just don’t think our fans come to games with the idea of holding their phones up to stream video for 20 minutes,” Bowman said. While he does note that such streaming is patently illegal, Bowman also said that MLB didn’t want to alienate fans over something that wasn’t yet causing any big problems. Currently, he said, most fans are using social media to share photos of themselves at baseball games, a type of free promotion MLB and teams go out of their way to encourage.

What hasn’t happened — yet — is large numbers of fans using the livestreaming services, something that could potentially clog up the cellular and Wi-Fi networks inside the stadiums since live, streaming video inherently uses up a large amount of bandwidth. Bowman, whose MLBAM operation spent some $300 million over the past couple years in a project that is bringing advanced cellular and Wi-Fi networks to all MLB parks, said that if livestreaming becomes a bandwidth issue, it will be addressed.

“We just put all these new networks in, and the last thing the stadiums want is [people] using the network for these types of activities,” Bowman said. “If we’re wrong, we’ll review it. But I just don’t think our fans are there to stream the game.”

AT&T sees almost double DAS traffic for MLB’s season-opening series

Head-end room cabling at AT&T Park. Credit: AT&T/San Francisco Giants

Head-end room cabling at AT&T Park. Credit: AT&T/San Francisco Giants

According to AT&T, the season-opening series for Major League Baseball saw fans use almost twice as much cellular data as the year before, across the 19 ballparks where AT&T has in-stadium cellular networks in place.

Remember, these numbers represent only cellular traffic and only for AT&T customers on the AT&T stadium-specific networks, which are almost all of the distrubuted antenna system (DAS) type. Though some stadiums saw much more traffic than others, the average series-long total of 215 gigabytes per venue was almost double the same statistic from the 2014 season-opening series, where AT&T saw an average of 111 GB per venue. And if AT&T’s traffic is doubling you can probably safely bet that all other metrics — Wi-Fi, and traffic for other carriers — has increased as well.

Thanks to our friends at AT&T, here is the full list for series-long DAS traffic at MLB venues where AT&T has stadium-specific networks. Stay tuned to MSR for our Q2 Stadium Tech Report later this spring, when we’ll take a team-by-team look at MLB technology deployments, specifically focusing on Wi-Fi and DAS. So far, it looks like fans are already in mid-season selfie form.

OPENING SERIES DAS TOTALS (AT&T customer traffic only, on AT&T stadium-specific networks)

1. Arlington, TX (Rangers): 655GB

2. St. Louis (Cardinals): 466GB

3. Los Angeles (Dodgers): 396GB

4. Atlanta (Braves): 375GB

5. Anaheim (Angels): 270GB

6. Denver (Rockies): 251GB

7. Philadelphia (Phillies): 250GB

8. Chicago (Cubs): 227GB

9. New York (Yankees): 189GB

10. Cincinnati (Reds): 185GB

11. Miami (Marlins): 183GB

12. Boston (Red Sox): 162GB

13. San Francisco (Giants): 158GB

14. Oakland (A’s): 149GB

15. Seattle (Mariners): 139GB

16. Washington, D.C. (Nationals): 132GB

17. Milwaukee (Brewers): 129GB

18. Houston (Astros): 102GB

19. Minnesota (Twins): 86GB

20. Phoenix (Diamondbacks): 85GB

21. New York (Mets): 80GB

22. San Diego (Padres): 58GB

AT&T: DAS network also set traffic records for 2014 World Series games at AT&T Park

Giants CIO Bill Schlough (left) talks with workers in the park's main DAS head end facility.

Giants CIO Bill Schlough (left) talks with workers in the park’s main DAS head end facility.

In addition to the over-the-top Wi-Fi usage numbers, it turns out that the in-stadium AT&T cellular network also experienced record usage during the recent World Series games at AT&T Park, with Saturday night’s crowd using 477 Gigabytes of data, according to AT&T.

In case you’re not familiar with the DAS acronym, it stands for distributed antenna system, and basically is a network of small antennas that bring cellular service to the tightly packed fans inside stadiums or other large public venues. At AT&T Park, AT&T runs the network as a neutral host, meaning that Verizon Wireless, Sprint and T-Mobile customers can also use the DAS to connect. If you’re in the park and you haven’t enabled your device to connect via Wi-Fi, you’re probably connected via the DAS. However, the stats provided here are only for AT&T customers on the DAS, since AT&T doesn’t have visibility into the other carriers’ metrics. Since the Wi-Fi network is open to all, the Wi-Fi numbers we reported earlier are for all customers, no matter who their provider is.

But even just the AT&T cellular numbers are pretty impressive, perhaps not surprisingly so since a good-weather World Series game is a bucket-list event for most attendees, meaning that texts, tweets, selfies and Vines were likely flowing freely at all three games in San Francisco. Here’s a breakdown from AT&T about how much more data was used during the games last weekend:

— Fans used an average of approximately 447GB of data per game over the weekend on the AT&T cellular network. This is equivalent to more than 1.27M social media post with photos.

— The numbers represent an increase of approximately 29 percent in cellular data usage compared to the average game during the League Championship series vs. St. Louis.

— It’s an increase of approximately 109 percent in cellular data usage compared to the average game during the final home series of the regular season vs. San Diego (9/25-9/28).

— The peak hour of data usage during three home games was on 10/25 was from 5-6pm PT, the hour in which the first pitch occurred. In this hour more than 83GB of data crossed the AT&T venue-specific cellular network.

According to AT&T, the total combined Wi-Fi and AT&T DAS traffic hit a record high of 2.09 Terabytes on Oct. 25, the highest single-game total in AT&T Park network history. What would be even more interesting would be if we could get DAS statistics from Verizon, Sprint and T-Mobile to get the total-total number for wireless data consumed during the biggest games of the year.

Half a stadium using Wi-Fi? 2014 World Series games set Wi-Fi records at AT&T Park

Though this year’s World Series still isn’t over, the three games this past weekend in San Francisco set new local records for fan Wi-Fi use, arguably cementing AT&T Park’s title as baseball’s best-connected stadium.

One of the more stunning numbers provided to us by Bill Schlough, the senior vice president and CIO for the Giants, was that nearly half the crowd in AT&T Park for Sunday’s Game 5 used the Wi-Fi network — a “take rate” we’ve never heard of before at any sporting event of any kind. According to Schlough the Giants’ Wi-Fi network had 20,638 unique users during Game 5, a take rate of 49.7 percent. (According to the official box score, the attendance that night was 43,087, give or take a Panda or four.) The game also set an AT&T Park record for Wi-Fi upload traffic with 747 Gigabytes of data being sent by people at the park out to the network.

(Including one special photo tweeted out by a former Giant you may have heard of.)

A record for downloaded Wi-Fi traffic, of 937 GB, was set during Friday night’s Game 3, and the record for a combination of download and upload traffic was set during Game 4, with a total of 1.62 Terabytes. According to Schlough, the total data transferred during the eight games of the 2014 postseason at AT&T Park was 10.3 TB, which is easily the high-water mark for any baseball stadium network we know of. (The total does not include regular cellular traffic carried by the AT&T Park DAS; we will add those numbers in when they become available.)

And while individual games such as Super Bowls or contests down the San Francisco Peninsula at Levi’s Stadium may set higher overall marks (thanks in part to having more fans in the building), we’d be interested to see if any other stadium networks can match AT&T Park’s season-long data number of 59.3 TB, which according to Schlough represents total traffic over 2014’s 91 games (regular and postseason) at AT&T Park.

One more fun number from Schlough: Since the park turned on its fan-facing Wi-Fi network in 2004, there have been 4,085,518 Wi-Fi connections during the 946 game days. Kansas City’s Kaufmann Stadium, which just got its Wi-Fi network installed recently, has one or two more games this season to start trying to catch the Giants from a networking perspective.

UPDATE: Looks like the Royals are claiming some already-decent usage of their Wi-Fi network:

Confirmed: KC has MLB-provided Wi-Fi, part of plan to bring Wi-Fi and DAS to all MLB stadiums

KC fans at seriesThe rumored Wi-Fi network at Kansas City’s Kauffman Stadium is now a confirmed entity, according to Joe Inzerillo, executive vice president and CTO for MLB.com. In a phone call Thursday, Inzerillo said the Royals’ new network is part of a league-wide effort to bring Wi-Fi to all MLB parks, a task he thinks may take another year or two to complete.

As MLB.com has stated publicly before — but maybe not spelled out in so much detail — it has a program under which Major League Baseball teams can “opt in” to have MLB.com and the nation’s top four wireless carriers participate in the funding and building of both Wi-Fi and DAS networks in MLB stadiums. Though he wouldn’t divulge the specific financial commitments for specific deals, Inzerillo said that under the program “everyone has some skin in the game,” though he did allow that the league and the carriers, not the teams, foot the bulk of the bills.

Still, Inzerillo stressed that individual teams play a huge role in the Wi-Fi deployments, from design to deployment to management on site. “It’s not just like we show up and we’re the Wi-Fi fairies,” Inzerillo said. “This program wouldn’t be possible without the teams and the work they do.”

Screen Shot 2014-10-23 at 1.27.54 PMGiven MLB’s synchronized digital strategy of having the same app — and only the same app — available for fans in each ballpark, Inzerillo said that having high-quality cellular and Wi-Fi networks in each stadium was a key necessity, especially to make MLB app functions like seat upgrades and concession purchases work.

“You need to have the right [network] plumbing in place or none of the other stuff matters,” Inzerillo said.

Kauffman Stadium’s network, Inzerillo said, was just one of about a dozen MLB Wi-Fi projects that got underway this year. That it was finished in time for postseason play was just luck, and not some last-minute installation due to the Royals’ on-field successes. “It was just a fortuitous thing that it was ready,” said Inzerillo, who said that construction of Wi-Fi at Kauffman had been ongoing for the past 5 to 6 months.

Though the network wasn’t promoted on the team’s website or anywhere else on the Internet, Inzerillo said the Royals were promoting it at the stadium. Even without a lot of advertising, fans found the network, he said, claiming “tens of thousands” of Wi-Fi connections during the Royals’ postseason run. However, Inzerillo also said some extra cellular trucks were brought in by some of the carriers for the Royals playoff games because the DAS at Kauffman isn’t quite finished yet.

Inzerillo said that anywhere from 22 to 26 teams will eventually end up using some combination of league-provided Wi-Fi and/or DAS. The league’s goal of having every stadium fully wired should be nearly complete by opening day of 2015, he said, with a more likely “final” goal of complete coverage reached sometime in 2016. In 2014, Mobile Sports Report research showed that 10 of the 30 MLB stadiums didn’t have fan-facing Wi-Fi; some of those teams (like Kansas City) will be getting MLB networks, while some other program participants are upgrading existing systems, Inzerillo said.

Under the MLB network deal, the DAS in each stadium will be a neutral-host deployment hosted by one of the four major U.S. cellular carriers — AT&T, Verizon Wireless, Sprint and T-Mobile — depending on things like regional/historical market share and existing contracts, Inzerillo said. While the carriers will operate the MLB DAS deployments, the Wi-Fi networks will be deployed, run and monitored by MLB, either from its New York City or San Francisco network operation centers, Inzerillo said. Wi-Fi gear will come mainly from Cisco, though Inzerillo said there is also a small percentage of Meru Networks gear based on teams that had previously installed Meru equipment.

Though Inzerillo said MLB may make some overall announcement once the league-wide project is completed, he didn’t necessarily think that having working Wi-Fi and DAS in stadiums was such a big deal.

“It’s kind of a weird thing to think about bragging about,” said Inzerillo, who compared Wi-Fi and DAS to plumbing as a basic stadium necessity, not an amenity. Having high quality networks, he said, “are table stakes for a modern facility.”

UPDATE: The Kansas City Royals are now officially promoting the Wi-Fi, with some usage claims:

https://duwit.ukdw.ac.id/document/pengadaan/slot777/

https://mtsnupakis.sch.id/wp-content/zeusslot/

https://insankamilsidoarjo.sch.id/wp-content/slot-zeus/

https://smpbhayangkari1sby.sch.id/wp-content/slot-zeus/

https://alhikamsurabaya.sch.id/wp-content/slot-thailand/

https://mtsnupakis.sch.id/wp-content/bonus-new-member/

https://smptagsby.sch.id/wp-content/slot-bet-200/

https://lookahindonesia.com/wp-content/bonus-new-member/

https://ponpesalkhairattanjungselor.sch.id/wp-content/mahjong-slot/

https://mtsnupakis.sch.id/wp-content/slot777/

https://sdlabum.sch.id/wp-content/slot777/

https://sdlabumblitar.sch.id/wp-content/bonus-new-member/

https://sdlabumblitar.sch.id/wp-content/spaceman/

https://paudlabumblitar.sch.id/wp-content/spaceman/