Stadium Tech Report: Boingo, AT&T answer call for more DAS bandwidth at Chicago’s Soldier Field

Soldier FieldWhether or not the Chicago Bears will beat the Green Bay Packers this Sunday is a question still answerable only by higher deities. But if those powers that be decide to unleash weather conditions of biblical proportions again, at the very least the Soldier Field folks can be confident that their in-stadium network will stand up to the challenge.

Earlier this year, tornado weather hit Soldier Field and caused an unusual 2-hour delay during the Nov. 17 Bears-Ravens game. While the deluge was too dangerous for football, for Doug Lodder and Luca Serra, it was a big test to see if the stadium wireless network upgrades they’d recently helped install could stand up to a huge, unexpected traffic surge.

“We thought we’d probably see some good [network] usage with the weather delay and tornados,” said Lodder, vice president of business development at Boingo Wireless, which recently concluded a big network upgrade at the home of the NFL’s Chicago Bears.

“We saw double the usual traffic from both AT&T and Verizon customers during the 2-hour delay,” said Serra, director of sponsorship and media for SMG, the outside entity that manages Soldier Field for its owners, the Chicago Park District. “During that break, we didn’t hear a single complaint [about the wireless network] from the media or the fans. It was a great test — not a desired one in any way, shape or form, but a great test.”

Upgrades needed at ‘oldest’ big stadium Wi-Fi network

Luca Serra, SMG

Luca Serra, SMG

Even though you can’t really plan or design for surges that might happen during extreme situations, Serra and Lodder and others running the Soldier Field network knew last year that an overall upgrade was needed. Home of one of the earliest (if not the first) big-stadium DAS and Wi-Fi networks, Soldier Field was showing 80 percent year-over-year traffic growth on the Wi-Fi network, Lodder said in a recent joint phone interview with Serra.

Along with DAS partner AT&T, SMG and Boingo spent last year adding more DAS capacity so that Solider Field now has 250 DAS antennas to go along an undisclosed number of new Wi-Fi APs. Though AT&T did some specific upgrades to its DAS presence at Soldier Field (including adding support for its new 4G LTE network), all the major wireless carriers are represented in the DAS via the neutral hosting of Boingo.

Why did the firms decide Soldier Field needed an upgrade? While Lodder said its DAS, which first started operating in 2011, was “pretty cutting edge at the time,” during the past couple years, data usage at Soldier Field has skyrocketed.

“Everyone has a device, and there’s a lot more social engagement and sharing,” Lodder said. “We’ve seen a huge increase in data use.”

“With this large increase in use, we said ‘let’s address the issue’ and not let it get to a point where it’s a problem,” Serra said.

Moving fast and finding space

DAS gear at Soldier Field. Credit: Boingo Wireless

DAS gear at Soldier Field. Credit: SMG/Soldier Field

Both Serra’s and Lodder’s teams were familiar with the need to balance performance and aesthetics at the venerable stadium perched on the edge of Lake Michigan. Since the network was put in after Soldier Field’s extensive renovation was finished in 2003, special attention needed to be paid to placements of Wi-Fi and DAS antennas.

“It wasn’t just bolting APs on,” Serra said. “We had to be very protective of the aesthetics of the building.”

For Lodder, that sometimes means that the best placement isn’t always available.

“There’s always challenges with architecture at stadiums,” Lodder said. “In a perfect world, there might be a certain place you want to put an antenna. In the real world, you can almost always count on changing that 20 percent of the time.”

Doug Lodder, Boingo

Doug Lodder, Boingo

A bigger construction concern of late is where to put the extensive amount of back-end telco gear needed to support the DAS infrastructure and upgrades. “Carrier base stations need several thousand square feet of space, and those need to be rooms with connectivity and power,” Lodder said. At Solider Field, Serra said some unfinished space north of the stadium proper was converted into a DAS equipment room, since there wasn’t enough available space in the main building. Lodder said for the recent upgrade, additional power upgrades were installed as well.

Still, even with all the challenges the Boingo, AT&T and SMG teams were able to go from design to finished construction of the recent upgrade in 45 days this past summer, in part thanks to some interesting work schedules.

“After one preseason [football] game we had scaffolding going up at night, with three 24-hour shifts,” Lodder said. “It was a pretty accelerated build.”

There is no such thing as free Wi-Fi

One other interesting twist to the Boingo Wi-Fi network at Soldier Field is its cost — the company charges a $1.99 per day fee for Wi-Fi use, or you can purchase a regular Boingo monthly plan for $7.95 which will also provide access at Soldier Field.

Wi-Fi antennas hidden under stands at Soldier Field. Credit: SMG/Soldier Field

Wi-Fi antennas hidden under stands at Soldier Field. Credit: SMG/Soldier Field

While many if not most professional teams are making Wi-Fi services free to paying ticketholders, Serra and Lodder said that the Bears aren’t.

“The Bears are a tenant of the Park District, and no one [from the team] has ever said ‘we’re going to pay to turn it [Wi-Fi] on for free,’ ” Serra said. Lodder noted that in the end, someone somehow has to pay the cost of connectivity.

“Someone’s always got to pay for it — Wi-Fi isn’t free,” Lodder said. “Some teams may feel they’ve made enough on the tickets to offer [Wi-Fi] free, while in other cases users pay either with money or with eyeballs, by viewing some ads.”

While the NFL itself is still somewhat vague on its league-wide plans for in-stadium Wi-Fi — other than saying that teams should provide it — Serra and Lodder are confident that Soldier Field will be at the top of the chart when it comes to network performance.

“If and when they [the NFL] roll out standards for network performance, we’re going to exceed them,” Lodder said.

Friday Grab Bag: ESPN adds more SEC programming for 2014

Not enough SEC Football? ESPN has you covered
ESPN’s top markets for its college football broadcasts are all in SEC football’s heartland and apparently the powers that be at the sports broadcasting giant believe that you can never have too much of a good thing. (Apparently they have never sat by an unwatched plate of fudge.)

So with the kickoff of next season’s NCAA football viewers can expect “SEC Nation” according to SI, a two-hour program that will run from 10 to noon EST as the main component in its SEC Network. It will cover other sports aside from football, which is news to the rest of us that SEC schools

Facebook buys SportsStream
A month ago we mentioned that Facebook was looking to increase its profile in the sports markets in part with a partnership with sports analytics company SportsStream. Well according to TechCrunch it likes SportsStream so much it purchased it.

The move I designed to help it fend off rival social media companies such as Twitter that are increasingly becoming to the go to sites for up to the second news and gossip. Terms of the deal were not revealed.

Are those Twitter numbers real?
Good news! If you have 61 followers on Twitter you are a median poster and if your number has reached 1,000 active users then you, my friend, are in the 96th percentile. Since it seems like there are millions on Twitter what exactly does this mean?

Well according to Oreilly.com it means that the huge percentage of activity is much smaller than it appears. One interesting outtake from the article is that fake posters and infrequent ones often have huge followings.

Will Android fragmentation continue?
Ever wonder how many types of Android operating systems and devices are out there? How about Apple iOS? Well A recent Forbes article points out that there are currently 4,700 different types of devices using the two operating systems.

Now you can probably list the Apple devices off the top of your head, but the dizzying array of Android has caused concern for developers for some time, and the article points out that as long as it stays heavily fragmented it will benefit Apple.

Worst Announcers in the NFL
Some poor soul at SportsonEarth charted announcers for a series of NFL games in order to find out which ones used the most, and the fewest, clichés, leaps of faiths and illogical statements during a broadcast.

I am assuming that author is not now committed and I think most will agree with the general results, although people always have their favorites to dislike-I know that I certainly do.

5 Bars Inside now inside Anaheim Stadium
The Los Angeles Angels of Anaheim have announced that 5 Bars Inside is the official distributed Antennae system (DAS) and Wi-Fi provider for the team and that the stadium will get a makeover so that fans have solid connectivity.

Stadium Tech Report: Wi-Fi ‘coaches’ help fans find network in New England and Philly

Wi-Fi coach in the stands at Gillette Stadium. Credit: Extreme Networks

Wi-Fi coach in the stands at Gillette Stadium. Credit: Extreme Networks

What good is a stadium Wi-Fi network if the fans don’t use it? To help avoid the obvious answer, the providers of wireless services in the New England Patriots’ Gillette Stadium and the Philadelphia Eagles’ Lincoln Financial Field are now supplying “Wi-Fi coaches,” roaming groups of helpers who help fans get connected to the in-stadium network. With assistance from the digital sidelines, fans at those two stadiums can now conquer what may be the biggest hurdle to in-stadium connectivity: Just figuring out how to make your device work.

In a phone interview with John Brams, director of hospitality and venues with Enterasys (the stadium networking company recently acquired by Extreme Networks), we asked about what problems the providers had experienced with their new networks at Gillette and the Linc. Perhaps not surprisingly, one of the biggest concerns wasn’t technical, but simply user confusion.

“After we installed the networks, we quietly realized that the [biggest] fundamental issues weren’t with the design — it was user issues,” Brams said. When you consider that the average age of a Patriots fan at the game is 51 or 52, Brams said, it’s understandable that many of them might not know how to do things like download an app or find a Wi-Fi connection for their phone. Communicating with fans at the games is also problematic, he said.

“You can’t just pick up the phone and call people, and you can’t really have a help desk at the stadium,” Brams said. What the Enterasys/Extreme folks hit on was a plan familiar to any NFL fan, who knows all about the long list of assistants for things like offensive line play and special teams: Give the fans their own Wi-Fi coaches.

Coaches roam stands to help fans connect

Wi-Fi coach in the stands at the Linc. Credit: Extreme Networks

Wi-Fi coach in the stands at the Linc. Credit: Extreme Networks

Now at both Gillette and the Linc the Extreme folks have 16 or more Wi-Fi coaches roaming the stands and concourses during games, readily identifiable in team-color jackets that announce them as friendly types to help fans connect to Wi-Fi. According to Brams, it’s not just the fans who benefit from the outreach program — the teams and stadiums also get instant feedback from users, which can help them quickly pinpoint any network problems.

“From a team perspective, the coaches are a great way to interact with the fans,” Brams said. “You get immediate feedback and if there are any problems, you get the information from the fan and quickly close the loop.”

The personal show-me coaching works very well in a stadium situation, Brams said, because fans will quickly pass on successful tactics to those sitting nearby. “If one person learns how to do something, it spreads like a chain reaction,” Brams said. One problem the Extreme coaching team has been communicating to users was a known bug with the team app and Android platforms, which could be fixed with a simple download. The bottom line, Brams said, is to ensure a good network experience the first time out — otherwise, fans might never try to connect again.

“I ran into one fan at Gillette who said the network didn’t work — but the last time he tried it was 2 years ago,” Brams said. “You need to deliver the first time they use it.”

Steady increase in Wi-Fi usage

At both Gillette, where the network was installed last season, and at Lincoln Financial, which is in its first year of Wi-Fi, Extreme is seeing a steady increase in usage, Brams said. An interesting side note is that while numbers of users are rising steadily, the amount of data being consumed overall is growing more rapidly, suggesting that the fans who do connect are starting to do more.

On the coaching front, Extreme is looking into helping its team customers find ways to expand the program, perhaps with some device-charging stations staffed by more Wi-Fi coaches. The idea is also open to possibilities for branding and advertising on the coaches’ jackets as well as at the charging stations, Brams said, with the final options limited only by the stadium owner and operators’ imaginations.

The best part of the program is when a coach helps a fan connect to the Wi-Fi, which is often a game-changing experience in the world of stadium connectivity. “These places [stadiums] used to be such a connectivity black hole,” Brams said. “Now people are excited when they connect.”

Stadium Tech Report: Partners the key to fast Wi-Fi deployment at Staples Center

Staples centerDuring the 2012 NHL playoffs, no other team could stop the Los Angeles Kings. But as the Kings’ fans were celebrating their team’s run to the Stanley Cup, their cheers from Staples Center stayed inside the walls. Without a Wi-Fi network or good cellular connectivity in the venue, fans there were shut off from sharing their joy with the outside world — a frustration they let team officials know about, in no uncertain terms.

“Not having Wi-Fi was really impacting the fan experience, especially during that championship run,” said Denise Taylor, chief information officer for arena owner AEG. “Our fans went through a lot of pain. And we heard them.”

Fast-forward to the 2013 playoffs, and while the LA Kings weren’t as successful on the ice, at least the fans were able to stay fully connected while at the games thanks to a new HD Wi-Fi network and a neutral DAS deployment at Staples Center. With some impressive help from infrastructure partners Cisco and Verizon, AEG was able to get Wi-Fi and a full DAS deployment installed in both Staples and the adjoining LA Live center in just two and a half months, despite nearly constant activity in the venue that never sleeps.

With its network now in place, AEG and Staples are keeping all the Los Angeles teams’ fans connected while busily looking for more opportunities to not only further enrich the fan experience, but to also add to the teams’ and facilities’ bottom lines. And AEG is already learning lessons about how to better deploy Wi-Fi at its arenas and venues worldwide, including the knowledge that fan network use is showing no signs of slowing down.

Building a network in a building that never sleeps

According to Taylor, deciding that Staples Center needed a Wi-Fi network was the easy part “It was a must-have,” she said. “We knew had to get it in immediately.”

The LA Kings celebrating the 2012 Stanley Cup win at Staples Center

The LA Kings celebrating the 2012 Stanley Cup win at Staples Center

The hard part? Building that network into a building that hosts not one but two NBA teams, the Los Angeles Lakers and the Los Angeles Clippers, as well as the NHL’s Los Angeles Kings and the WNBA’s Los Angeles Sparks, plus is host to numerous concerts special events and the Grammy Awards. When the business plan was put in place in late 2012 to deploy the networks, Taylor said there were only four “dark” nights during the two and a half months of construction.

“That’s when it pays to have really great partners,” she said. Though there was heavy lifting necessary to retrofit the 20,000-seat venue, including opening up walls to install conduit and access points, Taylor said AEG’s partners worked around the busy Staples event schedule, with most work taking place between the hours of 2 a.m. and 2 p.m. When the dust settled, Cisco had installed a total of 463 APs in both Staples Center and L.A. Live, meaning that fans could stay connected inside the stadium and outside at the adjoining cluster of stores, restaurants, movie theaters and other facilities.

Almost instantly, fans were talking about events, sharing pictures and messages on social media networks. “It was a great feeling to see people talking about the games,” Taylor said. “Our partners really delivered.”

Future Advice: Don’t make any small network plans

LA Kings fans, who previously couldn’t even use the team’s app while at Staples, are now the biggest network users. According to Taylor, an average of 3,000 fans are connected to Wi-Fi during LA Kings games at Staples Center, just ahead of the facility’s average NBA game network user number of 2,410. Even at this early stage of the game, Taylor is seeing that her team’s predictions for network usage might have been low.

“We had originally designed [the network] for use by 25 to 30 percent of a capacity crowd,” Taylor said. But during a recent concert run of four sold-out shows by the band One Direction, Taylor said 35 percent of attendees were using the network during those shows.

“Use of mobile phones in venues is just going to grow and grow,” she predicted. “We’re seeing more and more penetration every day.”

And while pure connectivity is great, Taylor and AEG are now moving forward to enhance the fan experience in multiple ways, including implementation of mobile ticketing apps and apps for ordering food and concessions from your phone.

“We’re in the early stages, but all of that — concessions, ticketing, team apps — is in the works,” Taylor said. “Having a real HD Wi-Fi network creates so many business opportunities, for teams, venues and owner groups. There’s the whole area of captive analytics, which has tremendous potential upside. It’s just a question of how you monetize this big digital asset you have in your hands.”

More lessons: Don’t skimp on 2.4 GHz, and pick good partners

When it comes to Wi-Fi technology, the future is devices that work in the 5 GHz unlicensed spectrum band, which has tons of available bandwidth. Many newer devices, like Apple’s iPhone 5 series, include radios for connecting to 5 GHz Wi-Fi. But Taylor said venue owners and operators need to make sure they don’t skimp on building in support for the 2.4 GHz band, where a large legacy of devices still operate.

“Designers may prefer the 5 GHz range but if you only support that band the truth is you would eliminate a good portion of consumers,” Taylor said. “There are also back of house management applications that still require 2.4 GHz support. So you still have to build for the lowest common denominator of devices to make sure you are engaging every fan who comes in, even those with older phones.”

And since most networks built now are going to be put into arenas that already exist — meaning retrofits — there will probably be a Staples Center-like challenge of doing deployment work around events. That means, Taylor said, that owners and operators need to find partners who are “willing to work hours that aren’t always 8 to 5.”

As AEG develops overall plans for site operations at its numerous international locations, Taylor said that wireless networks are no longer seen as a luxury.

“The two most important things are having HD Wi-Fi and a robust DAS,” she said. “You give the fan what they want, and take advantage of all the opportunities to monetize the network. It’s just not an option anymore.”

Will sports help Apple win the indoor location market?

ibeacon

A few months ago Apple, Major League Baseball and the New York Mets showed off iBeacon, a technology that is embedded in Apple’s iOS 7 operating system and how it can be used for indoor location services.

If you are not too familiar with iBeacon that is not surprising, Apple really has not publicized the technology that much since it was rolled out as part of the iOS7 release last September. It is an indoor positioning system that is designed to enable a facility to push notifications, coupons and other material to enabled iOS mobile devices. It is built around a low powered version of Bluetooth technology and has the advantage of being very precise and essentially serves as an indoor GPS, but with a much greater degree of accuracy.

The advantages for a sporting facility are obvious. It can track where a user is and send them discount coupons when they are in front of a souvenir shop or a two for one hot dog offer when they are at the food stands. Facilities can see where fans visit and where they do not and customize both their offers to the fans, and the layout of the retail outlets to better meet fans usage models.

It is no surprise that MLB’s Advanced Media group, which has been very aggressive in delivering apps that both engage fans when they are in attendance and when they are not, would be interested in this technology. It has the potential to help increase sales while also enabling fans to take an unescorted but informative tour of ballparks such as Fenway and hear all of the history of the park.

After that announcement it seemed the technology fell off the radar but last week Macy’s said that it will use the technology to send alerts to shoppers when they enter stress over the holiday season at select stores via an app called Shopkick.

Apple delivered its own version of maps a while ago and emerged with egg on its face as the maps were in some cases very inaccurate and rival Google and others made fun of Apple’s efforts. A shakeup later Apple seems to have the map app working well and according to this piece from Mobile Marketer Apple has taken 23 million users from Google in the maps space, but still trails Google’s impressive lead in that space.

However indoor is a different area and Apple could be heading to a lead there, in part because of the effort by sports leagues to add enhanced networking capabilities to their facilities. In addition they have a great deal of familiarity developing for Apple’s platform as apps for Apple’s iOS are often the first to appear for sports leagues and fans can already use them to locate hot dog stands and swap seats, among other uses. An app that does significantly more would simply fit in with the fans already established mindset of using a mobile device to assist them in a facility.

Fans can get upset with the lack of access at a stadium when using a mobile device and all major US sports leagues are expanding and enhancing their Wi-Fi networks. Since teams want a return on investment aside from fan satisfaction this presents them with a solid opportunity.

I suspect that come next year we will be seeing an influx of apps not just from Apple developers but also Android and Windows 8 seeking to take advantage of the new networking and connectivity capabilities of stadiums and it will be interesting to track which ones are using indoor positioning as a feature since it certainly appears to present a solid advantage to its users.

Stadium Tech Report: Cal Bears, AT&T get creative with DAS deployment for Memorial Stadium

Cal's Memorial Stadium. Note the lack of overhanging structure.

Cal’s Memorial Stadium. Note the lack of overhanging structure.

What do you do when the construction of your big bowl stadium won’t allow you to install Wi-Fi? You start building the biggest DAS you can, to bring as much cellular capacity as possible to the fans who will fill the seats.

That problem and solution is now underway in Berkeley, at the University of California’s newly renovated Memorial Stadium. Though it now has a shiny new press box and suites on its west side, the 63,000-seat facility still has its old “big bowl” shape throughout the rest of the facility, an architectural quirk that leaves no place to put Wi-Fi antennas. To try and best overcome the design challenges, Cal and its partner AT&T are currently putting in place as big a DAS (distributed antenna system) as they possibly can, to bring as much cellular capacity as the stadium placements will allow.

At a press event Thursday to officially announce AT&T’s campus-wide Wi-Fi and DAS deployments — which are currently bringing welcome speedy connectivity to all the other parts of the school’s Berkeley footprint — Cal officials and AT&T execs noted that bringing better wireless to Memorial Stadium is also going to be a necessary evil, going forward.

Solly Fulp, COO of Cal Athletics

Solly Fulp, COO of Cal Athletics

“We compete with the couch, and the HD TV,” said Solly Fulp, the Deputy Director of Athletics and Chief Operating Officer at Cal, at a quick press conference kicking off the Thursday events. “If we do not deliver an outstanding fan experience, people will do other things with their time.”

Right now, the wireless experience inside Memorial isn’t cutting edge. Basically a big concrete bowl filled with seats, Memorial has none of the overhanging levels that allow you to more easily attach antennas underneath, be they Wi-Fi access points or the small antennas used in DAS deployments. While the new press box and suites and internal construction improvements completed last year have brought the facility up to date on one level, for supplemental wireless coverage right now AT&T is using COWs (cell towers on wheels) in several spots around the stadium, covered with stealthy black cloth. But construction for a big, new DAS deployment is already underway.

North end of Memorial Stadium. The lower poles seen here will house DAS antennas

North end of Memorial Stadium. The lower poles seen here will house DAS antennas

According to AT&T, the low poles sticking up from the top of the bowl, currently used to house speakers, will by next season be equipped with DAS antennas focused down into the sections of seats. There will also be DAS antennas housed inside boxes mounted above tunnel entrances, and along the underside of the press/suite facility wherever possible. While Wi-Fi can typically support faster communications and more bandwidth, the DAS AT&T is putting in will support AT&T’s 4G LTE network, which is as fast as cellular gets. As in most stadiums we’ve talked to, putting in a DAS is priority one for wireless, since it helps eliminate the “no signal” problem common to big facilities that don’t have any extra cellular or Wi-Fi services.

AT&T and Cal are walling off this section of concourse to house future DAS equipment

AT&T and Cal are walling off this section of concourse to house future DAS equipment

AT&T, which will act as a neutral host for the Memorial DAS (meaning that other service providers can connect their back-end systems to the AT&T-installed antennas) is even getting creative finding space for its head-end gear, the routers and switches needed on site to help complete the cellular connections. In one concourse underneath the stands, AT&T and Cal are walling off a small part of the internal walkways to house the necessary DAS gear.

One of the COWs (cell on wheels) installed outside Memorial's front gates

One of the COWs (cell on wheels) installed outside Memorial’s front gates

From there, optical fiber will connect the stadium to a nearby AT&T central office, where more call processing can take place.

Will the big DAS be enough to keep Cal fans connected? From what we heard at Thursday’s press event both the school and the carrier think that the wireless networks will always be a work in progress. And according to Cal’s Fulp, better cellular is already one of the top three concerns voiced by season ticket holders.

“It’s the simple things that matter,” Fulp said, saying that for the most part, fans at games want to be able to contact people at home — “sending a text to the babysitter, saying you’ll be late” — or to share a picture or a video with friends who couldn’t be at the game. “People expect to be connected, and at sports events it’s amplified,” Fulp said.

When the members of Cal’s class of 2014 toss their graduation caps into the air next spring, Fulp wants to see a “best in class” DAS in action at Memorial. What will that take from partner AT&T? “Every place we can find to put an antenna, we will,” said one AT&Ter who showed us around Memorial. Sounds creative to us.

(all photos credit: Paul Kapustka, Mobile Sports Report; user hint: click on photos to see more detail!)

An artsy shot of Cal's famed Sather Tower, aka the Campanile

An artsy shot of Cal’s famed Sather Tower, aka the Campanile

Another COW deployment, the fuzzy black things behind the stadium wall

Another COW deployment, the fuzzy black things behind the stadium wall

Find the antenna!

Find the antenna!