Verizon Wireless

Final Four final score: 11 Terabytes of total wireless traffic

April 27, 2015 by Paul Kapustka

We finally have some final wireless results in from the Final Four, and the total wireless-traffic number of 11 terabytes used over the weekend is just another sign that mobile device use at “big” sporting events is still increasing, with no top in sight.

According to figures sent to us by Ryan Marketing Group Technologies — an official outlet for the NCAA’s men’s basketball tournament tech operations — there was 9.47 terabytes of wireless data used over the Final Four weekend in and around Lucas Oil Stadium in Indianapolis, with 5.3 TB of that running over the stadium’s Wi-Fi network and the balance running on the building’s DAS. According to Ryan Marketing Group they believe the DAS numbers only reflect Verizon Wireless customer traffic (we have asked Verizon to confirm the numbers, but have not yet received a response from Verizon). The NCAA numbers also include usage for attendees at the accompanying Fan Fest events, in addition to the Saturday and Monday night games.

Separately, AT&T had said that it saw 1.52 TB of cellular traffic on its own DAS for the Final Four games, so adding the figures together we get 10.99 TB of total data, a number that should give pause to stadiums or arenas with “big events” on the horizon. Following huge wireless traffic numbers from recent big events like the Super Bowl and the College Football Playoff championships, it’s clear that there is still no plateau in sight for mobile-device wireless usage at big events. And just when you thought networks might be able to keep pace with users, devices and apps, you have the recent emergence of livestreaming apps like Meerkat and Periscope to potentially swamp arena networks with even more data demands. So, big event holders… what’s in your network? Is it ready for the terabyte age?

Filed Under: 4G LTE, AT&T, Basketball, DAS, Duke University, NCAA Hoops, Verizon Wireless, Wi-Fi Tagged With: 4G LTE, AT&T, DAS, Final Four, Lucas Oil Stadium, March Madness, NCAA, NCAA Basketball, NCAA Tournament, Stadium Wi-Fi, Verizon Wireless

Stadium Tech Report: Average connectivity doesn’t seem to hurt Avaya Stadium experience

March 23, 2015 by Paul Kapustka

Panoramic view of the packed house at Avaya Stadium for the official debut.

From a strictly wireless perspective, the opening-day performance of the Avaya Stadium Wi-Fi network was good in some spots and very poor in others, leading to an overall grade of average at best. But the Wi-Fi issues didn’t seem to take anything away from the smashing debut of a facility purpose-built for soccer and well-designed for an easy, fun fan experience, even with a sellout crowd of 18,000 on hand.

Mobile Sports Report visited Avaya Stadium for its “official” debut, Sunday’s San Jose Earthquakes’ MLS season home opener against the Chicago Fire, which ended in a 2-1 San Jose victory. But the team on the field wasn’t the only winner, as fans seemed to be smiling and enjoying every part of the new $100 million venue, from its huge end-zone bar and its close-to-the-field seats, to the pre-game picnic area with food trucks, music, and space for kids to run around. Well-planned parking and traffic operations seemed to cause few problems, with most fans finding their way to their seats in the new park in time for the just-after-4 p.m. kickoff.

If my unofficial walk-around testing was any true barometer, my guess is that the only problem some fans might have had Sunday was trying to connect to the Internet to post the thousands of selfies I saw being taken with smartphones. With almost zero cellular communication inside the stadium, and very low Wi-Fi readings in much of the seating bowl, my tests lead me to conclude that while the stadium is wonderful right now for watching futbol, its wireless connectivity is still a work in progress but one that should get better soon when the planned neutral-host DAS from Mobilitie gets installed and becomes operational.

Parking and traffic a breeze

Since I arrived early and had an employee-lot parking pass (thanks to the Earthquakes for the media pass and parking) I didn’t encounter any traffic at all either in my drive down 101 or on the streets leading to the stadium. Approaching from the north on 6-lane wide Coleman Avenue, there was very clear signage for each of the parking lots, and no backups in sight at 2 p.m., two hours before the scheduled start.

Fans waiting outside the main gate

Since it’s about one-fourth the size of its neighbor to the north, the San Francisco 49ers’ Levi’s Stadium, Avaya Stadium probably won’t have the same kinds of transit and parking issues that plagued Levi’s during its inaugural season. It also seems like the Avaya Stadium location is a much better setup for getting in and out of the stadium, with the wide Coleman Avenue and the huge dirt lots directly adjacent to the venue. Walking past some early bird tailgaters I was at the stadium gates in a couple minutes. In both the employee lot and the closest regular parking lot, I couldn’t get a Wi-Fi signal at all but cellular connectivity was pretty good (7+ Mbps on Verizon 4G LTE), as I could see several large cell towers around the edges of the lots. Even with a packed parking area, fans should still be able to get a signal on their way in.

For my early entry time I didn’t see any issues with stadium entry technology, but the lack of metal-detector gates (security personnel used handheld wands to scan each fan as they entered) might be something that slows down the process of getting into the stadium. I did notice larger lines around 3:30 p.m., but like anywhere else the entry procedures will likely only improve with time.

Before coming to Avaya Stadium I downloaded the new team app, which seemed a little bare-bones. Since I didn’t have a ticket I couldn’t test the digital season-ticket integration, but I was able to use the directions to the stadium feature and the stadium map, which provides a helpful picture-view of all amenities that can be found in the U-shaped seating area as well as the open-air bar. The map is interactive, giving you a description of each amenity (bathrooms, team store, etc.) when you touch the associated icon. As of yet there is no way to use the app to pay for concessions or to view any live or archived video. Like other stadium apps, including Levi’s, the Avaya Stadium app will likely grow in functionality over time.

Wi-Fi performance: Great on the concourses, weak in the seats

Just after finding my “exterior press box” seat in possibly the “worst” part of the stands — the upper northwest corner — I quickly saw how Avaya Stadium was going to deliver its Wi-Fi signals to the seating area, by looking up at the metal beams supporting the awnings that are the open-air “roof.” On each beam I could see anywhere from two to three Wi-Fi access points, all targeted directly down at the seats below them. The Avaya Wi-Fi deployment has no under-the-seat APs or any handrail APs that I could see, but there are lots of other APs visible on top of concession stands and other places around the single, ground-level concourse. There are also some APs attached to the huge bar area that spans across the open east end of the stadium. Gaining access to the network was a snap, done by just clicking on the “proceed” button that popped up on the splash screen that appears after you select the “GOQUAKES” SSID on your device. There was no login credential or password required.

The view from our seat, probably the “worst” in the place

How did the network perform? Before the stadium filled up, my rooftop seat had a signal between 5 and 7 Mbps on the download and upload sides, a figure that would decline steadily as the day progressed. Walking down the steep stairs into the largely empty seating bowl, the Wi-Fi speeds decreased, with a couple readings in the 2-3 Mbps download range near the lowest row of seats.

Hungry because I hadn’t had lunch, I ventured out past the huge end-zone bar to a large grassy area that was lined with food trucks and filled with soccer fans having impromptu picnics with lots of kids running around. There were various booths for soccer clubs and from sponsors, as well as a band, which made the area seem (in a good way) more like a county fair than a pro sporting event. I couldn’t get a Wi-Fi connection out on the lawn, but I was able to get a good cellular signal, around 8 Mbps, on my Verizon device (an iPhone 6 Plus). Feeling thirsty I headed to the bar, where Wi-Fi kicked in again, with one signal of 22 Mbps down and 17 Mbps up.

Starting to fill up here at @AvayaStadium and found better wifi: 20 mbps at outside bar, 15 mbps in lower concourse pic.twitter.com/BqvPjyjOvt

— paulkaps (@paulkaps) March 22, 2015

Heading back through the now-crowded concourse toward my seat, I stopped and got a Wi-Fi reading of almost 16 Mbps down and 9 Mbps up, in the middle of a large throng of fans. But I wouldn’t hit that mark again the rest of the afternoon, which makes me wonder how well the network held up under a full-house load.

Up close and personal areas a hit with fans

Panoramic view from the cheering section

Since I’d never been to a professional soccer game before I decided to soak in as much fan flavor as I could. At Avaya Stadium I headed down to the space behind the west end zone, in the closed end of the stadium, where there are several rows of standing-room only spaces where some of the loudest fans congregated (there was one group with a band, and many flags). Directly above the standing section was a seat section reserved for the team’s ardent followers, many of which spent the entire game standing, cheering, chanting and singing. Down below, I was fortunate enough to be close to the action and saw the Earthquakes’ first goal in their new home arena, a double header off a corner kick.

And though I was able to catch the score on video, because there was basically zero Wi-Fi signal there (I was directly underneath the bottom row of the stands) I wasn’t able to immediately post it to Twitter or Vine. Not that I cared that much, since it was fun to be swept up in the chanting and cheering and streamer-tossing that followed the goal. So even if I wasn’t connected wirelessly, I was certainly connected to the fans right around me — which, I think, is what Avaya Stadium is all about.

Quakes goal! pic.twitter.com/Jl2opi0dcS

— paulkaps (@paulkaps) March 22, 2015

I’m no wireless engineer, but I was hardly surprised that the Wi-Fi signal in the seats wasn’t strong; looking way up at the APs on the roof, they seem too far away to be able to provide a high level of connectivity to the seats below, especially the ones closest to field level. Other stadiums we’ve covered in the near past have already either started or are making plans to increase the Wi-Fi APs at field level, since that’s one of the toughest areas to put an AP.

But like in the standing section, I’m not sure that Wi-Fi connectivity is a big deal for fans in the seats during the game action, which in case you’ve not watched soccer, has no breaks like timeouts or inning changes. I’m generalizing here but I think that the continuous-flow of soccer action inherently results in fans who simply watch the game instead of taking breaks to check their phones (Mark Cuban, here’s your sport!). So maybe the expense of bringing Wi-Fi to all the seats at Avaya Stadium isn’t justified.

Halftime view of fans checking phones

That said, it seemed like during halftime there were a lot of people looking at devices in their stadium seats, but I didn’t hear any howls or complaints or see any obvious frustration. I do know that at my seat on the stadium’s top walkway (which can get very very very windy in the late afternoon) the Wi-Fi signal was weak the whole game, never registering more than 1 Mbps on the download side from the start of the game through the second half.

But again, this is just one phone and one person, a person who was also walking around a lot and connecting to multiple APs, a factor that sometimes makes network connections inconsistent. I did find that turning Wi-Fi off and on again helped get a better signal; when we hear back from the stadium network team we’ll ask if the network has been optimized for roaming connections. I did notice that the beer stand on the top deck just behind my “press box” seat was using cell phones and a payment-device gizmo to take credit card payments; when I asked the staffer running the stand she said she’d been taking payments all game using the regular Wi-Fi and hadn’t had any connectivity issues. So, the connectivity mileage may vary.

DAS to the rescue

Though team executives have talked a lot about the stadium’s networking plans, it would be better for fans right now to have a more realistic estimate of what is going on, and when future enhancements like video and food ordering will become a reality. Some improvement will happen in a big way when Mobilitie gets the neutral-host DAS up and running, since many people never think of joining a stadium Wi-Fi network, they just pull out their phones and hope for the best. With advanced cellular in the building, the connectivity loads will be shared between cellular and Wi-Fi, increasing overall capacity. Sunday, I wasn’t able to get either an AT&T 4G device or my Verizon phone to even register with Speedtest.com to get a figure anywhere inside the stadium using a cellular-only connection. While most fans might have been able to send text messages or get regular voice calls, it’s a good guess that many like me were stymied trying to do simple data tasks like post messages to Twitter. It will be interesting to see what the network folks from Avaya Stadium say when they give us the opening-day report.

Cheers to Avaya Stadium from the end zone bar!

In the end, my first impression from a wireless point of view is that Avaya Stadium has a basic, average level of connectivity for a new stadium, with enough reasons to believe it’s going to get better over time. I’m also cutting them some slack since the technology supplier for the venue changed wholesale last year when Avaya came in as a title sponsor, leaving just a few short months for Avaya to get its own gear in the building and in working order. Again, I’m no engineer but I did see things like electrical tape holding some antenna connections in place, the kind of stuff you don’t expect to see in a professional stadium deployment.

And while the connectivity didn’t particularly stand out as awesome, it also was good enough in enough places to make sure there wasn’t the dreaded “no signal” issue that could have soured things for lots of fans. In the end, there was so much to like about the facility — even in my top-row seat I felt close to the action on the field — that it’s hard to call the day anything short of a smashing success, especially if you are a Bay area soccer fan who’s had to endure sub-par stadium experiences in the past. Those days are gone, and Avaya Stadium should be a fast favorite place going forward.

LOTS OF PHOTOS BELOW! Click on any picture for a larger image. Credit all photos: Paul Kapustka, MSR.

Avaya Stadium, from the employee parking lot

Tailgate action before the game

Connecting to Wi-Fi was easy

Interactive stadium map was one of the best things in the app

Wi-Fi APs attached to roof beams

More APs, underneath the canopy roof

Still more AP views

Another AP, out on the end of the stanchion

A view to give perspective on how far away the roof-beam APs are from the stands

Seats with promo scarves. The team asked fans to donate if they wanted to keep the scarves.

More Wi-Fi APs, on the concourse level. This was above a bathroom entrance.

Wi-Fi APs atop small building near the open end zone

Good view of standing-room area in front of end zone bar. It was packed all game.

Panoramic view of the picnic lawn. Hey there Quakes fan!

Fans waiting to get in, about a half hour before game time

Where the rich folks watch from: Over the gate view of a club level area and their nice buffet

Lots of selfies being taken Sunday

You can see the big screen from just about everywhere in the place — great resolution

Filed Under: 4G LTE, Apps, AT&T, DAS, Levi's Stadium, MLS, Soccer, Verizon Wireless, Wi-Fi Tagged With: 4G LTE, Apple, AT&T, Avaya, Avaya Stadium, Chicago Fire, DAS, iPhone, Levi's Stadium, MLS, Mobile Sports Report, Mobilitie, Paul Kapustka, Ruckus Wireless, San Jose, San Jose Earthquakes, Soccer, Stadium Wi-Fi, Verizon Wireless, Wi-Fi

Stadium Tech Report: Wireless connectivity brings fans and business benefits to the Palace at Auburn Hills

March 23, 2015 by Paul Kapustka

The Palace at Auburn Hills. Credit all photos, Palace at Auburn Hills (click on any photo for a larger image).

Not too long ago, a marketing executive who was new to the Detroit Pistons’ ownership team tried to post a tweet during a game at the Palace of Auburn Hills. But to get the message to send, he had to … step outside the building.

Fast forward a few years, and the situation is completely reversed. Not only is there a storm of wireless connectivity inside the Palace, that same Wi-Fi and cellular traffic is keeping fans inside, bringing new fans in and giving the Pistons management team better insight into what all those fans want.

Using technology as an “accelerant,” the Pistons have changed the game for the fans and for themselves, seeding a process that seems destined to help the team build business success as their fan base evolves into one that expects and delights from an always-connected experience.

In a story-telling twist, we’ll tell you one of the early ends to this tale now: Even though the Pistons have suffered on the court of late, finishing the 2013-14 season with a 29-53 record, the team this year had a season ticket renewal rate above 80 percent, according to the Pistons management. Either there’s massive optimism in Motor City, or the Pistons ownership team is doing something very right in making the game-day experience something fans want to keep experiencing.

If you want to believe more in the latter reason, then listen to what Dennis Mannion, president and CEO of Palace Sports and Entertainment and the Detroit Pistons, has to say: There are a lot of smart programs now in place and some that are just starting to take off, but none of it happens, he said, without a solid communications core.

“It always comes back to the great accelerator, technology,” Mannion said.

Building the network inside the building

Editor’s note: This profile is part of our new Stadium Tech Report HOOPS AND HOCKEY ISSUE, available for free download. In addition to this story it contains additional profiles and team-by-team tech capsules for all 30 NBA teams. Download your copy today!

Mannion, who joined the Pistons in 2011 following a long career in sports management that included time as the Los Angeles Dodgers’ president and COO, and stints with the Baltimore Ravens, Denver Nuggets and Philadelphia Phillies, said the new commitment to technology started at the Pistons’ ownership level, where they knew things needed to change.

“When I came in to the job we knew there needed to be some technology improvements [at the Palace],” Mannion said. “We needed to make games more of a happening for our fans.” As part of an overall stadium renovation project that cost somewhere between $13 and $15 million, the Pistons started bringing the Palace into the world of wireless connectivity.

Screen shot of Pistons app

For help, Mannion and the Palace team enlised a Detroit sports-stadium technology consulting firm called Nuvuz Sports, run by CEO Scott Wruble, a former tech exec with the NBA and MLB, and chief marketing officer Jim Wolski. According to Wolski, when the technology revamp of the Palace started in 2012, there was a bit of a connectivity hill to climb.

“The entire building had nine cable modems, total,” Wolski said.

The first step was to start with a DAS, or distributed antenna system, to make sure cellular carriers could connect with their customers. The Palace team wound up picking Verizon Wireless to act as lead on a neutral-host DAS deployment, which is currently in the process of adding AT&T and Sprint to its system.

Opened in 1988, the Palace had some physical challenges when it came to wireless technology, including the omnipresent question of retrofits — where do you put the DAS head end? In the Palace’s case, the solution was a new building built outside, in a Palace parking lot.

Verizon helped on the next step in the deployment process by bringing in their longtime technology partner Ericsson to deliver the gear for the building’s Wi-Fi network. Though somewhat of a newcomer in the stadium Wi-Fi space, Ericsson’s long history in telecommunications equipment showed through with the Wi-Fi deployment, according to Mike Donnay, vice president for brand networks at Palace Sports & Entertainment.

“The performance from the Ericsson [Wi-Fi] equipment is super high,” said Donnay, who was the exec who had to leave the building to send the tweet before the renovation happened. Now, with 238 Wi-Fi access points in the venue (which seats 22,076 for basketball) Donnay doesn’t have any problems tweeting.

Nuvuz’s Wolski said that in addition to providing fan Internet access, the internal Wi-Fi network also powers the game-day ticketing operation as well as concession point of sale. It also is the base for a large suite of fan engagement platforms, which are hyper-targeted to the many different types of Pistons or other event fans who walk in the doors.

Clusters within the clusters

Kevin Grigg, vice president of public relations for the Pistons, said Mannion’s “big picture” ROI is to use technology to engage fans in a one-on-one relationship.

“The biggest challenge [in the past] has been if you are a fan and you buy a ticket, we didn’t know who you are,” Grigg said. “Now, we’re in a position to identify people during the engagement.”

Nuvuz’s Wolski concurs.

“It chages the game when you know who they [the fans] are,” Wolski said.

Mannion said the Pistons’ IT team is already well into the implementation of an elaborate fan-engagement system that breaks the fan base into multiple segments that can each be targeted with programs tailored to their wants and needs.

“We have different fan bases, from those who buy courtside seats, to those in corprate suites to families,” Mannion said. “We have affinity groups, like ‘future Pistons,’ women’s groups, and people interested mainly in attending concerts. We have clusters within clusters. And we use a combination of media, memories and merchandise for each cluster.”

On the team side, Mannion said that “even perceived ‘inside access’ is a real turn-on. Fans like to join, belong and brag.” Some ways fans can get “closer” to the team is to use the stadium app’s seat-upgrade feature (powered by partner PogoSeat) to purchase seats closer to the action, or by special access like post-game shootarounds on the court floor.

There is also a big focus on Pistons-related content, both from the team as well as from fans themselves. And increasingly, fans are turning to social media to share this content. The Pistons have responded with tricks like putting selfies tagged with the #pistonspride hashtag right onto the new huge center court video board.

The tech team also just added beacon technology to the Palace, and are already offering beacon-powered features like seat upgrades to fans who have seats in the upper levels, and discounts on merchandise to fans who are walking right outside the team shops. There are also scoreboard trivia contests and real-time fan polling, all of which keep fans engaged with the team and game even as they stare at their mobile devices.

“We wouldn’t be able to do all this without a connected arena,” Donnay said.

Connected now and for the future

Right now, Donnay said that approximately one-third of a regular Pistons game audience uses the Wi-Fi network, a take rate among the higher of reported crowd participation numbers. The Palace team also uses the Wi-Fi network to connect to 30 “smart carts,” which are mobile food carts that can be moved around (even outside the building) to take advantage of being where hungry fans are.

“It’s interesting to watch the heat maps of how the building is filling, and being able to tell concessions where to beef up,” Mannion said. In the past, he noted, many such game-day operations were decided by “gut feelings,” which could be right or could be wrong. Now, he said, there are facts and figures to back up the guesswork.

“Now you can still act on gut feelings, but you also have ways to prove what’s going on,” Mannion said. “You can do a digital promotion for food and beverage, and immediately measure the impact. That’s really fascinating.”

Even as they build on early successes, the team behind the Palace’s new tech operations knows that these are early days, and much work lies ahead. One current “big drag,” Mannion notes, is the plethora of social-media sharing platforms, a mix that includes Twitter, Facebook, Instagram, Vine, Snapchat and others, almost too many to track coherently.

There’s also a big challenge to keep pace with all the potential partners and digital features that could be added, like instant replays or food ordering for the stadium app; Mannion noted: “The question is, how do you create the right kind of [feature] incubation system, without spending too much money?”

At the very least, the Pistons team has grabbed a leading position in the connected-arena future, one where having advanced connectivity and engagement programs is table stakes to attract new potential season ticket buyers.

“We’ve seen colleges experience heartaches” when their fans leave because the stadiums don’t have connectivity, said Donney. “That’s the fan base that’s coming to the NBA. They’re going to expect that technology, so we’re going to have to be very good at it.”

So far, the Pistons and the Palace seem to be ahead of the curve of the new era of connected stadiums and the fans who fill them. Mannion, for one, knows he’s now in a much different business than the one he’s spent most of his career in.

“It’s a lot different now,” he said, “than just opening up the window and selling tickets.”

Filed Under: 4G LTE, Apps, Basketball, DAS, Ericsson, MSR Guides, NBA, Verizon Wireless, Wi-Fi Tagged With: 4G LTE, DAS, Detroit Pistons, Ericsson, Hoops and Hockey issue, Mobile Sports Report, NBA, Palace at Auburn Hills, Paul Kapustka, Stadium Tech Report, Stadium Wi-Fi, Verizon Wireless, Wi-Fi

Stadium Tech Report: Blazing a Trail to Connectivity at Portland’s Moda Center

March 16, 2015 by Paul Kapustka

Portland’s Moda Center, home of the NBA Trail Blazers. Credit all photos: Moda Center (click on any photo for a larger image)

When it comes time to build a stadium network there are two obvious choices of how to get it done: You let someone else build and run it, or do it yourself.

When neither of these options appealed to the Portland Trail Blazers and their planned networking upgrade at the Moda Center, they went with a third option: Using a neutral host provider for both DAS and Wi-Fi.

After turning to neutral host provider Crown Castle to build out the enhanced cellular DAS (distributed antenna system) network as well as a fan-facing Wi-Fi network, Portland’s home at the Moda Center now has a level of wireless connectivity that mirrors the team’s excellent on-court performance — challenging for the NBA lead and looking to keep improving.

With Wi-Fi gear from Aruba Networks and an app developed by YinzCam, fans at Trail Blazers games can use the stadium Wi-Fi to gain access to live-action video streams, player stats and even a radio broadcast of the game. The app also supports seat upgrades, a feature powered by Experience.

Now in its second year of existence, the network and all its attributes are a big hit with Portland fans, according to Vincent Ircandia, vice president for business analytics and ticket operations for the Trail Blazers, and Mike Janes, vice president of engineering and technology for the Trail Blazers.

“We do a lot of fan surveys, and the approval [for the network] continues to go up,” said Janes. “We want to figure out what we can do next.”

Third-party the third and correct choice

Aruba Wi-FI AP in the rafters at Moda Center

If you dial the clock hands back a few years you would find an arena in Portland with not much connectivity at all, and fans who made their displeasure over the situation known, loud and clearly.

“Trail Blazers fans are not shy about letting us know how they feel,” Ircandia said. “Two years ago we learned that fans didn’t have much connectivity here [at the Moda Center]. That was a real gap in the customer experience.”

Janes said the top two methods of bringing a network in — allowing a carrier to build and operate it, or to build and run it themselves — both had significant drawbacks.

“You could hand it over to a carrier [to be a neutral host] but I’ve seen that before, where one carrier has to ride on another carrier’s DAS,” Janes said. “That’s not a good solution.”

The DIY option, Janes said, would mean that the Trail Blazers team would have to build the networks themselves, and hire someone to manage it.

“That would mean we would have to deal with the [multiple] carriers directly, and we didn’t want to do that,” Janes said.

In the end, Portland went with the neutral host option, with Crown Castle building and running the DAS and installing a Wi-Fi network as well. With its wide experience in building and managing DAS deployments, Crown Castle already has AT&T and Verizon as tenants on the Moda Center DAS, with Sprint coming on soon and possibly T-Mobile joining in the near future.

And on the Wi-Fi side, the team itself “owns” the network and the associated data — “and that’s good, because we are starting to take deeper dives into that,” Janes said.

And how’s it all operating?

“No complaints [recently],” Janes said. “The fans are pretty quiet when it’s just working.”

User numbers flat, data use doubles

Maybe those users are quiet because they are busy posting pictures to Instagram or posts to Facebook, two of the leading applications being used at the Moda Center, according to the Trail Blazers. According to Ircandia, what’s interesting about the usage patterns from last year to this year is that while the number of fans using the network has remained fairly stable (at about 25 percent of attendees), the amount of wireless data consumed has just about doubled from last season to this season, meaning that people are doing more things on the network.

Toyota pregame show on the Moda Center concourse

Some of that may have to do with a redesigned app, which added four live streams of video action, as well as live radio broadcast coverage.

“We didn’t leave [interaction] to chance,” Janes said. “We spent a lot of effort improving the app and redesigning the web presence to make it more enticing and robust.”

While having more features is always a good step, a big part of the challenge for any team is just getting fans to use the network and the team app. At Moda Center, there is a unique method of network promotion, which also directly benefits the team: By selling the “sponsorship” of the network to local-area Toyota dealers, the stadium operators now have a partner who actively promotes the network to fans walking into the building.

According to Ircandia, the Toyota dealers also sponsor a pregame show, set on the arena concourse with the team’s dancers in attendance to help attract attention. “They [Toyota] want to sell cars, so they have a lot of signage [about the network],” in addition to the show, Incardia said.

If there is such a thing as a good problem, there is one involving the ticket upgrade feature: Since the Trail Blazers have such passionate fans and are doing so well, there aren’t many available seats to offer for upgrades. “It’s a bit of a constrained inventory,” Janes said. Still, the team is using the Experience feature to offer last-minute ticket deals to students in the area, alerting them that there are seats available that may have a chance to be upgraded.

“They [students] have the time to drop whatever they’re doing to come over to the game,” Janes said.

Retrofitting an old flower

Previously known as the Rose Garden, the arena, which was built in 1995, clearly wasn’t initially designed with wireless connectivity in mind.

“We were retrofitting for technology that didn’t exist [when the building was built],” Ircandia said. “That’s where creativity comes in.”

DAS headend gear

Some of the creativity in network design included separate Wi-Fi antennas for 2.4 GHz and 5 GHz transmissions, so that older devices didn’t have to compete with newer devices for the best Wi-Fi connection. Ircandia also said that the Moda Center “created” some new DAS headend space up in the highest reaches of the building.

“We built catwalks above the catwalks,” Ircandia said.

Looking ahead to what’s next, Janes said the network team is looking to use wireless to connect to food and beverage carts, so that point of sale operations don’t need to be tied to a plug in the wall.

“If we go wireless we can move the carts around, even put them outside, which gives us a business case improvement with no impact to the fans,” Janes said.

And now that those fans know what is possible, they will want what they have now, and more.

“That’s their expectation now,” Janes said, “that it will work when they walk in the building.”

Filed Under: 4G LTE, Apps, AT&T, Basketball, DAS, NBA, Sprint, T-Mobile, Verizon Wireless, Wi-Fi, YinzCam Tagged With: 4G LTE, Aruba Networks, AT&T, Crown Castle, DAS, Mobile Sports Report, Moda Center, NBA, Paul Kapustka, Portland Trail Blazers, Sprint, Stadium Tech Report, Stadium Wi-Fi, T-Mobile, Verizon Wireless, Wi-Fi, YinzCam

Hockey crowd melted down Levi’s Stadium network and app, overwhelmed light rail

February 24, 2015 by Paul Kapustka

Levi’s Stadium scoreboard during Stadium Series hockey game. Credit all images: Paul Kapustka, MSR (click on any photo for larger image).

From a financial and publicity standpoint Saturday’s Coors Light Stadium Series hockey game at Levi’s Stadium was a success, with 70,205 fans packing the football facility to watch the San Jose Sharks lose to the Los Angeles Kings, 2-1. But while the sellout crowd contributed to the general electricity that filled the venue, the mass of people also caused problems with the stadium’s vaunted wireless network, knocking out some parts of the Wi-Fi and cellular networks and overwhelming the unique feature of the stadium app designed to allow fans to have food and drinks delivered to their seats.

Hockey fans also swamped the VTA light rail system, causing some fans to wait as long as two hours before they could catch a bus or train to get home from the stadium. Though light rail officials said they will work on correcting the problems, the commuting jam does not bode well for a facility that is scheduled to host Super Bowl 50 in less than a year’s time, especially since many Super Bowl fans are expected to be traveling from San Francisco to the Santa Clara, Calif., neighborhood where Levi’s Stadium sits.

According to Roger Hacker, senior manager for corporate communications for the San Francisco 49ers, the Levi’s Stadium network team identified “isolated interruptions” of the Wi-Fi network, due to “frequency coordination issues” that the network team had not seen at previous events. Hacker also said that one unnamed wireless carrier had “issues” with its base station firmware, but said that the problems were resolved by game’s end. (For the record, I am a Verizon Wireless customer and I had “issues” getting cellular connectivity Saturday, so draw your own conclusions.)

Since the Niners’ full explanation is somewhat light on facts and numbers, we will first offer a “fan’s view” of the events Saturday night, under the caveat that Mobile Sports Report was not attending the game as press, but instead as just a regular hockey fan (one who purchased two full-price tickets) who was looking forward to using the stadium’s technology to enhance the game experience. Unfortunately for this fan, the Levi’s Stadium network, app and transit services all fell down on the job.

Light show a dud

Though the MSR team had no problems getting to the stadium — our light rail train out of Mountain View at about 5:30 p.m. was relatively empty — I noticed some irregularities in network connections during the pregame ceremonies, when I tried to join in the fan-participation light show, a technology feature recently added to the Levi’s Stadium app especially for the Stadium Series game. Like many people in our area, I couldn’t get the app to work, leaving me staring at a spinning graphic while others in the stadium saw their phones contribute flashing lights during pre-game music.

After the light show segment ended, I noticed that the Levi’s app was performing erratically, quitting on its own and kicking my device off the Wi-Fi network. After rebooting the device (a new Apple iPhone 6 Plus) I still couldn’t connect to the Wi-Fi, an experience I’ve never had at Levi’s. Turning off the Wi-Fi didn’t help, as cellular service also seemed poor. Since I wasn’t really there to work — I just wanted to enjoy the game with my older brother, who was in town for the event — I posted a quick tweet and went back to just watching the Sharks play poorly for the first 20 minutes.

Lots if #fail for the @LevisStadium app interactive feature. Everyone around us couldn’t get it to work. Also keep getting kicked off wifi

— paulkaps (@paulkaps) February 22, 2015

One of the benefits of being a close follower of Levi’s Stadium technology is that when you tweet, people listen. By the middle of the first intermission, I was visited personally by Anoop Nagwani, the new head of the Levi’s Stadium network team, along with a technician from Aruba Networks, the Wi-Fi gear supplier at the stadium. Even with laptops and scanners, my visitors couldn’t immediately discern the network problem; they were, however, visited by a number of other nearby fans, who figured out who they were and relayed their own networking problems to them.

To be clear: I didn’t spend the game as I usually do at Levi’s, wandering around to see how the network is performing at as many spots as I can. But even if the outage was only in our area, that’s a significant problem for Levi’s Stadium, which has touted its technology every chance it gets. I also noticed problems with cellular connectivity all night, which leads me to believe that the network issues were more widespread than just at my seating area.

The official statement from Hacker describing the problems doesn’t pin any specific blame, but a guess from us is that perhaps something in the mix of systems used by the entertainment performers (there was a small stage to one side of the rink where musicians performed) and media new to the facility caused the Wi-Fi problem. Here is the official statement on the Wi-Fi issues:

The Levi’s Stadium network team identified isolated interruptions of the WiFi system in specific sections on Saturday night due to frequency coordination issues previously unseen at the venue and unique to this event. Saturday’s event featured extra radio systems not typical to previous stadium events, some of which were found to be unauthorized by event frequency coordinators. To avoid similar situations in the future, Levi’s Stadium management will be initiating additional frequency control protocols for all events.

Hacker said the network team did not track exactly how widespread the outages were, so could not provide a number of fans affected. But enough apparently did connect, since according to Hacker, the Levi’s network saw near-record traffic Saturday night, with a total of 3.0 terabytes of data carried, second only to the season-opening Niners game back in September, which saw 3.3 TB of data used on the Wi-Fi. Hacker said there were 24,792 unique devices connected to Wi-Fi during Saturday’s event, with a peak concurrent user number of 17,400 users, also second highest behind the season-opener total of 19,0000. The Stadium Series game did set a new mark for throughput with 3.5 Gbps on the network just before the start of the game, a surge that seems to be behind some of the other problems.

Food ordering overwhelmed

During the intermission, my brother and I went out on the 300-level concourse to get something to eat and drink — and encountered one of the untold stories of Levi’s Stadium: the incredibly long and slow lines for concessions. While I haven’t researched this problem in depth, after 10 minutes of inertia in our line I told my brother I would use the app’s food and drink ordering function to get us some vittles and beverages. Finally able to connect via Wi-Fi while on the concourse I placed an order for two beers and two hot dogs, and didn’t worry that the delivery time was 20 minutes. That would put it at the very latest near the end of the second period, which was fine by me since it meant I didn’t have to wait in lines. Or so I thought.

Back in my seat, I was troubled by the fact that even halfway through the period, the app had not switched yet from ordered to “en route.” I also got some error messages I had never seen at Levi’s Stadium before:

Unreal #fail at @LevisStadium pic.twitter.com/M2T8ByIMqy

— paulkaps (@paulkaps) February 22, 2015

When the period ended and there was still no movement from the app (which I only checked sporadically since Wi-Fi never fully connected in my seat), I went back on the concourse where I found a small, angry crowd around the food-runner window at the closest concession stand. Pretty much, everyone there had the same problem I had: We’d ordered food and the app had said that the order had been taken, but nothing had happened since then.

Fans trying to figure out why their food orders weren’t delivered

The situation wasn’t good since nobody at the food-runner window had any technology that would allow them to communicate with the app or network team; they couldn’t even cancel orders or make sure credit card refunds would be processed, which only served to increase the frustration for the fans who were just trying to use the services as advertised.

In the end, the staff at the delivery window did the best they could — which at one point resulted in someone producing slips of paper which the waiting fans used to write down their orders; one staffer then tried to fulfill those orders as best he could, going to the concession stand and bringing them out one by one. After waiting nearly the full intermission (missing Melissa Etheridge) I was given two cold hot dogs and two draft beers. Since there were no food holders left at the stand, I had to put the hot dogs into my jacket pockets and hold both beers. At least I didn’t starve or go thirsty, but it was a far cry from the delivered-to-the-seat functionality I had raved about to my brother that simply didn’t materialize.

During this process I sent an email to Louise Callagy, vice president of marketing at stadium app developer VenueNext. Her in-game response was:

“Levi’s Stadium app usage exceeded any previous event and set new records, causing delivery and order fulfillment delays. As always, we will do a post mortem after the event, and make the necessary adjustments to operational and staffing support, including systems performance analysis. We apologize to any fans who were inconvenienced.”

According to Hacker, the Levi’s Stadium food-runner staffing was at the same level as a regular-season Niners’ game; however, Hacker said the hockey fans broke the previous ordering records before the first period was over. Here is the official statement on the food ordering snafu:

With more than 31,000 new downloads of the Levi’s Stadium App – 20 percent more than had ever been seen at any previous stadium event – the [food ordering] system experienced 50 percent higher order volume in the just first hour of the game than had been seen during any previous event. The dramatic increase led to the extended wait times and cancelled orders experienced by some fans.

In a separate email, Hacker did not provide an exact number for how many fans were represented by the term “some,” but he did confirm that “no customers were charged for unfulfilled orders.”

Still, the system shouldn’t have had any unfulfilled orders, at least not according to the Niners’ consistent hype of the network and the app. Remember, Niners officials had long been confident that their network would be able to stand up to any load. Such was not the case Saturday night.

The long wait home

VTA line following Levi’s Stadium hockey game

After an exciting third period and a game that went down to the final horn, we left the stadium and were immediately greeted by a mass of people packing in to the VTA departure area. With too many people and not enough trains and buses, we spent almost an hour moving like slow cattle until we eventually got on a train to Mountain View. We considered ourselves lucky, since it looked like the folks heading south on VTA were in for an even longer wait.

When we got to the Mountain View station, we waited almost another hour to leave since Caltrain (nicely) kept its last train at the station until two more VTA trains brought the stragglers in from Levi’s. Though VTA has since claimed there were more than twice the “normal” number of riders than it saw at Niners games this season, there was no explanation why VTA didn’t or couldn’t provide more capacity after it saw more fans use the service to get to the game. What was most unpleasant was the overall unorganized method of boarding the trains, just a massive group line with one VTA person on a bullhorn telling everyone to make sure they bought a ticket.

@paulkaps we delivered svc for 7k-9k as at 49ers games. Ended up with nearly double that so it took twice as long to clear post-game ^CK

— VTA (@VTA) February 23, 2015

In the end, the time it took to get from the start of the VTA line to my house in San Mateo was three hours — almost as long as the game itself. With some other “special” events like Wrestlemania and concerts coming up at Levi’s and the Super Bowl 50 next year, it’s clear there is lots of work that needs to be done to make it a good experience for all who purchase a ticket, especially those looking to use public transport and the app features to enhance their game-day experience.

Sharks and Kings on the ice at Levi’s Stadium

Filed Under: 4G LTE, Apps, AT&T, DAS, Levi's Stadium, MSR Guides, NFL, NHL, San Jose Sharks, Verizon Wireless, Wi-Fi Tagged With: Coors Light Stadium Series, DAS, Levi's Stadium, Los Angeles Kings, Mobile Sports Report, NHL, Paul Kapustka, San Francisco 49ers, San Jose Sharks, Stadium Wi-Fi, VenueNext, Wi-Fi

Super Bowl XLIX sets new stadium Wi-Fi record with 6.2 Terabytes of data consumed

February 4, 2015 by Paul Kapustka

University of Phoenix Stadium. Credit: Arizona Cardinals.

The Super Bowl is once again the stadium Wi-Fi champ, as fans at Sunday’s Super Bowl XLIX in Glendale, Ariz., used 6.23 terabytes of data during the contest, according to the team running the network at the University of Phoenix Stadium.

The 6.23 TB mark blew past the most recent entrant in the “most Wi-Fi used at a single-day single-stadium event” sweepstakes, the 4.93 TB used at the Jan. 12 College Football Playoff championship game at AT&T Stadium. Prior to that, pro football games this past season at Levi’s Stadium in Santa Clara, Calif., and at AT&T Stadium had pushed into the 3-plus TB mark to be among the highest totals ever reported.

The live crowd watching the New England Patriots’ 28-24 victory over the Seattle Seahawks also used about as much cellular data as well, with Verizon Wireless, AT&T and Sprint claiming a combined total of 6.56 TB used in and around the stadium on game day. All three carriers were on the in-stadium and outside-the-stadium DAS deployments being run by neutral host Crown Castle. If those figures are correct (more on this later) it would put the total wireless data usage for the event at 12.79 TB, far and away the biggest single day of wireless data use we’ve ever heard of.

Apple OS updates still the application king

Handrails with Wi-Fi antenna enclosures from AmpThink. Credit: Arizona Cardinals.

Mark Feller, vice president of information technology for the Arizona Cardinals, and Travis Bugh, senior wireless consultant for CDW, provided Mobile Sports Report with the final Wi-Fi usage numbers, which are pretty stunning for anyone in the stadium networking profession. According to Feller the new CDW-deployed Wi-Fi network with Cisco gear at the UoP Stadium saw 2.499 TB of data downloaded, and 3.714 TB uploaded, for a total of 6.213 TB of Wi-Fi usage. Bugh of CDW said there were 25,936 unique devices connecting to the network on game day, with a peak concurrent usage of 17,322, recorded not surprisingly at halftime.

Peak download usage of 1.3 Gbps was recorded before the game’s start, while peak upload usage of 2.5 Gbps was hit at halftime. The top applications by bandwidth use, Feller said, were Apple (mobile update), Facebook, Dropbox and Snapchat.

DAS numbers also set new record, but clarification needed

The only reason we aren’t yet trumpeting the 6.564 TB of reported DAS use as a verified record is due to the differences in clarity from each of the reporting providers. We also haven’t yet heard any usage totals from T-Mobile, so it’s likely that the final final wireless data use number is somewhere north of 13 TB, if all can be believed.

Parking lot light poles, Westgate entertainment district. Can you spot the DAS?

As reported before, AT&T said it saw 1.7 TB of cellular wireless activity from its customers on game day, with 696 GB of that happening inside the stadium, and the balance coming from the outside areas before and after the game. We’d also like to welcome Sprint to the big-game reporting crew (thanks Sprint!), with its total of 754 GB of all 4G LTE traffic used in and around the stadium on game day. According to Sprint representatives, its Super Bowl coverage efforts included 5 COWs (cell towers on wheels) as well as expanded DAS and macro placements in various Phoenix-area locations. The Sprint coverage included the 2.5 GHz spectrum that uses TDD LTE technology.

As also previously reported, Verizon Wireless claimed 4.1 TB of customer traffic in and around the stadium on game day, which Verizon claims is all cellular traffic and does not reflect any Verizon Wireless customer use of the stadium Wi-Fi network. Verizon also reported some other interesting activity tidbits, which included 46,772 Verizon Wireless devices used at the game, of which just 59.7 percent were smartphones. Verizon also said it saw 10 million emails sent on its networks that day, and 1.9 million websites visited, while also seeing 122.308 videos sent or received over wireless connections.

We’re still waiting to see if we can get usage numbers from the Super Bowl stadium app (we’re especially interested to see if the instant replay feature caught on) but the warning for stadium owners and operators everywhere seems to be clear: If you’re hosting the big game (or any BIG game), make sure your network is ready for 6 TB and beyond!

Filed Under: 4G LTE, Apple, AT&T, Cisco, DAS, iPhone, MSR Guides, NFL, Seattle Seahwaks, Sprint, Super Bowl, Verizon Wireless, Wi-Fi Tagged With: AT&T, COWs, DAS, Dropbox, Mobile Sports Report, New England Patriots, Paul Kapustka, Seattle Seahawks, SnapChat, Sprint, Stadium Wi-Fi, Super Bowl, Super Bowl XLIX, University of Phoenix Stadium, Verizon Wireless, Wi-Fi