Levi’s Stadium scoreboard during Stadium Series hockey game. Credit all images: Paul Kapustka, MSR (click on any photo for larger image).
Hockey fans also swamped the VTA light rail system, causing some fans to wait as long as two hours before they could catch a bus or train to get home from the stadium. Though light rail officials said they will work on correcting the problems, the commuting jam does not bode well for a facility that is scheduled to host Super Bowl 50 in less than a year’s time, especially since many Super Bowl fans are expected to be traveling from San Francisco to the Santa Clara, Calif., neighborhood where Levi’s Stadium sits.
According to Roger Hacker, senior manager for corporate communications for the San Francisco 49ers, the Levi’s Stadium network team identified “isolated interruptions” of the Wi-Fi network, due to “frequency coordination issues” that the network team had not seen at previous events. Hacker also said that one unnamed wireless carrier had “issues” with its base station firmware, but said that the problems were resolved by game’s end. (For the record, I am a Verizon Wireless customer and I had “issues” getting cellular connectivity Saturday, so draw your own conclusions.)
Since the Niners’ full explanation is somewhat light on facts and numbers, we will first offer a “fan’s view” of the events Saturday night, under the caveat that Mobile Sports Report was not attending the game as press, but instead as just a regular hockey fan (one who purchased two full-price tickets) who was looking forward to using the stadium’s technology to enhance the game experience. Unfortunately for this fan, the Levi’s Stadium network, app and transit services all fell down on the job.
Light show a dud
Though the MSR team had no problems getting to the stadium — our light rail train out of Mountain View at about 5:30 p.m. was relatively empty — I noticed some irregularities in network connections during the pregame ceremonies, when I tried to join in the fan-participation light show, a technology feature recently added to the Levi’s Stadium app especially for the Stadium Series game. Like many people in our area, I couldn’t get the app to work, leaving me staring at a spinning graphic while others in the stadium saw their phones contribute flashing lights during pre-game music.
After the light show segment ended, I noticed that the Levi’s app was performing erratically, quitting on its own and kicking my device off the Wi-Fi network. After rebooting the device (a new Apple iPhone 6 Plus) I still couldn’t connect to the Wi-Fi, an experience I’ve never had at Levi’s. Turning off the Wi-Fi didn’t help, as cellular service also seemed poor. Since I wasn’t really there to work — I just wanted to enjoy the game with my older brother, who was in town for the event — I posted a quick tweet and went back to just watching the Sharks play poorly for the first 20 minutes.
Lots if #fail for the @LevisStadium app interactive feature. Everyone around us couldn't get it to work. Also keep getting kicked off wifi
— paulkaps (@paulkaps) February 22, 2015
One of the benefits of being a close follower of Levi’s Stadium technology is that when you tweet, people listen. By the middle of the first intermission, I was visited personally by Anoop Nagwani, the new head of the Levi’s Stadium network team, along with a technician from Aruba Networks, the Wi-Fi gear supplier at the stadium. Even with laptops and scanners, my visitors couldn’t immediately discern the network problem; they were, however, visited by a number of other nearby fans, who figured out who they were and relayed their own networking problems to them.
To be clear: I didn’t spend the game as I usually do at Levi’s, wandering around to see how the network is performing at as many spots as I can. But even if the outage was only in our area, that’s a significant problem for Levi’s Stadium, which has touted its technology every chance it gets. I also noticed problems with cellular connectivity all night, which leads me to believe that the network issues were more widespread than just at my seating area.
The official statement from Hacker describing the problems doesn’t pin any specific blame, but a guess from us is that perhaps something in the mix of systems used by the entertainment performers (there was a small stage to one side of the rink where musicians performed) and media new to the facility caused the Wi-Fi problem. Here is the official statement on the Wi-Fi issues:
The Levi’s Stadium network team identified isolated interruptions of the WiFi system in specific sections on Saturday night due to frequency coordination issues previously unseen at the venue and unique to this event. Saturday’s event featured extra radio systems not typical to previous stadium events, some of which were found to be unauthorized by event frequency coordinators. To avoid similar situations in the future, Levi’s Stadium management will be initiating additional frequency control protocols for all events.
Hacker said the network team did not track exactly how widespread the outages were, so could not provide a number of fans affected. But enough apparently did connect, since according to Hacker, the Levi’s network saw near-record traffic Saturday night, with a total of 3.0 terabytes of data carried, second only to the season-opening Niners game back in September, which saw 3.3 TB of data used on the Wi-Fi. Hacker said there were 24,792 unique devices connected to Wi-Fi during Saturday’s event, with a peak concurrent user number of 17,400 users, also second highest behind the season-opener total of 19,0000. The Stadium Series game did set a new mark for throughput with 3.5 Gbps on the network just before the start of the game, a surge that seems to be behind some of the other problems.
Food ordering overwhelmed
During the intermission, my brother and I went out on the 300-level concourse to get something to eat and drink — and encountered one of the untold stories of Levi’s Stadium: the incredibly long and slow lines for concessions. While I haven’t researched this problem in depth, after 10 minutes of inertia in our line I told my brother I would use the app’s food and drink ordering function to get us some vittles and beverages. Finally able to connect via Wi-Fi while on the concourse I placed an order for two beers and two hot dogs, and didn’t worry that the delivery time was 20 minutes. That would put it at the very latest near the end of the second period, which was fine by me since it meant I didn’t have to wait in lines. Or so I thought.
Back in my seat, I was troubled by the fact that even halfway through the period, the app had not switched yet from ordered to “en route.” I also got some error messages I had never seen at Levi’s Stadium before:
Unreal #fail at @LevisStadium pic.twitter.com/M2T8ByIMqy
— paulkaps (@paulkaps) February 22, 2015
When the period ended and there was still no movement from the app (which I only checked sporadically since Wi-Fi never fully connected in my seat), I went back on the concourse where I found a small, angry crowd around the food-runner window at the closest concession stand. Pretty much, everyone there had the same problem I had: We’d ordered food and the app had said that the order had been taken, but nothing had happened since then.
Fans trying to figure out why their food orders weren’t delivered
In the end, the staff at the delivery window did the best they could — which at one point resulted in someone producing slips of paper which the waiting fans used to write down their orders; one staffer then tried to fulfill those orders as best he could, going to the concession stand and bringing them out one by one. After waiting nearly the full intermission (missing Melissa Etheridge) I was given two cold hot dogs and two draft beers. Since there were no food holders left at the stand, I had to put the hot dogs into my jacket pockets and hold both beers. At least I didn’t starve or go thirsty, but it was a far cry from the delivered-to-the-seat functionality I had raved about to my brother that simply didn’t materialize.
During this process I sent an email to Louise Callagy, vice president of marketing at stadium app developer VenueNext. Her in-game response was:
“Levi’s Stadium app usage exceeded any previous event and set new records, causing delivery and order fulfillment delays. As always, we will do a post mortem after the event, and make the necessary adjustments to operational and staffing support, including systems performance analysis. We apologize to any fans who were inconvenienced.”
According to Hacker, the Levi’s Stadium food-runner staffing was at the same level as a regular-season Niners’ game; however, Hacker said the hockey fans broke the previous ordering records before the first period was over. Here is the official statement on the food ordering snafu:
With more than 31,000 new downloads of the Levi’s Stadium App – 20 percent more than had ever been seen at any previous stadium event – the [food ordering] system experienced 50 percent higher order volume in the just first hour of the game than had been seen during any previous event. The dramatic increase led to the extended wait times and cancelled orders experienced by some fans.
In a separate email, Hacker did not provide an exact number for how many fans were represented by the term “some,” but he did confirm that “no customers were charged for unfulfilled orders.”
Still, the system shouldn’t have had any unfulfilled orders, at least not according to the Niners’ consistent hype of the network and the app. Remember, Niners officials had long been confident that their network would be able to stand up to any load. Such was not the case Saturday night.
The long wait home
VTA line following Levi’s Stadium hockey game
When we got to the Mountain View station, we waited almost another hour to leave since Caltrain (nicely) kept its last train at the station until two more VTA trains brought the stragglers in from Levi’s. Though VTA has since claimed there were more than twice the “normal” number of riders than it saw at Niners games this season, there was no explanation why VTA didn’t or couldn’t provide more capacity after it saw more fans use the service to get to the game. What was most unpleasant was the overall unorganized method of boarding the trains, just a massive group line with one VTA person on a bullhorn telling everyone to make sure they bought a ticket.
@paulkaps we delivered svc for 7k-9k as at 49ers games. Ended up with nearly double that so it took twice as long to clear post-game ^CK
— VTA (@VTA) February 23, 2015
In the end, the time it took to get from the start of the VTA line to my house in San Mateo was three hours — almost as long as the game itself. With some other “special” events like Wrestlemania and concerts coming up at Levi’s and the Super Bowl 50 next year, it’s clear there is lots of work that needs to be done to make it a good experience for all who purchase a ticket, especially those looking to use public transport and the app features to enhance their game-day experience.
Sharks and Kings on the ice at Levi’s Stadium
