Stadium Tech Report: How VenueNext generates replays for Levi’s Stadium app

replay2During the final home game for the San Francisco 49ers this season, I was somewhat amazed to see a replay appear on the Levi’s Stadium app even before the next play had concluded. Clearly, the VenueNext team behind the app had progressed significantly from the early season, when the app’s replay function struggled somewhat.

During the Dec. 28 game against the Arizona Cardinals, the VenueNext team invited Mobile Sports Report to Levi’s Stadium for a behind-the-scenes look at how the replays are generated; here’s a quick take on how the video moves from playing field cameras to fans’ phones in a matter of seconds.

Step 1: Capturing what the cameras see

How do you make sure you have replays ready? You start by basically capturing all the action from all the cameras in the building. Those views are shuttled back to the main Levi’s Stadium video room, where 15 people have the somewhat enviable job of watching views from all the live cameras in the stadium. (We still haven’t heard a response to our offer of bringing a cooler in exchange for a spare seat next season.)

Inside the video room at Levi's Stadium. (Click on any photo for larger image) Credit all photos: Paul Kapustka, MSR

Inside the video room at Levi’s Stadium. (Click on any photo for larger image) Credit all photos: Paul Kapustka, MSR

According to VenueNext CEO John Paul, who led our tour at Levi’s, VenueNext has its own servers to cache video — here is a look at the Elemental gear used to store the video for the app’s replay function. Paul said that so far, VenueNext has only needed a little more than half of the capacity of the four servers pictured.

Elemental gear in data room next to video room at Levi's Stadium

Elemental gear in data room next to video room at Levi’s Stadium

Step 2: Labeling and thumbnailing the replay

While most of this kind of gear is not new to broadcast operations, the interesting part from an app perspective is the next layer, the choosing and human editing. One of my bigger questions was about how you exactly get a replay set up, labeled with a headline and a thumbnail, and into the app within seconds? The answer is: You write another app, and have two Android-based tablets with football-savvy operators in the press box so they can see the plays live and quickly attach the appropriate info to the replay clips.

In the two pictures below, you can see an over-the-shoulder view of the two tablet apps; the first one is a sort of play-by-play generator, which the operator uses to label the next replay clip as either a run, pass, punt, incomplete, etc., so that a fan looking at the app can quickly figure out what the play might be (“45-yard TD pass,” or something like that). On the second tablet app, another operator sees instant thumbnails gleaned from the video room from all the different camera angles, and when the play is over, that operator picks a thumbnail with a click and the replay is on its way to the app.

First replay tablet app, which adds info about the play

First replay tablet app, which adds info about the play

Second replay tablet app, which adds a thumbnail to the replay

Second replay tablet app, which adds a thumbnail to the replay

Since we only had a few minutes to watch the replay team in action (and since Paul was simultaneously hosting a group of representatives from a prospective future sports-team client), I didn’t get to ask detailed questions about how this all works but it does give you some idea. What’s impressive about the Levi’s Stadium app is that it offers four different camera angles of the same play, as well as a multi-cam view that shows all different angles at once. Though I’m not a fan of the all-at-once option I did find it enjoyable to watch a reply from multiple different perspectives, like watching a pass being caught from both a sideline and end-zone view.

Step 3: Educating fans how to use replay

Initially one of the most-hyped features of the planned stadium app, the replay function has seen only limited use this season, making it somewhat of a disappointment to Niners CEO Jed York. From our perspective, there were two big things holding back extended use of the replay function: The first was the ever-changing appearance of the app itself, which felt like a beta project for most of the season, with a different interface almost every single game; the second thing holding people back from using the app to view replays are the two excellent humungous video boards at Levi’s, which not only show live game action but also quickly show replays, sometimes from those multiple angle views as well.

There might even be a third thing holding fans back from trying out the replay function — that would be just getting fans to use the stadium app in the first place. While usage of the cellular and Wi-Fi networks started at record levels and remained strong from game to game, most fans at Levi’s Stadium were using their devices for other apps, like Twitter, Instagram, Facebook and good old email. As the season progressed, the team started promoting the app and the network more aggressively, adding more-frequent messages to the main video boards and dressing its stadium tech support crew (the “NiNerds”) in neon vests for easier recognition.

What’s next for next season? My guess is that with a year under their belt, the Niners and VenueNext will have a more-aggressive campaign in 2015 to steer more fans to the replay function, which is truly a wonderful enhancement to the game-day experience.

Niners’ CEO gives Levi’s Stadium operations a ‘B’ grade

Jed York, Niners CEO, speaking at tech summit at Levi's Stadium. Credit all photos: Paul Kapustka, MSR

Jed York, Niners CEO, speaking at tech summit at Levi’s Stadium. Credit all photos: Paul Kapustka, MSR

San Francisco 49ers CEO Jed York, who caused a stir last week by apologizing for his team’s play on Twitter, on Thursday gave the operation of his club’s new Levi’s Stadium an above-average grade of “B” for the first four months of its inaugural season.

Speaking at a technology-focused fan-experience and innovation summit held Thursday at the stadium’s “501” club, York said “I’d give us a ‘B’ on our execution [at Levi’s Stadium] this year,” citing parking issues and foot-traffic flow into the stadium as problems not yet fully solved for the 68,500-seat facility in Santa Clara, Calif.

Given the new stadium’s complicated location — in the middle of a busy corporate-headquarters area and right next to the Great America theme park — it was probably somewhat of a given that there would be parking struggles the first season, as fans, police, traffic directors and stadium workers all figured out how to make the dance work. Though some progress has been made during the season, York said that “parking and just getting people here” remain the biggest issue he sees at Levi’s Stadium.

While the Niners have tried to use technology to solve the problems of getting fans inside (with app-based parking maps and wayfinding), York said that despite plans to funnel fans through the correct gates to get more quickly to their seats, many fans still just head for the gate that’s closest to their parking or train arrival spot, which has sometimes led to big backups at the check-in lines.

On the networking side of things, York said all seemed to be going well with the stadium’s Wi-Fi and cellular networks, which have performed well in a series of ad hoc tests conducted by MSR in visits this season.

“We haven’t had any [network] glitches, but we’ve been doing a lot of tweaking,” York said. The Levi’s Stadium app has been the center of most of that tweaking, with continual upgrades to add features and fix problems like an Android bug that surfaced in a mid-November revision. York said that the Niners have been careful to make sure the human engineering behind the technology is solid before launching new things, like having enough runners to support the feature that allows food to be ordered and delivered to seats.

Levi's Stadium ready for the Pac-12 championship game

Levi’s Stadium ready for the Pac-12 championship game

And though it hasn’t been officially confirmed yet, York said the team might debut a planned feature of having team merchandise available for purchase and delivery to seats at the next home game, Dec. 20 vs. San Diego. John Paul, CEO of Levi’s Stadium app developer VenueNext, also spoke at the summit Thursday and said in an interview that the Dec. 20 game might also see the debut of a new feature that adds in mass transit and Uber wait times. (A good idea for fans going to the stadium is to check on game day morning for any updates to the app.)

On the app side, York said one surprise was the fairly low uptake of fans using the app’s instant replay features. After fans watched 7,800 replays during the regular-season home opener (the first game replays were available in the app), usage has gone down, with less than 4,000 replays watched during the last home game against Seattle.

“We thought that mobile replays would be absolutely a home run feature, but it hasn’t got that much traction,” said York. The culprit, he said, might be the twin HD displays above each end zone at Levi’s Stadium, which are somewhat stunning in their clarity. “We do have great video boards,” York said.

In his talk York also thought out loud about the possibility of using wearable technology to both better help prevent player injuries as well as being able to provide more rich detail for fans, like how hard a hit was on the field. But he also stressed that technology at Levi’s Stadium was not meant to be used for technology’s sake, but instead to improve the experience of being at the game. And improvement on the stadium staff’s execution, York said, will go a long way to making it all work together.

“When we get to an A, or A+, people will really be blown away,” York said.

Levi's Stadium at twilight

Levi’s Stadium at twilight

Niners’ home opener tops Super Bowl for Wi-Fi data traffic with 3.3 Terabytes

Fans take pictures of opening kickoff from southwest concourse. Credit, all photos: Paul Kapustka, MSR

Fans take pictures of opening kickoff from southwest concourse. Credit, all photos: Paul Kapustka, MSR

The first regular-season home game for the San Francisco 49ers in their new home, Levi’s Stadium, produced more Wi-Fi traffic and far more actual fan-to-network connections than the most-recent Super Bowl, according to statistics from the Niners’ tech team.

Dan Williams, vice president of technology for the 49ers, said the Levi’s Stadium Wi-Fi network carried 3.3 Terabytes of data during Sunday night’s game between the Niners and the Chicago Bears, topping the 3.2 TB mark reported from Super Bowl XLVIII in February. According to Williams, out of the 70,799 that filled Levi’s Stadium Sunday, more than 30,000 fans connected to the Wi-Fi network at some point, with peak usage of 19,000 fans all connecting at one time occurring just before the 5:30 p.m. local time kickoff. According to the Super Bowl stats, the peak number of fans on Wi-Fi at that game at MetLife Stadium in New Jersey was 13,500.

“We’re pretty excited by Sunday,” said Williams, who said that the Wi-Fi network stood up well even under peak data transfer rates of 3.1 Gbps right before kickoff, and another 2.6 Gbps peak around 7:30 p.m. Around the peaks, network traffic stayed “well over 1 gig per second for three and a half hours,” Williams said.

North scoreboard screen at Levi's Stadium.

North scoreboard screen at Levi’s Stadium.

During the Niners’ first preseason game against the Denver Broncos, the Levi’s Stadium Wi-Fi network carried 2.13 TB of data, and during the Aug. 24 preseason game against San Diego there was another 1.96 TB of Wi-Fi data. The figures do not include any reporting from the stadium’s DAS network, which carries cellular traffic from AT&T, Verizon Wireless and T-Mobile customer. If preseason games are any indication, Williams expects to see numbers in the terabyte range for DAS traffic as well.

The Wi-Fi numbers from Sunday showed that fans quickly figured out a name change in the network name (or SSID). During preseason games, the Wi-Fi network was identified as “Levi’s Stadium” in a device list of available networks; on Sunday the free stadium network used the name “xfinitywifi,” reflecting the brand of Wi-Fi sponsor Comcast. Some fans might have been confused since the “xfinitywifi” SSID is the same one used by Comcast for its public Wi-Fi networks.

“Some folks may have been scratching their heads,” said Williams. “We changed the name last Monday before the opener.”

Replay app gets 7,800 views

As previously reported by MSR, the instant replay feature of the Levi’s Stadium app had its debut Sunday, and according to Williams fans watched 7,800 replays via the app. The top replay view was of the early touchdown pass from Niners QB Colin Kaepernick to Michael Crabtree, which Williams said was viewed more than 1,000 times.

Fans on southwest concourse take photos of live action.

Fans on southwest concourse take photos of live action.

As MSR reported, the replay feature was somewhat limited in functionality, not working at all until late in the first half and then only offering the last two plays plus some scoring highlights for viewing. Previously, team executives had said the replay feature would offer multiple camera angles and multiple replay reviews all at the same time. According to Williams, more features will be added to the replay function in the near future.

“It’s not the finished product, by any means,” Williams said. “You’ll see some more polish on it.”

The most-used feature in the stadium app, Williams said, continues to be the food and beverage features, which allow fans to either purchase concessions for express line pickup, or to have their orders delivered to their seats. Williams said the Niners delivered 2,100 food orders to fans Sunday, the most for the Niners so far.

Perhaps the best news for Williams was the lack of complaints about the wireless network, which the team had asked fans to tweet about if they were experiencing problems. Though some fans with older devices that only work on the 2.4 GHz wireless bands might not see the same speeds as those with newer devices (which use the more roomy 5 GHz bands), Williams said his team only got a couple complaints about network issues, and one of those was solved before they could respond.

“Overall it just was a really good [wireless] experience,” said Williams, who always ends by noting that networks are never completely finished products. But with its Super Bowl-beating performance Sunday, the Levi’s Stadium network appears in midseason form.

“I think we’re close,” Williams said.

View from the north porch

View from the north porch

Instant replay feature debuts as network rocks at Niners’ home opener at Levi’s Stadium

Instant replay feature on Levi's Stadium app. Credit all photos: Paul Kapustka, MSR

Instant replay feature on Levi’s Stadium app. Credit all photos: Paul Kapustka, MSR

Niners fans may not have liked the final outcome on the scoreboard, but there was no complaining about the wireless network performance for the regular season home opener at Levi’s Stadium.

The somewhat limited debut of the highly anticipated instant replay feature for the Levi’s Stadium app was perhaps the technology highlight, with plays available for viewing seconds after they happened — when the feature finally got going later in the game. Our unofficial speedtests from various points around the stadium showed the Wi-Fi network and DAS network in top form, with solid results in the 20-plus Mbps range at most places for both networks.

We’re still waiting for the official postmortem results stats from the 49ers’ tech crew, but we did get some in-game messages from Niners VP of technology Dan Williams, who said that right before the 5:30 p.m. local time kickoff the usage peak was hit, with more than 19,000 simultaneous users on the Wi-Fi network, using throughput of 3.1 GB per second. Our unofficial guess is that the Niners’ opener may have set new single-game records for data traffic, but we’ll wait until we hear the final totals before we make any such proclamation.

The good news for Niners fans is that everywhere we went in the stadium, including on the top-level cheap-seat decks, the network was strong.

Replay feature sees limited action

If there was any tech downside, it was the limited availability of the instant replay feature. When talking about this feature earlier this year, team officials were adamant that it would be an unbelievable thing, with multiple camera angles and multiple choices of replays to watch. In reality, the feature wasn’t even available for most of the first half, and then when it did start working in the second half it only offered replays of the last two live plays, and two scoring highlights, one of which did not function at first.

Picture of app late in the first half.

Picture of app late in the first half.

Another beef we had with the replay feature was that it wasn’t clearly located on the “Game Center” part of the app; you had to know to click on the down arrow of the screen for the last play replay to appear. We found it only by accident, and would guess that not many fans found it or used it during the Niners’ loss to the Chicago Bears.

When we did watch the replays, however, we were simply amazed by the system’s performance — instant replays were available just seconds after a play had taken place, with a text play by play description as well. But again, there were no options for multiple camera angles, and there were only two “scoring play” highlights available when we last checked, and one (of an early field goal) didn’t have working video.

Other impressions from our stadium visit (which was made possible thanks to a press pass given to MSR by the Niners PR team):

— Concourses are fun places to watch: We watched the national anthem and opening kickoff and drive from the southwest concourse, one of the many places in Levi’s Stadium where you can stand and still see live game action. It was a packed house for the kickoff and first series, with many fans snapping phone pictures of the nearby field. East concourses were also good (though in one section an usher shooed us away from standing two-deep) as long as you kept the setting sun out of your eyes.

Fans take pictures of opening kickoff from southwest concourse.

Fans take pictures of opening kickoff from southwest concourse.

— Food runners were tired on the top deck: We stopped by for a quick chat with some of the food runners on the Pepsi deck on the north end of the stadium, and they looked pretty tired. Again, no official stats here but one runner said he’d been kept busy all game bringing food and drinks to fans who placed their orders via the app. “It’s pretty steep up here,” he said, pointing to the 400-level seating sections. “You get tired going up and down those steps.”

View from the Pepsi seating porch at the north end of Levi's Stadium

View from the Pepsi seating porch at the north end of Levi’s Stadium

— Clubs are the place to be: Several stories about the game noted that club-level seating was often empty, with fans perhaps spending time inside the comfortable beverage/food/gathering areas. Here’s a pregame look inside the United Club, located on the third level of the main building on the stadium’s west side.

United Club during pregame

United Club during pregame

— Signal strength: We also made several phone calls and did speedtests during halftime, when Snoop Dogg surprised us all with a mini-concert on the field. Not once during the game did we see any slowdowns in the network performance; the slowest Wi-Fi speed we found was 11 Mbps on the north concourse, where the team has admitted it didn’t put a lot of antennas (and is working to correct that).

Snoop on da big screen

Snoop on da big screen

— Light rail works fine: Then it was time to go home, and our choice to use the VTA light rail from Mountain View (instead of our press parking pass) was validated, as it took us less than an hour from the start of the line to the end of the line in Mountain View. All in all, an extremely solid home opener for one of the most ambitious stadium technology deployments out there.