Niners’ CEO gives Levi’s Stadium operations a ‘B’ grade

Jed York, Niners CEO, speaking at tech summit at Levi's Stadium. Credit all photos: Paul Kapustka, MSR

Jed York, Niners CEO, speaking at tech summit at Levi’s Stadium. Credit all photos: Paul Kapustka, MSR

San Francisco 49ers CEO Jed York, who caused a stir last week by apologizing for his team’s play on Twitter, on Thursday gave the operation of his club’s new Levi’s Stadium an above-average grade of “B” for the first four months of its inaugural season.

Speaking at a technology-focused fan-experience and innovation summit held Thursday at the stadium’s “501” club, York said “I’d give us a ‘B’ on our execution [at Levi’s Stadium] this year,” citing parking issues and foot-traffic flow into the stadium as problems not yet fully solved for the 68,500-seat facility in Santa Clara, Calif.

Given the new stadium’s complicated location — in the middle of a busy corporate-headquarters area and right next to the Great America theme park — it was probably somewhat of a given that there would be parking struggles the first season, as fans, police, traffic directors and stadium workers all figured out how to make the dance work. Though some progress has been made during the season, York said that “parking and just getting people here” remain the biggest issue he sees at Levi’s Stadium.

While the Niners have tried to use technology to solve the problems of getting fans inside (with app-based parking maps and wayfinding), York said that despite plans to funnel fans through the correct gates to get more quickly to their seats, many fans still just head for the gate that’s closest to their parking or train arrival spot, which has sometimes led to big backups at the check-in lines.

On the networking side of things, York said all seemed to be going well with the stadium’s Wi-Fi and cellular networks, which have performed well in a series of ad hoc tests conducted by MSR in visits this season.

“We haven’t had any [network] glitches, but we’ve been doing a lot of tweaking,” York said. The Levi’s Stadium app has been the center of most of that tweaking, with continual upgrades to add features and fix problems like an Android bug that surfaced in a mid-November revision. York said that the Niners have been careful to make sure the human engineering behind the technology is solid before launching new things, like having enough runners to support the feature that allows food to be ordered and delivered to seats.

Levi's Stadium ready for the Pac-12 championship game

Levi’s Stadium ready for the Pac-12 championship game

And though it hasn’t been officially confirmed yet, York said the team might debut a planned feature of having team merchandise available for purchase and delivery to seats at the next home game, Dec. 20 vs. San Diego. John Paul, CEO of Levi’s Stadium app developer VenueNext, also spoke at the summit Thursday and said in an interview that the Dec. 20 game might also see the debut of a new feature that adds in mass transit and Uber wait times. (A good idea for fans going to the stadium is to check on game day morning for any updates to the app.)

On the app side, York said one surprise was the fairly low uptake of fans using the app’s instant replay features. After fans watched 7,800 replays during the regular-season home opener (the first game replays were available in the app), usage has gone down, with less than 4,000 replays watched during the last home game against Seattle.

“We thought that mobile replays would be absolutely a home run feature, but it hasn’t got that much traction,” said York. The culprit, he said, might be the twin HD displays above each end zone at Levi’s Stadium, which are somewhat stunning in their clarity. “We do have great video boards,” York said.

In his talk York also thought out loud about the possibility of using wearable technology to both better help prevent player injuries as well as being able to provide more rich detail for fans, like how hard a hit was on the field. But he also stressed that technology at Levi’s Stadium was not meant to be used for technology’s sake, but instead to improve the experience of being at the game. And improvement on the stadium staff’s execution, York said, will go a long way to making it all work together.

“When we get to an A, or A+, people will really be blown away,” York said.

Levi's Stadium at twilight

Levi’s Stadium at twilight

Levi’s Stadium update: Wi-Fi traffic steady at 2.3+ TB per game, Amazon and Google lead apps use

Scoreboard promo for the Levi's Wi-Fi network

Scoreboard promo for the Levi’s Wi-Fi network

Usage of the Wi-Fi network at the San Francisco 49ers’ new Levi’s Stadium continues to hold steady at 2.3-plus Terabytes of data offloaded per game, according to the most recent statistics provided by the Niners’ network staff.

Though he’s no longer the team’s vice president of technology, former Levi’s network guru Dan Williams remains as a game-day consultant to the team, and he shared some recent network statistics with MSR, including some app usage marks that show Amazon Cloud Drive and Google APIs being among the top two applications being used over the Levi’s Stadium Wi-Fi network.

According to Williams’ numbers, for the Niners’ Nov. 23 day home game against Washington, the Levi’s Stadium Wi-Fi network had 22,095 unique users, 35 percent of the total attendance; the peak number of concurrent users was 14,700, reached at 2:50 p.m. (near halftime), and the total data used was 2.31 TB, between the hours of 9 a.m. and 5 p.m.

During the Thanksgiving day game against the Seahawks, the network had 23,371 users (36 percent of attendees) who used 2.38 TB of data, with a peak of 16,800 concurrent users at 7:15 p.m.

Video-replay statistics from the Levi’s Stadium app apparently do better when there are more home-team highlights to watch. During the victory over Washington, 1,074 fans watched 4,885 replays, with the 30-yard TD pass from Colin Kaepernick to Anquan Boldin racking up 953 total views from 398 unique users — meaning many users are watching the same replay more than once (maybe showing it to people sitting around them?). For the Seahawks game, 1,407 fans watched 3,875 replays, with a Kaepernick pass to Michael Crabtree accounting for the most views, 487 total from 180 unique users.

The top 4 apps used by fans on the network varied a bit from the two close games, but according to Williams Amazon Cloud Drive was the top app for both recent games. For the Seahawks game, the next three top apps were Google APIs, Facebook and SnapChat, while for the Washington game the next three top apps used were stadium video, Google APIs and Apple iTunes.

YinzCam gains huge access to college market with Learfield partnership deal

Sports stadium app developer YinzCam, the Pittsburgh-based concern that has numerous app development wins across the NFL, NBA and NHL, scored a big entree to the collegiate market with a deal that makes YinzCam the “preferred supplier” of mobile apps for Learfield Sports, the giant marketing firm that sets up media, sponsorship and promotional deals for many of the nation’s top universities.

In a press release announcing the deal, Learfield said YinzCam would become a preferred solutions provider for the nearly 100 colleges and conferences Learfield represents — a list that stretchs alphabetically from the Alabama Crimson Tide to the Xavier Musketeers — as well as for the 750 college programs that use Learfield subsidiary Sidearm Sports‘ digital production services for websites and other sports-related content.

While YinzCam has already scored several college-app development deals thanks to its NFL experience (like the recent Baylor University app and a forthcoming app for Texas A&M University), YinzCam founder and CEO Priya Narasimhan said the Learfield deal could provide a far more rapid expansion path into the university mobile-app market.

While we are working on a much deeper in-depth profile of YinzCam — coincidentally, we just finished a long (and long in planning) interview earlier this week with the always-busy Narasimhan, who provided some background information on how YinzCam has basically come from nowhere to leading the stadium mobile-app development field in its 5 years of existence. One thing we’ve always wondered about is how a 30-person company in Pittsburgh is able to develop apps for 20 NBA teams and 25 NFL teams, among its 91 customers. The secret? Automation.

According to Narasimhan, YinzCam is able to scale well because it automates most of its core app functionality — for instance, when it comes to making replays available to the mobile device, Narasimhan said a YinzCam algorithm breaks down the video feed, instead of having someone doing the task by hand. Narasimhan also said that the company uses all its clients as a kind of crowdsourcing idea pool, and quickly shares any new innovations with all existing clients. Narasimhan also said that YinzCam’s core app has “well documented third-party guidelines” for integrating outside feature-apps like ticketing and seat upgrade programs, which can then be melded into an overall YinzCam-produced team app.

One final YinzCam nugget for now:

— Where did the name come from? Narasimhan says YinzCam is a mashup of the Pittsburgh term “You ones” (a linguistic equivalent of the Southern “y’all”), which when pronounced quickly in a Pittsburgh accent sounds like “Yinz” and “camera” for the personal video the app supplies.

Stay tuned for more on YinzCam and the Learfield deal…

Stadium Tech Report: Arizona Cardinals get stadium ready for Super Bowl with Wi-Fi upgrade

University of Phoenix Stadium. Credit all photos: Arizona Cardinals.

University of Phoenix Stadium. Credit all photos: Arizona Cardinals. (click on photos for larger image)

As they get set to host their second Super Bowl this February, the IT team at the University of Phoenix Stadium in Glendale, Ariz., knows now what they didn’t know then: The big game requires a big wireless network. Bigger than you think.

“It’s funny to look back now on that first Super Bowl,” said Mark Feller, vice president of information technology for the Arizona Cardinals, speaking of Roman numeral game XLII, held in the barely 2-year-old facility on Feb. 3, 2008. With a couple Fiesta Bowls and one BCS championship game (2007) under his belt in a facility that opened with Wi-Fi and DAS, Feller said he and his team “thought we had a good handle” on what kind of network was required for a Super Bowl crowd.

The NFL, he said, begged to differ. Those college games might have been big, but the Super Bowl was bigger.

“We had the Fiesta Bowl that year at night, and when the game was over there were people from the NFL there wanting to know when they could set up,” said Feller in a recent phone interview. “This year, we’re much better prepared. We know what the water temperature is this time.”

Rip and replace, with more and better gear

Wi-Fi railing antennas

Wi-Fi railing antennas

For Super Bowl XLIX, scheduled to take place in Glendale on Feb. 1, 2015, Feller and his team have not just tuned up their network — they have done a full rip and replace of the Wi-Fi system, installing new Cisco gear from back end to front, in order to support a wireless game-day demand that is historically second to none. Integrator CDW has led the Wi-Fi effort and Daktronics and ProSound did the installation of new video screens, and neutral host Crown Castle has overseen a revamp of the DAS system, again with more antennas added to bolster coverage. In all, there has been more than $8 million in wireless improvements before this year, Feller said, as well as another $10 million for two new video boards that are each three times larger than what was there before.

“The last three or four years there have been things we knew we needed to improve [before the Super Bowl],” Feller said. After extensive work with the NFL’s technical team — this time well before the Fiesta Bowl — Feller oversaw a “top to bottom” refurbishment that included replacing core Cisco networking gear with newer gear, and new and more Wi-Fi access points that now total somewhere north of the 750 mark, with some more to be added before the big game. The new network, which was in place for the start of the current NFL season, has undergone testing by CDW at each home game, Feller said. CDW also plans to expand the network outside the stadium before the Super Bowl, in part to handle the extra events that take place not just on game day but in the days leading up to the game.

“The plan is to install more [coverage] outside, in the plaza areas,” Feller said.

When it opened in 2006, the $455 million University of Phoenix Stadium was one of the first with full-bowl Wi-Fi, using Cisco gear from the inside out. “Cisco was in here before they called it [their solution] ‘connected stadium’,” Feller said. From core network switches to firewalls to edge switches, this year there is all new Cisco gear in the venue, as well as new 3700 series APs, with panel antennas and antennas in handrails.

“Handrail [antennas] are sometimes a bit of a challenge, because you need to drill through concrete that’s 40 feet up in the air, behind another ceiling,” said Feller, describing one particular design challenge. Another one was mounting antennas on drop rods from the catwalks below the stadium’s retractable roof, to serve the upper-area seating. There are also some new Wi-Fi APs on the front row of the seating bowl, pointing up into the crowd.

“It was a fun project,” Feller said.

Stadium with roof open

Stadium with roof open

All on board for the DAS

The upgrade for the stadium’s DAS, led by Crown Castle, was just finished a few weeks ago, Feller said, and included more coverage outside the stadium as well, with antennas placed on light poles and on the stadium’s shell.

“Crown Castle did a great job of managing the carriers” on what is a 48-sector DAS, Feller said. “It [the upgrade] really required a lot of creative thinking from their engineers.”

Since the stadium was originally designed with wireless in mind, Feller and his team didn’t need to build new head end room for the DAS upgrades. “But I wouldn’t say we have plenty of space left,” he added. “We’ve got a lot of new equipment.”

Though all the major carriers are expected to be on the DAS by the big game, league partner Verizon Wireless should have some special projects up its sleeve for the big game, including another demonstration of its LTE Broadcast technology, which optimizes things like live video over LTE cellular links.

New Cardinals app a preview of Super Bowl version?

The Cardinals also had a new version of the game-day team app for this season, built by stadium-app leader YinzCam. According to Feller the new app supports three different live video feeds, as well as instant replays.

Wi-Fi antenna on railing

Wi-Fi antenna on railing

“It’s really cool to have that ability to watch things like a touchdown pass at the end of the game,” Feller said. And while no details have yet been revealed, in an interview with NFL CIO Michelle McKenna-Doyle earlier this year MSR learned that the league and YinzCam are working on a Super Bowl app with its own new bells and whistles. (Stay tuned for more info on the Super Bowl app.)

In addition to two more regular-season home games in December, the University of Phoenix Stadium will have at least a couple more dry runs to help test the network, during the Dec. 31 Fiesta Bowl and during the NFL’s Pro Bowl, scheduled for Jan. 25. And though the Cardnials lost to the Atlanta Falcons Sunday, at 9-3 they are still tied with the Green Bay Packers for the best record in the NFC, something that has the Phoenix faithful optimistic about the postseason.

“We’re going to get some more test runs, on New Year’s Eve and during the Pro Bowl,” Feller said. “And maybe some home playoff games as well!”

(more photos below)

Wi-Fi antenna in roof rafters

Wi-Fi antenna in roof rafters

More antennas in rafters

More antennas in rafters

Wi-Fin antenna under overhang

Wi-Fi antenna under overhang

Big AT&T DAS weekend in Miami: 2.7 TB of traffic for two mid-November games

Screen Shot 2014-09-12 at 2.21.51 PMWe’re a couple weeks behind in catching up here, but it’s worth backtracking to look at a huge weekend of DAS traffic at Miami’s Sun Life Stadium that took place earlier this month. According to DAS traffic figures from AT&T, the two games held at Sun Life on Nov. 13 (Miami Dolphins vs. Buffalo Bills) and Nov. 15 (Florida State vs. Miami) generated a total of 2.735 terabytes of traffic on the AT&T-specific cellular DAS in the stadium — a pretty high mark for cellular-only traffic.

Since we know there’s also a high-capacity Wi-Fi network at Sun Life, it’s interesting to wonder how much total traffic there was for the two events. While we wait to see if the fine folks who run the stadium network will eventually provide us with the Wi-Fi details, we can drill down a bit more into the DAS numbers that AT&T is seeing across the largest stadiums this fall.

The two games in Miami that weekend were the tops for DAS traffic in both college and pro for AT&T networks, which according to AT&T is the first time one town has held the DAS crown for both spots. The FSU-Miami game, where the Hurricanes kept it close to the end, was the biggest single DAS traffic event of that weekend, college or pro, with 1,802 GB of data crossing the AT&T DAS network. What’s kind of stunning is to remember that these stats are for AT&T customer traffic only; full game traffic from the 76,530 in attendance at the FSU-Miami game was likely much higher but alas — we get no such comparable stats from other cellular providers.

Other big games between highly ranked teams also scored high in AT&T’s DAS rankings that particular weekend — Alabama’s home win over then No. 1 Mississippi State was second on the list with 849 GB of DAS traffic, while Georgia’s win over visiting Auburn that Saturday recorded 676 GB of DAS traffic.

On the pro side, the second-highest AT&T DAS traffic came interestingly from San Diego, where the Chargers eked out a 13-6 win over the Raiders. We’re wondering if the DAS mark from San Diego — 730 GB, which trailed only Miami’s Thursday night mark of 933 GB in its win over Buffalo — was higher because Qualcomm Stadium still doesn’t have Wi-Fi. And again, remember that traffic at some other stadiums might have been higher — these numbers reflect only AT&T stats from venues where AT&T has an operating DAS.

Stay tuned as the football seasons come to their conclusions — with any luck we’ll get some more DAS and Wi-Fi stats to get a more complete picture of stadium traffic this season, which — surprise! — seems to be continually growing. Verizon, Sprint and T-Mobile… lend us your stats!

Niners’ VP of technology, Dan Williams, departs for Dropbox

Dan Williams talks Wi-Fi while the Levi's Stadium new turf grows silently behind him.

Dan Williams talks Wi-Fi while the Levi’s Stadium new turf grows silently behind him.

Dan Williams, the human face behind the highly touted wireless network at Levi’s Stadium, has left his position as vice president of technology for the San Francisco 49ers, according to both sources inside the San Francisco 49ers’ stadium network operations and Williams’ own LinkedIn profile.

Williams, who was at his post as of at least last week, is now at online storage concern Dropbox, where his LinkedIn profile has him with the title of “new guy, technical operations.” So far, there is no official statement from the 49ers about Williams’ departure, or about who would take over as head of operations for the Levi’s Stadium network.

Previously a network guru at Facebook, Williams became a the public face for the Niners’ new stadium and its heavily hyped wireless network and app strategy when Kunal Malik, the Niners’ former CTO, left the team earlier this year. The sudden departure is surprising since Williams — a professed avid Niners fan — seemed to enjoy the hands-on work of getting the Levi’s network up and running, even doing “seat calls” during games to help solve connectivity issues. Williams also took great pride in the network’s performance, sending out detailed statistics on bandwidth and app usage.

So far, there has been no public announcement of Williams’ departure. We’ll update this post when we hear more from Dan and/or the 49ers.