Stadium Tech Report: THE COLLEGE FOOTBALL ISSUE looks at university Wi-Fi deployments

collegethumbIf there was a college football playoff for stadium wireless network deployments, which four teams would be in? Electing myself to the committee, I think my top picks would be the same venues we’re profiling in our latest Stadium Tech Report – Baylor, Nebraska, Stanford and Texas A&M. All four are pursuing high-end networks to support a better fan experience, leading the way for what may turn out to be the largest “vertical” market in the stadium networking field – sporting venues at institutions of higher learning.

To be sure, network deployments at major universities in the U.S. are still at the earliest stages — in our reporting for our latest long-form report, we found that at two of the top conferences, the SEC and the Pac-12, only four schools total (two in each conference) had fan-facing Wi-Fi, with only one more planned to come online next year. Why is the collegiate market so far behind the pro market when it comes to network deployment? There are several main reasons, but mostly it comes down to money and mindset, with a lack of either keeping schools on the sidelines.

Leaders look for NFL-type experiences

But at our “playoff” schools, it’s clear that with some ready budget and a clear perspective, college stadiums don’t need to take a back seat to anyone, pro stadiums included. The networks, apps and infrastructure deployed for this season at Baylor’s McLane Stadium and Nebraska’s Memorial Stadium are among the tops anywhere in sports, and the all-fiber infrastructure being put in place at Texas A&M should make that school’s Kyle Field among the most-connected if all work gets completed on time for next football season. Read in-depth profiles on these schools’ deployments, along with team-by-team capsule technology descriptions and an exclusive interview with Mississippi State athletic director Scott Stricklin in our latest report, available for free download from our site.

We’d like to take a second here to thank our sponsors, without whom we wouldn’t be able to offer these comprehensive reports to you free of charge. For our fourth-quarter report our sponsors include Crown Castle, SOLiD, Extreme Networks, Aruba Networks, TE Connectivity, and Corning.

Niners’ CEO gives Levi’s Stadium operations a ‘B’ grade

Jed York, Niners CEO, speaking at tech summit at Levi's Stadium. Credit all photos: Paul Kapustka, MSR

Jed York, Niners CEO, speaking at tech summit at Levi’s Stadium. Credit all photos: Paul Kapustka, MSR

San Francisco 49ers CEO Jed York, who caused a stir last week by apologizing for his team’s play on Twitter, on Thursday gave the operation of his club’s new Levi’s Stadium an above-average grade of “B” for the first four months of its inaugural season.

Speaking at a technology-focused fan-experience and innovation summit held Thursday at the stadium’s “501” club, York said “I’d give us a ‘B’ on our execution [at Levi’s Stadium] this year,” citing parking issues and foot-traffic flow into the stadium as problems not yet fully solved for the 68,500-seat facility in Santa Clara, Calif.

Given the new stadium’s complicated location — in the middle of a busy corporate-headquarters area and right next to the Great America theme park — it was probably somewhat of a given that there would be parking struggles the first season, as fans, police, traffic directors and stadium workers all figured out how to make the dance work. Though some progress has been made during the season, York said that “parking and just getting people here” remain the biggest issue he sees at Levi’s Stadium.

While the Niners have tried to use technology to solve the problems of getting fans inside (with app-based parking maps and wayfinding), York said that despite plans to funnel fans through the correct gates to get more quickly to their seats, many fans still just head for the gate that’s closest to their parking or train arrival spot, which has sometimes led to big backups at the check-in lines.

On the networking side of things, York said all seemed to be going well with the stadium’s Wi-Fi and cellular networks, which have performed well in a series of ad hoc tests conducted by MSR in visits this season.

“We haven’t had any [network] glitches, but we’ve been doing a lot of tweaking,” York said. The Levi’s Stadium app has been the center of most of that tweaking, with continual upgrades to add features and fix problems like an Android bug that surfaced in a mid-November revision. York said that the Niners have been careful to make sure the human engineering behind the technology is solid before launching new things, like having enough runners to support the feature that allows food to be ordered and delivered to seats.

Levi's Stadium ready for the Pac-12 championship game

Levi’s Stadium ready for the Pac-12 championship game

And though it hasn’t been officially confirmed yet, York said the team might debut a planned feature of having team merchandise available for purchase and delivery to seats at the next home game, Dec. 20 vs. San Diego. John Paul, CEO of Levi’s Stadium app developer VenueNext, also spoke at the summit Thursday and said in an interview that the Dec. 20 game might also see the debut of a new feature that adds in mass transit and Uber wait times. (A good idea for fans going to the stadium is to check on game day morning for any updates to the app.)

On the app side, York said one surprise was the fairly low uptake of fans using the app’s instant replay features. After fans watched 7,800 replays during the regular-season home opener (the first game replays were available in the app), usage has gone down, with less than 4,000 replays watched during the last home game against Seattle.

“We thought that mobile replays would be absolutely a home run feature, but it hasn’t got that much traction,” said York. The culprit, he said, might be the twin HD displays above each end zone at Levi’s Stadium, which are somewhat stunning in their clarity. “We do have great video boards,” York said.

In his talk York also thought out loud about the possibility of using wearable technology to both better help prevent player injuries as well as being able to provide more rich detail for fans, like how hard a hit was on the field. But he also stressed that technology at Levi’s Stadium was not meant to be used for technology’s sake, but instead to improve the experience of being at the game. And improvement on the stadium staff’s execution, York said, will go a long way to making it all work together.

“When we get to an A, or A+, people will really be blown away,” York said.

Levi's Stadium at twilight

Levi’s Stadium at twilight

Levi’s Stadium update: Wi-Fi traffic steady at 2.3+ TB per game, Amazon and Google lead apps use

Scoreboard promo for the Levi's Wi-Fi network

Scoreboard promo for the Levi’s Wi-Fi network

Usage of the Wi-Fi network at the San Francisco 49ers’ new Levi’s Stadium continues to hold steady at 2.3-plus Terabytes of data offloaded per game, according to the most recent statistics provided by the Niners’ network staff.

Though he’s no longer the team’s vice president of technology, former Levi’s network guru Dan Williams remains as a game-day consultant to the team, and he shared some recent network statistics with MSR, including some app usage marks that show Amazon Cloud Drive and Google APIs being among the top two applications being used over the Levi’s Stadium Wi-Fi network.

According to Williams’ numbers, for the Niners’ Nov. 23 day home game against Washington, the Levi’s Stadium Wi-Fi network had 22,095 unique users, 35 percent of the total attendance; the peak number of concurrent users was 14,700, reached at 2:50 p.m. (near halftime), and the total data used was 2.31 TB, between the hours of 9 a.m. and 5 p.m.

During the Thanksgiving day game against the Seahawks, the network had 23,371 users (36 percent of attendees) who used 2.38 TB of data, with a peak of 16,800 concurrent users at 7:15 p.m.

Video-replay statistics from the Levi’s Stadium app apparently do better when there are more home-team highlights to watch. During the victory over Washington, 1,074 fans watched 4,885 replays, with the 30-yard TD pass from Colin Kaepernick to Anquan Boldin racking up 953 total views from 398 unique users — meaning many users are watching the same replay more than once (maybe showing it to people sitting around them?). For the Seahawks game, 1,407 fans watched 3,875 replays, with a Kaepernick pass to Michael Crabtree accounting for the most views, 487 total from 180 unique users.

The top 4 apps used by fans on the network varied a bit from the two close games, but according to Williams Amazon Cloud Drive was the top app for both recent games. For the Seahawks game, the next three top apps were Google APIs, Facebook and SnapChat, while for the Washington game the next three top apps used were stadium video, Google APIs and Apple iTunes.

Stadium Tech Report: Arizona Cardinals get stadium ready for Super Bowl with Wi-Fi upgrade

University of Phoenix Stadium. Credit all photos: Arizona Cardinals.

University of Phoenix Stadium. Credit all photos: Arizona Cardinals. (click on photos for larger image)

As they get set to host their second Super Bowl this February, the IT team at the University of Phoenix Stadium in Glendale, Ariz., knows now what they didn’t know then: The big game requires a big wireless network. Bigger than you think.

“It’s funny to look back now on that first Super Bowl,” said Mark Feller, vice president of information technology for the Arizona Cardinals, speaking of Roman numeral game XLII, held in the barely 2-year-old facility on Feb. 3, 2008. With a couple Fiesta Bowls and one BCS championship game (2007) under his belt in a facility that opened with Wi-Fi and DAS, Feller said he and his team “thought we had a good handle” on what kind of network was required for a Super Bowl crowd.

The NFL, he said, begged to differ. Those college games might have been big, but the Super Bowl was bigger.

“We had the Fiesta Bowl that year at night, and when the game was over there were people from the NFL there wanting to know when they could set up,” said Feller in a recent phone interview. “This year, we’re much better prepared. We know what the water temperature is this time.”

Rip and replace, with more and better gear

Wi-Fi railing antennas

Wi-Fi railing antennas

For Super Bowl XLIX, scheduled to take place in Glendale on Feb. 1, 2015, Feller and his team have not just tuned up their network — they have done a full rip and replace of the Wi-Fi system, installing new Cisco gear from back end to front, in order to support a wireless game-day demand that is historically second to none. Integrator CDW has led the Wi-Fi effort and Daktronics and ProSound did the installation of new video screens, and neutral host Crown Castle has overseen a revamp of the DAS system, again with more antennas added to bolster coverage. In all, there has been more than $8 million in wireless improvements before this year, Feller said, as well as another $10 million for two new video boards that are each three times larger than what was there before.

“The last three or four years there have been things we knew we needed to improve [before the Super Bowl],” Feller said. After extensive work with the NFL’s technical team — this time well before the Fiesta Bowl — Feller oversaw a “top to bottom” refurbishment that included replacing core Cisco networking gear with newer gear, and new and more Wi-Fi access points that now total somewhere north of the 750 mark, with some more to be added before the big game. The new network, which was in place for the start of the current NFL season, has undergone testing by CDW at each home game, Feller said. CDW also plans to expand the network outside the stadium before the Super Bowl, in part to handle the extra events that take place not just on game day but in the days leading up to the game.

“The plan is to install more [coverage] outside, in the plaza areas,” Feller said.

When it opened in 2006, the $455 million University of Phoenix Stadium was one of the first with full-bowl Wi-Fi, using Cisco gear from the inside out. “Cisco was in here before they called it [their solution] ‘connected stadium’,” Feller said. From core network switches to firewalls to edge switches, this year there is all new Cisco gear in the venue, as well as new 3700 series APs, with panel antennas and antennas in handrails.

“Handrail [antennas] are sometimes a bit of a challenge, because you need to drill through concrete that’s 40 feet up in the air, behind another ceiling,” said Feller, describing one particular design challenge. Another one was mounting antennas on drop rods from the catwalks below the stadium’s retractable roof, to serve the upper-area seating. There are also some new Wi-Fi APs on the front row of the seating bowl, pointing up into the crowd.

“It was a fun project,” Feller said.

Stadium with roof open

Stadium with roof open

All on board for the DAS

The upgrade for the stadium’s DAS, led by Crown Castle, was just finished a few weeks ago, Feller said, and included more coverage outside the stadium as well, with antennas placed on light poles and on the stadium’s shell.

“Crown Castle did a great job of managing the carriers” on what is a 48-sector DAS, Feller said. “It [the upgrade] really required a lot of creative thinking from their engineers.”

Since the stadium was originally designed with wireless in mind, Feller and his team didn’t need to build new head end room for the DAS upgrades. “But I wouldn’t say we have plenty of space left,” he added. “We’ve got a lot of new equipment.”

Though all the major carriers are expected to be on the DAS by the big game, league partner Verizon Wireless should have some special projects up its sleeve for the big game, including another demonstration of its LTE Broadcast technology, which optimizes things like live video over LTE cellular links.

New Cardinals app a preview of Super Bowl version?

The Cardinals also had a new version of the game-day team app for this season, built by stadium-app leader YinzCam. According to Feller the new app supports three different live video feeds, as well as instant replays.

Wi-Fi antenna on railing

Wi-Fi antenna on railing

“It’s really cool to have that ability to watch things like a touchdown pass at the end of the game,” Feller said. And while no details have yet been revealed, in an interview with NFL CIO Michelle McKenna-Doyle earlier this year MSR learned that the league and YinzCam are working on a Super Bowl app with its own new bells and whistles. (Stay tuned for more info on the Super Bowl app.)

In addition to two more regular-season home games in December, the University of Phoenix Stadium will have at least a couple more dry runs to help test the network, during the Dec. 31 Fiesta Bowl and during the NFL’s Pro Bowl, scheduled for Jan. 25. And though the Cardnials lost to the Atlanta Falcons Sunday, at 9-3 they are still tied with the Green Bay Packers for the best record in the NFC, something that has the Phoenix faithful optimistic about the postseason.

“We’re going to get some more test runs, on New Year’s Eve and during the Pro Bowl,” Feller said. “And maybe some home playoff games as well!”

(more photos below)

Wi-Fi antenna in roof rafters

Wi-Fi antenna in roof rafters

More antennas in rafters

More antennas in rafters

Wi-Fin antenna under overhang

Wi-Fi antenna under overhang

Big AT&T DAS weekend in Miami: 2.7 TB of traffic for two mid-November games

Screen Shot 2014-09-12 at 2.21.51 PMWe’re a couple weeks behind in catching up here, but it’s worth backtracking to look at a huge weekend of DAS traffic at Miami’s Sun Life Stadium that took place earlier this month. According to DAS traffic figures from AT&T, the two games held at Sun Life on Nov. 13 (Miami Dolphins vs. Buffalo Bills) and Nov. 15 (Florida State vs. Miami) generated a total of 2.735 terabytes of traffic on the AT&T-specific cellular DAS in the stadium — a pretty high mark for cellular-only traffic.

Since we know there’s also a high-capacity Wi-Fi network at Sun Life, it’s interesting to wonder how much total traffic there was for the two events. While we wait to see if the fine folks who run the stadium network will eventually provide us with the Wi-Fi details, we can drill down a bit more into the DAS numbers that AT&T is seeing across the largest stadiums this fall.

The two games in Miami that weekend were the tops for DAS traffic in both college and pro for AT&T networks, which according to AT&T is the first time one town has held the DAS crown for both spots. The FSU-Miami game, where the Hurricanes kept it close to the end, was the biggest single DAS traffic event of that weekend, college or pro, with 1,802 GB of data crossing the AT&T DAS network. What’s kind of stunning is to remember that these stats are for AT&T customer traffic only; full game traffic from the 76,530 in attendance at the FSU-Miami game was likely much higher but alas — we get no such comparable stats from other cellular providers.

Other big games between highly ranked teams also scored high in AT&T’s DAS rankings that particular weekend — Alabama’s home win over then No. 1 Mississippi State was second on the list with 849 GB of DAS traffic, while Georgia’s win over visiting Auburn that Saturday recorded 676 GB of DAS traffic.

On the pro side, the second-highest AT&T DAS traffic came interestingly from San Diego, where the Chargers eked out a 13-6 win over the Raiders. We’re wondering if the DAS mark from San Diego — 730 GB, which trailed only Miami’s Thursday night mark of 933 GB in its win over Buffalo — was higher because Qualcomm Stadium still doesn’t have Wi-Fi. And again, remember that traffic at some other stadiums might have been higher — these numbers reflect only AT&T stats from venues where AT&T has an operating DAS.

Stay tuned as the football seasons come to their conclusions — with any luck we’ll get some more DAS and Wi-Fi stats to get a more complete picture of stadium traffic this season, which — surprise! — seems to be continually growing. Verizon, Sprint and T-Mobile… lend us your stats!

Levi’s Stadium ‘NiNerds’ get high-visibility wardrobe upgrade

NiNerd sporting the new neon vest. (Click on any photo for a larger image) Credit all photos: Paul Kapustka, MSR

NiNerd sporting the new neon vest. (Click on any photo for a larger image) Credit all photos: Paul Kapustka, MSR

I needed a one of the NiNerds Sunday at Levi’s Stadium, and thanks to a new wardrobe addition, they were a lot easier to find.

Discovering a problem with the Levi’s Stadium app, I looked around for one of the stadium’s walk-around technology helpers — aka the “NiNerds” — and found one quickly thanks to the new neon-yellow vests many were wearing during Sunday’s game between the San Francisco 49ers and Washington.

Earlier this season, the NiNerds were much more nattily dressed in their gingham-check shirts, bow ties and fake horn-rim glasses. While cool and cute, the outfits proved hard to recognize in the crowd, especially since the NiNerds’ red check shirts looked a lot like the jerseys and t-shirts worn by the many faithful fans. Perhaps in order to make the NiNerds stand out more, the team dressed them in neon Sunday, like Wi-Fi “coaches” in other stadiums have done.

Unfortunately, the NiNerd I talked to wasn’t able to solve my problem (it seems to be related to a known bug in the newest Android release of the stadium app) but I did notice during my visit Sunday that the NiNerds in general seemed to be more numerous and visible, and they even got a nice shout-out on the Levi’s Stadium big screen (see photos below). Below are some thoughts and observations on the network performance, the app performance and the overall fan experience at Levi’s, which I hadn’t visited since the season opener back on Sept. 14.

Wi-Fi network struggles at 2.4 GHz, soars at 5 GHz

Speed test results from outer concourse location, Levi's Stadium, pregame

Speed test results from outer concourse location, Levi’s Stadium, pregame

On the network-performance side of things, the Wi-Fi system seemed as robust as ever for new devices, including the AT&T LG Optimus G Pro I’ve been test-driving lately. With the Android device and its 5 GHz Wi-Fi connection I hit speeds of 31 Mbps download and 29 upload before the game on the Levi’s Stadium outside concourse, and then had a 14 Mbps download connection in my seat in section 229 (south end zone) at kickoff. In the third quarter I wandered up to the top (7th) level of seats, and got a 28/33 Mbps reading while waiting in a concession-stand line.

With my older Verizon Droid 4 device, however, I struggled to connect to the Wi-Fi network. Since the phone only runs on the 2.4 GHz Wi-Fi frequency, it doesn’t do well at Levi’s Stadium, where the Wi-Fi is more heavily tuned for newer, 5 GHz-capable devices.

Full charging station... before the game starts

Full charging station… before the game starts

The best Wi-Fi speedtest I could get with the Droid device was a 1.06 Mbps download/3.04 Mbps upload mark, from the same spot on the outer concourse where the newer device recorded blazing connectivity speeds. Switching it over to its Verizon 4G LTE radio, I was able to get much faster connectivity, including one mark of 21.60/9.58 on the main level inside concourse.

I also ran out of juice on the Verizon phone before the end of the game — which could have been either the device draining due to its inability to get a solid connection, or due to the fact that it’s getting old and the battery doesn’t hold a charge that well anymore. Judging from the crowds around the Levi’s Stadium recharging stations (the picture to the left was taken during pregame) I am not alone in my device-energy woes.

App problem derails beacon test

One of the main tasks I had planned for Sunday was to see how well the beacon-assisted wayfinding feature in the stadium app worked. Only problem was, in the new update of the Levi’s Stadium app that was released in the past week (which I downloaded to both devices Sunday morning), several features were missing, including the “Maps” feature.

Picture of app fail in Levi's Stadium Android app

Picture of app fail in Levi’s Stadium Android app

A NiNerd I talked to outside the stadium on the Faithful Mile area showed me the maps/wayfinding feature on his iPhone, and pulled up a GPS-supported direction message that pointed the correct way for me to enter the stadium. But neither he nor the NiNerd I talked to inside the stadium could figure out why both of my Android devices weren’t showing the maps feature, or several other features on the left-upper-corner pull-down menu.

According to Louise Callagy, vice president of marketing for app developer VenueNext, the new version of the app released this week did have a known Android bug. In an email response Sunday night Callagy said, “we know we have a bug where Android gets confused and won’t return results from the network,” adding that rebooting the device might have fixed the problem; however, I did reboot both devices during the game and the problems were not corrected.

Callagy said the Levi’s network also had issues Sunday in getting location information from the beacons. “Our plan is to re-write the code [for the app] and solve this issue, releasing a new version before the Seahawks game on Thursday,” Callagy added in her email.

On the good-news side the replay function of the Levi’s Stadium app was more impressive than earlier versions, with highlights appearing in the app in mere seconds after the original play had concluded. After the Niners’ first TD pass of the day, I was able to view the highlight of the Colin Kaepernick-to-Anquan Boldin pass just after the extra point had been kicked.

I was also able to see the red light/green light system for restroom wait times that drew so much attention when it was talked about earlier this year. However, in real-life practice it’s doubtful anyone thinks of looking at the app when it’s time to go. (It’s also quite likely that while you are looking at the app for a short restroom line, a natural break in action will occur and restroom lines will predictably lengthen everywhere.) I found a quick trick for Levi’s attendees that might pay off in the future: If the restroom you’re aiming toward has a long line, walk a small bit farther to find its back door — where there is often no line at all.

New version of app, with clearer icons on main screen

New version of app, with clearer icons on main screen

I messed up later in the game, however, in thinking that it would just be easier to find a concession stand than to use the app’s express window ordering function. At least the 10 minutes I spent in line behind three women who were apparently ordering for their entire row (five hotdogs, six orders of fries, two orders of wings, two beers and one large ice water) gave me time to conduct a couple more Wi-Fi speedtests. Next time, I’m using the food-ordering features on the app.

I also made great use of the app’s ability to let fans watch live game action (I chose the feed from the main video screens in the stadium). Since I had to leave early I was on the first VTA train when I saw the game-winning TD run courtesy of the app’s live action broadcast. The live video, incidentally, kept playing seamlessly over the AT&T 4G LTE network as I sped away from the stadium, allowing me to watch the final game-sealing sack as I beat most of the traffic home.

VTA trains a smooth ride, once you figure out how to get on

I also had another smooth ride to and from the stadium using the VTA light rail trains from Mountain View — once I was on the train it took just a little over 30 stress-free minutes both coming and going. Getting on the trains, however, is a process that could still use some work. The Mountain View station, which is logistically hampered by having to share space with the Caltrain tracks and station, has very little signage on game day, and has a lot of confusing temporary gates to try to flow foot traffic toward the ticket-verification checkers.

Packed VTA train en route to Levi's Stadium

Packed VTA train en route to Levi’s Stadium

Once I figured out the maze and was guided across the Caltrain tracks I was directed to one of two waiting trains — but then a VTA staffer looked into the train and told people there were also express buses that wouldn’t stop on the way to Levi’s (unlike the trains, which stop at numerous stations en route). The quizzical advice — nobody said if it was any faster to take the buses — had many people wondering what they were supposed to do, causing a delay in closing the train doors as people made up their minds without any more information.

Once we arrived at the stadium, on the exit platform there was no person or sign directing fans in the proper direction. Good thing for the many newbies on the train (the train 2 hours before kickoff was packed) several of us were veterans and directed everyone down the proper ramp. For the return trip the Mountain View line suffered from similar lack of information and signing — and after one train passed the station with signs that said “Not in Service” we got on a second train that also said “Not in service” but whose doors opened anyway and one person in a yellow vest told everyone, “get inside.”

Overall impressions: Levi’s experience and technology still a work in progress

While I continue to be impressed by the network and app performance at Levi’s Stadium, I also felt several times on Sunday like the technology, the stadium and the entire fan experience is still a work in progress — perhaps something to be expected for a venue in its first year of events. But I have to wonder a bit about releasing a new version of the app in midseason, without apparently testing it enough to make sure it worked well on all devices that might want to use it.

I’m also still skeptical on how well the wayfinding feature will work in real world situations; though it sounds great to be able to get GPS-like directions to places inside the stadium, the reality of trying to walk around looking down at your phone on one of Levi’s Stadium crowded concourses is more likely to lead you into someone’s backside. Anyone with tales to tell of Levi’s Stadium technology experiences, please chime in below in the comments or send me an email to kaps at mobilesportsreport.com. I’d be especially interested to know if anyone else saw my app problems Sunday on Android phones. More Levi’s pictures below.

A NiNerd (no vest) helps fans outside the stadium.

A NiNerd (no vest) helps fans outside the stadium.

Kickoff view from Section 229. Thanks to the Niners for the free media access.

Kickoff view from Section 229. Thanks to the Niners for the free media access.

Niners fans get their phone cameras busy for kickoff ceremonies.

Niners fans get their phone cameras busy for kickoff ceremonies.

Scoreboard plug for the app.

Scoreboard plug for the app.

Scoreboard promo for the NiNerds (one in a series)

Scoreboard promo for the NiNerds (one in a series)

Second in the series. This one got laughs from the crowd.

Second in the series. This one got laughs from the crowd.

Probably the first time many fans heard the term "NiNerds"

Probably the first time many fans heard the term “NiNerds”

Nothing says geek like a bow tie

Nothing says geek like a bow tie