Verizon, NFL fumble opening-day live video for NFL Mobile app — for the 2nd year in a row

Screen shot of the message NFL Mobile users are getting used to seeing on opening day. Credit: Paul Kapustka, MSR.

Screen shot of the message NFL Mobile users are getting used to seeing on opening day. Credit: Paul Kapustka, MSR.

It’s not a streak you want to continue, especially if you are the NFL, Verizon Wireless, or one of the frustrated customers who weren’t able to see live video using Verizon’s NFL Mobile application during Sunday’s season-opening games. But just like last year, when activation servers for a new version of the app got overwhelmed, issues arose again on Sunday, making live video — one of the premium features of NFL Mobile that makes the app and service so attractive — unavailable for most of the day, including the large slate of early games.

Verizon executives who spoke with MSR Monday declined to comment on the NFL Mobile issues, instead referring all questions to their partner, the NFL media team. Verizon wireless support was also apparently offline on Sunday, leaving the task of responding to the many irate messages and tweets to the small but valiant NFL Mobile Support team.

Alex Riethmiller, vice president of communications for NFL Media, said the issue once again was due to high traffic overwhelming the authentication servers, which (putting it simply) identify whether or not a customer is a Verizon subscriber, and whether or not that customer has access to the premium content. Some of the additional traffic may have been due to two changes Verizon made in NFL Mobile subscriptions from last season; one was to give free premium access (which lets you see live action) to all customers on a More Everything data plan, and the other was to charge all of those More Everything viewers another $1.99 a month if they wanted to also watch the NFL’s RedZone channel, one of the more attractive features of the app (since RedZone gives you lots of live action, even on games that might otherwise be blacked or not viewable for other reasons). Verizon customers who are not More Everything customers must pay $5 a month for access to the NFL Mobile premium content.

Theoretically, you should be able to order RedZone from this "settings" page of the app. But it didn't work when we tried to click the button.

Theoretically, you should be able to order RedZone from this “settings” page of the app. But it didn’t work when we tried to click the button.

For many NFL Mobile customers, yours truly included, any live video at all was not available for long stretches of Sunday, and after a brief 2-minute “preview” period the RedZone content was also unavailable, and the app was not allowing it to be purchased. As of Monday afternoon, the MSR official testing lab still couldn’t get the RedZone box to work — clicking it kept taking us to a page explaining the charges but not offering a way to add it to our bill.

“NFL Mobile requires a number of backend systems to ensure only Verizon customers can get live NFL video,” said Riethmiller, in a prepared statement. “We experienced issues with one of the systems that validates customers, and it took longer than anticipated to resolve because of tremendous demand. We are confident we have addressed the issue going forward.”

Kristin Rooney, Verizon Wireless director of sponsorships and branded entertainment, would not comment on the NFL Mobile outages in a phone interview Monday morning. However, Rooney was happy to talk about the new features in NFL Mobile this season, including the ability to watch local games live, even if they are out of the traditional prime-time window NFL Mobile has access to. In addition to the local games, NFL Mobile premium customers are supposed to be able to watch live games on Sunday nights, Monday nights and Thursday nights, and also have access to RedZone broadcasts during the day. That is, when the servers are working.

BUY... MORE... SERVERS

BUY… MORE… SERVERS

Another unclear option is what happens to RedZone access when NFL Mobile customers visit NFL stadiums — while Verizon blocks access to RedZone at some stadiums, at others it allows it, usually through a partnership with a team’s stadium app. Rooney, however, did not have a list of stadiums where NFL Mobile customers could watch RedZone broadcasts during a game; most famously, the service was blocked at last year’s Super Bowl in MetLife Stadium.

Rooney also declined to say just how many customers Verizon has for its NFL Mobile app, though the Google Play store says the app has between 10 and 50 million installs. However, since a non-premium version of the app is available to customers with contracts from other carriers, it’s unclear how many “premium” members were overwhelming Verizon’s servers. It’s worth noting that Verizon is no stranger to large numbers of single-day activations, as its systems support more than 100 million wireless subscribers, nation-wide.

In a related note, NFL Mobile wasn’t the only streaming product having issues Sunday. According to a report at TVPredictions.com, the DirecTV streaming service for NFL games also suffered outages Sunday, also for the second straight year.

College DAS Update: AT&T and Verizon partner for DAS at Oregon; Minnesota taps AT&T for DAS at TCF Bank Stadium

Talk about an unlikely partnership: AT&T and Verizon Wireless announced that they have co-installed a DAS (distributed antenna system) deployment at the University of Oregon’s Autzen Stadium, a 200-antenna upgrade that should make cellular access a lot better for Ducks fans this football season.

There are few business competitions as balanced and as fierce as the one between the top two U.S. cellular carriers — so why are they doing the Kumbaya thing in Eugene? We haven’t heard any confirmation yet, but our guess is that it was somewhat of a forced partnership, due to an legacy deal between Oregon and Verizon, and the new Pac-12 infrastructure deal AT&T forged last year. (If either carrier would like to call in to confirm or deny, our lines are open.)

If you’re a follower of telecom business you might want to bookmark the Oregon press release announcing the deal and deployment, since it may be the first and last time you see Verizon and AT&T together in a single press release (that isn’t about keeping the government out of telecom regulation, that is). You can read between the lines when you see that AT&T is quoted first in the press release, Verizon second. Good news is, Ducks fans will have better cellular no matter which of the two major carriers they are a customer of.

We haven’t heard yet from AT&T about whether or not the conference is fully covered with DAS yet — according to this news report from last year there were only four Pac-12 schools whose DAS wasn’t up to snuff, with Autzen clearly being one of them. We are trying to get AT&T and Pac-12 folks on the line for a general update, so stay tuned.

AT&T bringing DAS to Minnesota’s TCF Bank Stadium

Moving to the other side of the Rose Bowl rivalry, AT&T announced it had installed a DAS at the University of Minnesota’s TCF Bank Stadium, a network of 339 antennas that will also be used by Minnesota Vikings fans since the NFL team is also playing there this year and next while their new stadium gets built.

According to AT&T, fans at TCF Bank Stadium used 106 GB of data on the new network during the Gophers’ Aug. 28 home opener against Eastern Illinois.

Niners: More than 1,000 fans used in-seat food delivery at 2nd Levi’s Stadium preseason game

Screen grab from Levi's Stadium app showing in-seat food delivery option. Credit all photos: Paul Kapustka, Mobile Sports Report.

Screen grab from Levi’s Stadium app showing in-seat food delivery option. Credit all photos: Paul Kapustka, Mobile Sports Report.

One of the more unique features the San Francisco 49ers are introducing at their new home this year is the ability for all fans to have food and drink delivered to them, no matter which one of Levi’s Stadium’s 68,500 seats they are in. And according to the Niners, the feature is quickly catching on, with more than 1,000 in-seat orders delivered at the second preseason game at Levi’s on Aug. 24.

At a Levi’s media technology tour Wednesday, the Niners’ tech staff provided an in-depth and up-close look at some of the new stadium’s network infrastructure, including a quick glimpse of one of the several data-center rooms. Dan Williams, vice president of technology for the Niners, said that at the first preseason game at Levi’s on Aug. 17, more fans used the “express pickup” service to order food that they could then pick up at concession stand windows than the seat-delivery feature. But at the second game against the San Diego Chargers, more fans went for the in-seat option, perhaps a sign that Niners fans are learning and testing the new stadium services as they go.

“It’s going to be an ongoing education process,” said Martin Manville, business operations analyst for the Niners and one of the key tech leaders on the Levi’s app team. Manville said the Niners had learned a lot about food delivery in test situations at Candlestick Park last season — and some of those lessons are now evident in the Levi’s delivery menu, which is stripped down to ensure the food runners can get grub to fans before it gets cold (or warm, in the case of cold beverages). According to Manville the average delivery time at the Aug. 24 game was between 10 and 12 minutes, but the team expects that “normal” delays during the regular season will be closer to 15 to 20 minutes per order.

Still, that’s not a bad option if you don’t want to leave your seat and you don’t mind the extra $5 delivery charge. One early interesting stat from the food-delivery process is that the Niners found more orders coming from the “cheap seats” in the north and south stands at Levi’s than from the 50-yard-line seats where the high rollers sit. According to Manville, since the fans in the club seats have easy access to numerous uncrowded concession stands they may not see the need for the in-seat options.

Wi-Fi APs: 600 in Levi’s bowl seating

Under-seat Wi-Fi AP at Levi's Stadium.

Under-seat Wi-Fi AP at Levi’s Stadium.

Some other news nuggets from the tech tour day: According to Williams, of the 1,200 Wi-Fi access points at Levi’s, 600 of those are distributed in the seating areas (aka “the bowl”), with the other 600 placed in suites, concourses and other stadium areas. Williams said the Aruba Networks Wi-Fi antennas are basically split into three types — regular enterprise-type APs for suite and office areas, regular outdoor APs for concourse areas, and more specialized versions (including the under-seat APs) for bowl placements.

— For the app itself, the Niners said that there have been 80,000 downloads so far, with almost half of the season ticket holders having put their ticketing information into the app. The “NiNerds,” the geek-dressed help squad that provides fans with personal assistance with the app, is now at about 50 or 60 strong at each event (originally the team had said it wanted to hire 150 such Wi-Fi coaches). The Niners said the NiNerds will be doing more pro-active app education going forward, doing things like approaching fans in concession lines to see if they know about the express line or in-seat ordering options.

— Though Comcast’s 10-year deal with the Niners calls for the cable provider to bring in two 10-gigabit backbone pipes, the Niners are often quoted as saying they have 40 GB of backbone bandwidth. We solved this mystery today: According to Comcast, the other two 10-GB pipes are a redundant channel from another (unnamed) provider. So: the stadium does have four 10 GB bandwidth pipes, by far the most capacity in any stadium we’ve heard of.

— More traffic stats: Though we will break these down in a separate post, the Niners said that for the Aug. 24 game fans used 1.96 Terabytes of Wi-Fi traffic, just a bit lower than the 2.13 TB used at the first preseason game on Aug. 17. The team also provided some DAS stats, claiming fans used another 1.02 TB of cellular data at the two preseason games combined.

Tech tour photos follow… including a sighting of some (shhhh!) Cisco equipment in the data center racks… click on photos for larger images.

Niners VP of technology Dan Williams (left) and COO Al Guido kick off the Wednesday tech tour.

Niners VP of technology Dan Williams (left) and COO Al Guido kick off the Wednesday tech tour.

Ted Girdner, Comcast VP of business services for California, talks stadium networking.

Ted Girdner, Comcast VP of business services for California, talks stadium networking.

Dan Williams talks Wi-Fi while the Levi's Stadium new turf grows silently behind him.

Dan Williams talks Wi-Fi while the Levi’s Stadium new turf grows silently behind him.

Mystery Cisco gear inside Levi's Stadium data center. Alert! Intruder!

Mystery Cisco gear inside Levi’s Stadium data center. Alert! Intruder!

Brocade router at Levi's Stadium data center. One of many. As in, many many.

Brocade router at Levi’s Stadium data center. One of many. As in, many many.

Wi-Fi gear in Levi's Stadium data room.

Wi-Fi gear in Levi’s Stadium data room.

Franks and DAS: DGP DAS antennas above food station.

Franks and DAS: DGP DAS antennas above food station.

Screenshot of food feature on Levi's Stadium app. Note the green light buttons to show expected wait times for express option.

Screenshot of food feature on Levi’s Stadium app. Note the green light buttons to show expected wait times for express option.

Obligatory Levi's Stadium selfie. MSR shirts complete the style.

Obligatory Levi’s Stadium selfie. MSR shirts complete the style.

Stadium Tech Report: Aruba Networks chief engineer on building the ‘ultimate stadium’ network at Levi’s

Chuck Lukaszewski, Aruba Networks

Chuck Lukaszewski, Aruba Networks

For Chuck Lukaszewski, getting the job to design the Wi-Fi network at Levi’s Stadium was a bit of a personal perfect storm: Having long desired to build the ultimate stadium Wi-Fi network, he’d finally been matched with a customer with plans and a budget to fulfill that dream.

Lukaszewski, the senior director for outdoor solutions engineering at Levi’s Wi-Fi gear supplier Aruba Networks, said the aggressive vision for connected fan engagement put forth by the San Francisco 49ers and their vice president of technology Dan Williams was a plan Lukaszewski had long wanted to pursue.

“I had a vision of how to build an ‘ultimate stadium,’ and Dan said he wanted an ultimate stadium,” said Lukaszewski in a recent phone interview. “He gave me the freedom to design something that had never really been done before. It was a real meeting of the minds.”

By early results, the joint effort is by all standards a success, with network traffic statistics from the first football game at Levi’s showing a robust Wi-Fi network with Super Bowl levels of user traffic. According to Lukaszewski, by the time kickoff rolled around for the Niners’ preseason game against the Denver Broncos on Aug. 17, the Wi-Fi network at Levi’s Stadium had already surpassed the three most recent Super Bowls when it came to concurrent network usage, and had also exceeded the peak network usage statistics from the recent World Cup finale in Brazil.

While the network still hasn’t been tested in a regular-season game, or with the planned instant-replay feature active on the Levi’s Stadium app, it’s probably safe to say that at the very least, Lukaszewski and Williams have hit the ground running successfully. And while the network team did have some factors in their favor – including the ability to blend the wireless infrastructure into the overall stadium buildout and the budget for 1,200 Wi-Fi APs – there was still significant work that had to be done to produce a wireless experience with the magnitude of Levi’s offering.

Avoiding the ‘circular firing squad of RF’ in antenna deployment

According to Lukaszewski, he’s had experience deploying Wi-Fi networks in many unusual or difficult places, including on oil tankers and on financial-market trading floors. But stadiums, he said, especially open-air arenas like football stadiums, require a much higher level of deployment expertise, especially when you are trying to provide an extremely high level of connectivity to every seat in the stadium, as the Niners wanted to do at their new 68,500-seat home in Santa Clara, Calif.

“The 49ers have a completely unique view of the connectivity experience in a stadium – it’s the whole relationship of a fan with home,” said Lukaszewski. “Technology is just one piece of that. And [wireless] connectivity, is the last yard.”

Under-seat Wi-Fi AP at Levi's Stadium.

Under-seat Wi-Fi AP at Levi’s Stadium.

But just like that last three feet into the end zone, the last yard is often the toughest distance to cover when it comes to providing Wi-Fi connectivity. Unlike cellular operators, who own and control their licensed swath of spectrum, Wi-Fi networks operate in the unlicensed band of airwaves, and have an unknown number and type of end-user customers to support. According to Lukaszewski, designing and deploying a large-scale stadium Wi-Fi network is extremely more difficult than building a similar cellular network, mainly because of the interference situation.

“The classic design for a cellular DAS [distributed antenna system] in a stadium is to just put antennas high up, all the way around the bowl, pointing down,” said Lukaszewski. “That’s completely inefficient for Wi-Fi – we call it the ‘RF circular firing squad.’ Cellular owners can do that because they are exclusive owners of their spectrum and can bathe the bowl in signals. With Wi-Fi we have to ‘share the air’ in a ‘listen before talk’ environment.”

APs under seats, close to the fans

Without getting too deep into the specifics of Wi-Fi networking, what Lukaszewski, Williams and their teams needed to do was to figure out how many Wi-Fi access points (APs) they would need to provide high-quality connectivity to every seat in the house, and where to put them not just to provide the connection, but to also avoid interfering with signals from other close-by APs. While some of the planning can be done beforehand, Lukaszewski said that like in a restaurant kitchen, the final product often involves some hands-on refinement.

“You have a recipe, but you taste it and change it as you go,” Lukaszewski said, comparing the Levi’s deployment to a dinner dish. “We start out by going with the plan, then take a lot of data and make changes in real time.”

In the Niners’ favor was the fact that Levi’s Stadium was built from the ground up with the connectivity in mind, which made it much easier to deploy the network than in older stadiums where wiring and antennas need to be retrofitted into existing structures. Things like the many under-the-seat APs at Levi’s, with their wiring holes and mounting infrastructure designed into blueprints, might require concrete drilling and unsightly cable runs at stadiums built before networking was a pressing need.

“It’s always better when you can design it in,” said Lukaszewski of the wireless infrastructure. “The key is to have lots of low power cells, right among the people. That’s how we extract capacity. To get that you have to mount APs, and get the wiring to them. That can be very expensive in a retrofit. And it’s not always going to look very nice when you do it.”

More recipes for different events

Even with the impressive stats from the initial game, Lukaszewski was fast to note that the Levi’s Stadium wireless network is far from a finished product.

“We still have a lot of work to do – we’re definitely not done trying yet.” Lukaszewski said. “Every game, we want to get a little better.”

Typically in a stadium situation, Lukaszewski said, it may take 5 to 6 big events for the network team to really lock in on the tuning and tweaking necessary to make the network run at optimal performance. But even then, there’s still work to be done since the network requirements often differ greatly for different event types and crowds.

“There can be, more than one [network] recipe,” Luckszewski said. “The use case for a concert, for example, is way different than a football game. In an indoor stadium where you have both basketball and hockey, you may have 3 or 4 recipes, depending on the event.” Levi’s Stadium, he noted, is already scheduled to host several different types of events, including soccer games, high school football games, and an NHL game early next year.

But even as Lukaszewski, Williams and their teams continue to improve on their work, the early returns seem to suggest that the Niners have accomplished what they set out to build at their new home: A connected stadium that provides a fan experience unlike that found in most other large public venues.

“I think the 49ers have set an example for any team, in any sport,” Lukaszewski said.

Stadium Tech Report: MLB stadium technology reports — NL Central

Editor’s note: The following team-by-team capsule reports of MLB stadium technology deployments are an excerpt from our most recent Stadium Tech Report for Q2 2014, which focuses on Major League Baseball. To get all the capsules in one place as well as our featured reports, interviews and analysis, download your free copy of the full report today.

NL CENTRAL

Reporting by Chris Gallo

Chicago Cubs
Wrigley Field
Seating Capacity: 41,160
Wi-Fi: Yes
DAS:Yes
Beaconing: No

The iconic Friendly Confines of Wrigley Field enters its 100th season in style.
The Chicago Cubs are celebrating all season long with special promotions and the experience is even better with Wi-Fi throughout the historic ballpark. The Cubs do not yet offer iBeacon technology, but the free Wi-Fi is not bad for a ballpark that’s been around for a century.

And with a proposed $500 million dollar restoration and renovation plan, it’s possible that the digital experience at Wrigley might improve in the coming years. Potential enhancements include new restrooms, concessions, a 6,000 square foot video board in left field, which has drawn opposition from neighboring rooftop owners and historic-minded fans, who don’t want to see the old Wrigley ruined. At the very least, pretty much everyone can get behind improved connectivity. But it remains to be seen if the Cubs’ owners will get approval for their renovation plans.

St. Louis Cardinals
Busch Stadium
Seating Capacity: 45,399
Wi-Fi: No
DAS:Yes
Beaconing: Yes

The reigning National League Champions enter their eighth season at Busch Stadium in 2014. Public facing Wi-Fi is absent, but the Cardinals unveiled non-digital enhancements with the Ballpark Village and AT&T Rooftop this year.
Busch Stadium does have several DAS antennas installed and is one of many teams to test out iBeacon technology with MLB At the Ballpark this season.

Milwaukee Brewers
Miller Park
Seating Capacity: 42,200
Wi-Fi: Premium suites only
DAS:Yes
Beaconing: No

Rain or shine, the game is always on at Miller Park. Opened in 2001, the stadium remains one of the more modern ballparks with a retractable roof and a climbing wall as part of the Dew Deck. Unfortunately, the modernizations do not include any public Wi-Fi for fans. Maybe next year Brew Crew?

Pittsburgh Pirates
PNC Park
Seating Capacity: 38,496
Wi-Fi: No
DAS: No
Beaconing: No

After making the playoffs last year for the first time since 1992, the Pittsburgh Pirates are poised to improve stadium connectivity at PNC Park. The Bucs plan to install major upgrades with additional access points and antennas, hopefully
in time for another home playoff series in 2014.

Cincinnati Reds
Great American Ball Park
Seating Capacity: 42,036
Wi-Fi: Yes
DAS:Yes
Beaconing: Yes

Entering its second decade of existence, the Great American Ball Park welcomes fans with free Wi-Fi and more than 60 iBeacons throughout the ballpark. The stadium uses that network to distribute ballpark information and targeted offers to fans throughout the game.

The Reds also partnered with Miami University to build a dedicated social media area on the third-base concourse. The Reds Connect Zone includes Wi-Fi access, charging stations for mobile devices, and more than 25 screens of tweets, Instagram photos, Facebook polls, and Vine videos for fans to stay connected at the game.

To get all the capsules in one place as well as our featured reports, interviews and analysis, download your free copy of the full report today.

Holy Terabyte! First football crowd at Levi’s Stadium uses 2.13 TB of Wi-Fi traffic, with nearly 25K fans on Wi-Fi at once

Levi's Stadium from Section 244. All photos: Paul Kapustka, Mobile Sports Report

Levi’s Stadium from Section 244. All photos: Paul Kapustka, Mobile Sports Report

All those predictions about Silicon Valley people using a stadium network more than other fans? It looks like they’re true.

The network numbers are in for the first football game at Levi’s Stadium, and they are pretty amazing: According to Dan Williams, the vice president of technology for the San Francisco 49ers, the Levi’s Stadium Wi-Fi network carried 2.13 terabytes of data during last Sunday’s preseason game, with a peak of 24,775 fans on the Wi-Fi network at the same time. Those numbers are comparable to the latest Super Bowl at MetLife Stadium in New Jersey, where according to AT&T and Verizon there was approximately 2.5 TB of data used on wireless networks.

The kicker to the Niners’ stats — they do NOT include any traffic figures from the Levi’s Stadium DAS, the distributed antenna system that is meant to provide enhanced cellular coverage in the stadium. What follows is an email Q-and-A with Williams, who kindly answered our extensive list of questions. The real question is, if Niners fans are hitting terabyte levels during preseason games, what’s going to happen when the season starts for real? And the instant replay function in the team app is live? Read on for some great insight from Williams. Additional editor’s note: The companies talked about include Aruba Networks, the provider of Wi-Fi gear; Brocade, provider of back-end networking gear and integration; DAS Group Professionals, the integrator and deployment team behind the DAS (the network of small antennas that improve in-building cellular coverage).

Mobile Sports Report: what was the peak number for simultaneous Wi-Fi connections? The average?
Dan Williams: We peaked at 24,775 (roughly 38% of attendance) concurrent connections with an average of 16,862 (roughly 25% of attendance).

MSR: When did connections spike? When did they start and then tail off?
Williams: We had two spikes, 1:02 p.m. [editor’s note: kickoff was 1 p.m.] with a system wide peak of 2.3Gbps and then again at 1:53 p.m. with 1.7Gbps. We averaged more than 1Gbps for more than two hours.

Niners VP of technology Dan Williams attempts to fix my Droid 4 Wi-Fi issues (while trying not to laugh at the fact that I actually have and use a Droid 4)

Niners VP of technology Dan Williams attempts to fix my Droid 4 Wi-Fi issues (while trying not to laugh at the fact that I actually have and use a Droid 4)

MSR: What was the total data tonnage on the Wi-Fi network?
Williams: We offloaded 2.13 Terabytes during the event.

MSR: What were the usage patterns with the app — which feature did people use most?
Williams: We had a great deal of usage throughout Sunday. The food ordering app usage was top of the list with ticketing being a close second while video would take third from a feature standpoint.

MSR: What are the plans with the instant replay feature… when will it be live (and can you explain why it was held back)?
Williams: We felt a lot of folks were happy with the livestream, so we wanted to focus more on a couple of core features with food ordering and ticketing a bit more at this point. Replays will be available to all by the first regular season home game.

MSR: Can you explain exactly how the location feature works… does it require Bluetooth to be on?
Williams: The location service is mainly built around low-energy Bluetooth, BLE. We have a number of beacons placed throughout the open areas and points-of-interest which allow the app to identify your location through proximity. Aruba helped us build this as well. GPS is also used but the primary resource is Bluetooth. The app prompts users to enable Bluetooth to provide improved location awareness.

One of the big screens in Levi's Stadium.

One of the big screens in Levi’s Stadium.

MSR: Can you provide any stats on the DAS performance?
Williams: The DAS held up really well. Like WiFi, we found some areas that need tuning. Unlike WiFi, the carriers protect a lot of their specific data but they have told us they are very happy with the system DGP helped us with. It is important to note our DAS and WiFi have been built to compliment each other and I think between Aruba and DGP, we did that very well. Most come here looking to connect to WiFi but our story internally has been we are going to have an awesome connectivity play regardless of medium.

MSR: Could you guys see any [more] of the 2.4 GHz issues like the one I had?
Williams: As you know, 2.4GHz is limited with non-overlapping channels so we suspect a number of legacy devices may have some problems. That said, we had a ratio of 2:1 with respects to 5GHz to 2.4GHz [usage] which shows a good deal of 2.4GHz usage. We know we still have some optimizations to do in the upper bowl and upper concourse while we continue to fine-tune the main bowl and concourse as we noticed our cell edge was weaker than expected when the stands were full. Our Aruba team did a great job capturing real-time data during the event as there is really no other way to test this stuff without a full venue. We will make some tweaks and continue to learn more from every event we host. Between Aruba, Brocade, and the 49er tech staff, we are not resting on our laurels. We know there is more to do.