August 5, 2015

Super DAS: Crown Castle’s neutral host infrastructure aims to keep Super Bowl XLIX fans connected, inside and outside the stadium

Editor’s note: This story is part 1 of a series of profiles of the providers of the extensive Distributed Antenna System (DAS) deployment for Super Bowl XLIX at and around the University of Phoenix Stadium in Glendale, Ariz. and other parts of the Phoenix city area as well. Stay tuned all week as we highlight how DAS will keep Super Bowl fans connected, no matter where they roam in and around Phoenix and Glendale this week and weekend.

University of Phoenix Stadium getting its Super Bowl on. (Click any photo for a larger image) Credit all photos: Paul Kapustka, MSR

University of Phoenix Stadium getting its Super Bowl on. (Click any photo for a larger image) Credit all photos: Paul Kapustka, MSR

When is a big air-fan vent not an air-fan vent? When it’s a fake vent covering a hidden cellular antenna, put there to keep people from noticing the technology that’s keeping their cell phones connected. Before kickoff at Super Bowl XLIX Feb. 1 in Glendale, Ariz., many fans outside the University of Phoenix Stadium will walk right by a faux vent and its sheltered equipment, never knowing the attention to detail that goes into a major-venue Distributed Antenna System (DAS) deployment.

But to stadium technology connectivity professionals, such leaps of aesthetic deception are just part of a day’s, or perhaps month’s, DAS deployment work. For neutral host DAS provider Crown Castle, the fake vents on the shell of the University of Phoenix Stadium — and the powerful antennas behind — are just one part of a massive project to ensure there is excellent mobile-device connectivity both inside and outside the Super Bowl stadium, so that fans never get a dropped signal anywhere between the parking lot and their prized seat.

During a recent press tour, a small team of Crown Castle employees showed off some of the upgraded DAS network deployed at the University of Phoenix Stadium as well as in the surrounding Westgate Sports and Entertainment District, a sort of open-air mall that stretches from the UoP Stadium past numerous attached restaurants and stores, also encompassing the Gila River Arena, home of the Phoenix Coyotes of the NHL. Over the past year or so, Crown Castle has been upgrading the DAS inside and outside the arena, throughout the mall areas as well as into the huge parking lots that surround it and the football stadium, bringing connectivity to phones being used by customers from all the four major U.S. wireless carriers.

Since the mall and all its food outlets are conveniently located a short stroll from the stadium, it’s a good bet that a large portion of the Super Bowl crowd will spend time wandering around the Westgate area before and after the big game. Thanks to Crown Castle’s efforts, there shouldn’t be many connectivity problems, as antenna deployments on light poles, building rooftops and — yes, even behind fake vents — should be able to keep devices on the cellular networks without a glitch.

Game day connectivity starts in the parking lot

Since we couldn’t actually spend much time wandering around the stadium itself — even three weeks before the big game, the facility was already on NFL security lockdown — most of the Crown Castle tour consisted of walking around the Westgate mall/neighborhood, hearing about the various methods Crown Castle used to locate the necessary DAS antennas. In all, there are five separate DAS networks Crown Castle is responsible for in the area around the stadium: The football stadium itself; the Gila River Arena (which we will profile in an upcoming feature on hockey stadiums); the Westgate shops and restaurants; the nearby Renaissance Hotel; and the surrounding parking lots.

Parking lot light poles, Westgate entertainment district. Can you spot the DAS?

Parking lot light poles, Westgate entertainment district. Can you spot the DAS?

The curious start of the tour in a far-flung parking lot made sense when we found ourselves next to a small DAS equipment box and a light pole with multiple antennas (which had not yet been covered with their final aesthetic sheaths). Aaron Lamoureux, program manager for Crown Castle’s small cell solutions, served as tour guide, and said that for the Westgate area alone there were 18 individual node locations, with about 52 antennas total. Some were located on light poles, some on rooftops, and some along walkways between buildings, to conquer the unique RF characteristics of the open-air/large building outdoor mallish area that is Westgate. (See photos for DAS geek views)

For the University of Phoenix Stadium itself, Crown Castle deployed 228 DAS antennas inside (more on this in an upcoming profile) and at 21 different locations outside the stadium, 13 of those on parking lot poles and 8 mounted on the building itself. Why building-mounted antennas? If you’ve never been there, the University of Phoenix Stadium has a large plaza area on one side, which is used for pre-game activities like rallies, bands and other walk-up amenities where fans gather before entering. The challenge for Crown Castle was finding places to deploy antennas at a low enough height to cover crowds of people standing in one location. While some parts of the building allowed for regular antenna placements, a big part of the plaza faces part of the stadium wall that is a sheer sheet, with no aesthetic place to mount a DAS antenna — unless you add a fake vent or two to the existing design, that is.

Keeping everyone happy is part of the neutral host job

See the big air vents? Nobody would tell us which ones were 'faux vents,' there to hide DAS antennas

See the big air vents? Nobody would tell us which ones were ‘faux vents,’ there to hide DAS antennas

To people outside the industry it might seem silly to go to such lengths just to keep folks from noticing antennas, but anyone who’s deployed a network for a detail-oriented building owner knows why aesthetics are important. That’s why you paint antenna enclosures to match the surrounding walls, or build sheaths to keep wires and other obvious gear out of main sight. It’s part of the art of wireless network deployment, and not as simple as it sounds. Experience counts.

The complex owner and operator relationships involved in the stadium and surrounding-area DAS also seem tailor-made for a big, experienced provider like Crown Castle, which has a long history of deploying and operating multiple-tenant networks. With five different landlords and four different carriers, being the neutral DAS host for this year’s Super Bowl is a task with many moving parts; but, as Mike Kavanagh, president of sales for Crown Castle’s small cell solutions, said, “We understand how to run networks, how to manage them and deal with carriers. It’s high touch and very fluid. But we know that business.”

COMING UP NEXT: What’s inside the network inside the stadium.

MORE PICTURES BELOW! (Click on any picture for a larger image.)

Sky Harbor Airport: Ready for Super XLIX

Sky Harbor Airport: Ready for Super XLIX

Verizon's NFL Mobile ads were in airport walkways well before the Big Game

Verizon’s NFL Mobile ads were in airport walkways well before the Big Game

If you stumble off the escalator, Bud Light is there to catch you

If you stumble off the escalator, Bud Light is there to catch you

The Westgate uber-mall should see a lot of fan activity (and connectivity) on game day

The Westgate uber-mall should see a lot of fan activity (and connectivity) on game day

Here's the official Super Bowl replay HQ (actually a place with DAS antennas on the roof that you can't see)

Here’s the official Super Bowl replay HQ (actually a place with DAS antennas on the roof that you can’t see)

Mama Gina's will offer you pizza and DAS on the roof

Mama Gina’s will offer you pizza and DAS on the roof

More DAS antennas, on a Westgate walkway

More DAS antennas, on a Westgate walkway

Outside UoP Stadium, where the architecture allows for DAS antenna placement

Outside UoP Stadium, where the architecture allows for DAS antenna placement

Close-up of that placement. Still pretty well hidden.

Close-up of that placement. Still pretty well hidden.

Parking lot light mounts. These will have sheaths by Super Sunday.

Parking lot light mounts. These will have sheaths by Super Sunday.

Here's the remote equipment box that powers the light pole antennas. Also scheduled for more concealment.

Here’s the remote equipment box that powers the light pole antennas. Also scheduled for more concealment.

Every artist leaves a signature...

Every artist leaves a signature…

YinzCam’s Super Bowl stadium app will have instant replays, Super Bowl commericals, stadium maps and more

Screen shot of Super Bowl app for this year's game.

Screen shot of Super Bowl app for this year’s game.

We’ve been waiting for official word on what the YinzCam-developed app for the Super Bowl will look like, and though there’s no press release the page where we are guessing it will eventually be available is offering some details, like the availability of instant replays from different camera angles, video of Super Bowl commercials, and stadium maps.

On the Seahawks.com site we found a good how-to story for fans going to the game, which included a link to this page, where we are guessing the Super Bowl stadium app will be available for download. Here is the boilerplate:

New for Super Bowl XLIX, the Super Bowl Stadium App Presented by Verizon aims to take the fan experience inside University of Phoenix Stadium to the next level. Features that will enhance Super Bowl ticketholders’ experiences include exclusive in-stadium video content such as Super Bowl commercials and replays from four different camera angles, stadium seating and concession maps, once-in-a-lifetime gameday opportunities visible only to fans inside the stadium and the option to receive up-to-the-minute gameday notifications. Available on iOS, Android and Windows. Goes Live 23rd January 2015

(Looks like the app is already available in the App Store and in Google Play, but nothing is live; we downloaded the app and the only three buttons available, for highlights, commercials and memories, all say they will be available on Feb. 1 at the stadium, so no idea what the “goes live” on the splash page above means yet.)

Screen Shot 2015-01-23 at 11.40.41 PMYinzCam founder and CEO Priya Narasimhan had told us earlier this year that a Super Bowl app was in the making, and apparently it will contain features found in some of the latest YinzCam app deployments, such as the Seattle Seahawks’ new stadium app, which has multiple camera angle replays. The Super Bowl app is different from the Arizona Cardinals’ regular stadium app, which was also built by YinzCam, which also features instant replays.

We were able to download the app for iPhone (it’s free) and apparently you will need to be connected to the stadium Wi-Fi (which has the clever SSID of “Stadium WiFi”) in order to view highlights and other video options.

The good thing for fans at the big game, there will be plenty of networking horsepower to keep the app running, no matter where you are. If you’re inside the stadium there is a new Wi-Fi network and a refurbished DAS deployment to keep fans connected; stay tuned next week for our big breakdown of DAS deployments and carrier plans to keep the Super Bowl crowds super-connected.

NBC will live-stream Super Bowl online for tablets, computers; Verizon NFL Mobile will carry for smartphones

Screen shot 2015-01-20 at 10.37.03 AMIf for some reason you are banned from the living room couch for the Super Bowl, NBC has you covered — the network will be streaming the game live online, along with hours of pregame, postgame and halftime festivities — for anyone with an Internet connection and a laptop, PC or tablet.

According to a press release sent out Monday NBC said it will also not require viewers to have a qualifying cable or satellite contract to view the game, thereby eliminating the often annoying login process that accompanies many other online live sports streaming activities. You will, of course, be subject to multiple NBC advertisements but hey — a small price to pay for the convenience of being able to watch the game online.

(Mobile Sports Report is old enough to remember Super Bowl parties where we rented extra TV sets for the bathrooms and the kitchen; now you can just use in-house Wi-Fi and a tablet or laptop, perhaps with a splash guard.)

On the cellular side, if you are stuck somewhere and want to watch on your phone, the only option is having a Verizon contract and using the NFL Mobile app. If you are a More Everything customer the live viewing of the game is free, if not you must pay a $5 monthly charge for the one day in February that you will need premium access. (Pro tip for Verizon customers — don’t forget to cancel that premium access charge the day after the game, since Verizon will happily charge you $5 a month all summer long even though there are no NFL games during that time.)

It’s an easy guess so we will predict right now that this year’s Super Bowl will set new online records for most Internet viewers — without fail this has happened every year since the networks and the league started making the game available online. According to the NBC folks the online stream will have some handy extras, like the DVR feature that lets you scroll back to important plays, as well as additional camera angles and in-game stats.

Now our next dream is for the Shield and its broadcasters to follow ESPN’s lead on the college championships and provide online “Megacast” options for alternate announcers. A man can hope.

Seahawks hit local Wi-Fi record during playoff game with 2.6 Terabytes of traffic; Verizon maintains cone of silence over its investment in network

Screen Shot 2015-01-16 at 11.05.18 AMThe Wi-Fi network that debuted in CenturyLink Field this season had its highest Wi-Fi traffic day last weekend, when 2.6 terabytes of data was carried during the Seattle Seahawks’ Jan. 10 playoff victory over the Carolina Panthers.

According to the Seahawks’ tech staff, 18,899 of the 68,524 fans in attendance used the Wi-Fi network at some point, with a peak concurrent user mark of 15,662. The peak bandwidth utilization of 1.4 Gbps was reached just after Cam Chancellor sealed the win with his electric 90-yard interception touchdown return, the Seahawks tech staff said.

The Wi-Fi numbers represent traffic on both of the separate Wi-Fi networks in the stadium, one of which is reserved exclusively for Verizon Wireless customers. Verizon, which has declined to comment publicly on the specifics of its partnership with the Seahawks, is believed to have bankrolled a major portion of the Wi-Fi deployment at CenturyLink. Before the Wi-Fi partnership between the Seahawks, Wi-Fi gear provider Extreme Networks and Verizon was officially announced on Oct. 29, Verizon claimed it had “added an in-stadium Wi-Fi system” at CenturyLink prior to the start of the 2014 football season, as part of a national football stadium Wi-Fi map Verizon published on Sept. 4. The Wi-Fi deployment was somewhat of a surprise, since team officials had long said they were looking at 2015 as the year they might pull the trigger on a Wi-Fi expenditure. Apparently, having available funding from Verizon helped push the project forward faster than expected; but again, we have no official confirmation or explanation of the exact fiscal participation level of all the partners involved.

For Seattle fans, having Verizon as a Wi-Fi partner has additional perks — in addition to a separate Wi-Fi network for Verizon customers, all fans at the stadium also have access to the NFL Network’s popular RedZone channel, via the new Seahawks stadium app created by YinzCam. Because of various conflicting rights contracts, RedZone isn’t available in most stadiums for fans to watch. The Seahawks also have a number of live-action and replay views available via the app; however, the stadium’s tech team did not have any metrics on fan use of the app or the number of video replays watched.

Wi-Fi access point antennas visible on poles at CenturyLink Field, Seattle. Credit: Extreme Networks

Wi-Fi access point antennas visible on poles at CenturyLink Field, Seattle. Credit: Extreme Networks

Verizon’s reluctance to comment publicly on its Wi-Fi deployments is no surprise; repeated attempts and queries by MSR for interviews with Verizon executives about Wi-Fi deployments are routinely ignored by Verizon representatives, and public quotes like the one from Bobby Morrison, president for the Pacific Northwest and Alaska at Verizon Wireless, in the official press release, don’t offer any details about Verizon’s level of fiscal commitment to the CenturyLink deployment. Verizon has also declined to comment on its Wi-Fi network deployments at Ford Field in Detroit, and at the Staples Center in Los Angeles.

Verizon executives were also conspicuously absent from a stadium-technology event centered on the Wi-Fi network earlier this week, leading some reports to omit Verizon’s considerable participation in the network’s deployment, something NFL CIO Michelle McKenna-Doyle told MSR about earlier this year.

Though no statistics were available from Verizon about the DAS deployment it also put in at CenturyLink Field this year, AT&T did share some DAS stats from CenturyLink for the Seahawks’ Dec. 14 victory over the San Francisco 49ers. During that game, AT&T customer traffic on the AT&T DAS at CenturyLink was 395 GB, according to AT&T. It will be interesting to see how much wireless traffic this weekend’s NFC championship game between the Seahawks and the Packers will generate — we’ll track down as much of it as we can to see if it compares to the 6 TB mark set at the recent College Football Playoff championship game at AT&T Stadium in Arlington, Texas.

All NFL playoff games available online; Verizon only for smartphone watching

vzn_playoffWe’ve come a long way from the days when it was a struggle to even find NFL live action online. This season, all NFL playoff games, including the Super Bowl, will be available for online viewing, via a desktop computer, laptop or tablet, no matter which network is carrying the games. The league has even created a handy single web page to find instant access to the live streams, no small matter since network pages usually make you jump through several clicks to find the actual live stream.

Depending upon the broadcast network, you may need to have a qualifying cable or satellite contract to view the games. For this morning’s CBS game there was no confirmation process but for the afternoon Fox game I needed to submit cable provider info. Let me know what you see.

However, if you want to watch playoff games on your smartphone, your only option is to be a Verizon Wireless customer since that provider is the exclusive NFL live action host for smartphone devices. To view the live action, Verizon customers need to have either a More Everything plan or pay the $5 monthly premium fee for the NFL Mobile application. To be clear, customers of any wireless provider can download the NFL Mobile app to get all the stats and archived video it provides, but only Verizon customers can purchase access to live action on phone platforms.

UPDATE: Congrats to FOX for having the first playoff online broadcast fail:

Screen Shot 2015-01-04 at 3.01.07 PM

No word from Fox yet on when its stream will come back, or why it was down.

NFL’s CIO says teams need to share technology know-how

Michelle McKenna-Doyle, CIO, NFL

Michelle McKenna-Doyle, CIO, NFL

Editor’s note: the following interview with NFL CIO Michelle McKenna-Doyle was featured in our most recent long-form report, THE FOOTBALL ISSUE, which is available for free download. In addition to team-by-team capsules of technology deployments for all 32 NFL teams the issue also has several in-depth stadium technology profiles and an extensive look at the technology behind the San Francisco 49ers’ new Levi’s Stadium. Get your copy today!

In a league known for its intense rivalries, is it possible to get teams to work together and share information for the betterment of all? In the area of stadium technology, that task is on the to-do list for Michelle McKenna-Doyle, who is now in her third season as chief information officer for the National Football League.

In an interview with Mobile Sports Report, McKenna-Doyle outlined some of the league’s recent accomplishments in technology-related areas like instant replay and digital content, while also explaining how the league oversees stadium technology deployments. According to McKenna-Doyle, one of her office’s jobs is to act as a best-practices and lessons-learned clearinghouse, to better move the state of NFL stadium technology forward faster as a whole.

“One of our roles is helping teams help themselves” with technology deployment strategies, McKenna-Doyle said. “What we want is to provide a forum where clubs can share information with each other. If somebody’s done something and learned it doesn’t work, we can tell other clubs not to waste their time doing the same thing.”

The business use of technology

Focusing on the business uses of technology and not the 1s and 0s is somewhat of a natural fit for McKenna-Doyle, who spent 13 years at the Walt Disney Company in disciplines including finance and marketing before becoming a VP in IT for two years. During CIO stints at Centex Homes and Universal Orlando Resort, McKenna-Doyle said she focused on using technology to enhance the guest experience, a goal the NFL sought when it brought her aboard in September 2012.

Wi-Fi access point antennas visible on poles at CenturyLink Field, Seattle. Credit: Extreme Networks

Wi-Fi access point antennas visible on poles at CenturyLink Field, Seattle. Credit: Extreme Networks

“Part of my job is making sure our in-stadium experience for mobility meets the needs of our fans,” McKenna-Doyle said. While it is true that commissioner Roger Goodell said he wanted all NFL stadiums to have fan-facing Wi-Fi, and that the league does expect teams to meet a minimum set of connectivity standards, McKenna-Doyle said the NFL’s overall stadium-tech strategy is to be more of a guide than to dictate exactly which technologies or apps teams should deploy.

“People really are fans of their own team first, and we encourage clubs to have that engagement, and help them interact with fans,” McKenna-Doyle said. “There are minimum standards and we do grade their [technology] experience, and report that back to the club. But they manage it. We are more of a guide.”

Putting out a plan for stadium Wi-Fi

In the area of Wi-Fi, for example, McKenna-Doyle said that last year the league put together “a really deep-dive spec” that laid out all the basics necessary for stadium Wi-Fi deployments. “That was so teams didn’t have to start at square one for design,” McKenna-Doyle said.

The league also helped move Wi-Fi deployments forward faster by signing a preferred-supplier deal with Extreme Networks, under which teams get a discount on pricing in exchange for the league-wide sponsorship exposure. (Editor’s note: This week, the league announced that Extreme is now the official Wi-Fi technology supplier for the NFL.) Though teams are not required to use Extreme’s Wi-Fi gear, new Extreme-based Wi-Fi networks are in use this season at Seattle, Jacksonville, Tennessee, and Cincinnati, joining two previous Extreme installations in Philadelphia and New England.

“It’s a great option if teams choose Extreme, and it [the league deal] also creates a competitive environment for other suppliers like Cisco to step up,” McKenna-Doyle said.

Fans take pictures at Levi's Stadium. Credit: Paul Kapustka, MSR

Fans take pictures at Levi’s Stadium. Credit: Paul Kapustka, MSR

If necessary, the league can also play Wi-Fi matchmaker, as McKenna-Doyle said it did in bringing partner Verizon into Seattle, where the carrier deployed a Wi-Fi network at CenturyLink Field that is live this season.

At the writing of this report, 10 of the NFL’s 32 teams still had no fan-facing Wi-Fi services at their stadiums, a list that includes Green Bay, Baltimore, Cleveland, Buffalo, Houston, Oakland, San Diego, Washington, Minnesota and St. Louis. While McKenna-Doyle said that there are “a few more Wi-Fi announcements coming,” she also noted that some teams with lease uncertanties still don’t have firm plans to deploy Wi-Fi.

Help on the app side with partner YinzCam

McKenna-Doyle said the league also provides assistance in stadium application development via YinzCam Inc., a Pittsburgh-based company that has developed mobile apps for a number of pro sports teams. The NFL, which was an investor in YinzCam, uses the company’s technology in its league mobile apps, and McKenna-Doyle said YinzCam is also developing an app for the upcoming Super Bowl XLIX that will be “like nothing we’ve ever had before.”

Though teams are not required to use YinzCam – the San Francisco 49ers, for example, turned to newcomer VenueNext to develop their Levi’s Stadium app – McKenna-Doyle said that YinzCam may be a fit for other teams.

“For quality and speed to market, [YinzCam’s] product is very strong,” McKenna-Doyle said.

It’s about the fans, not the technology

Following a summer that saw her office overseeing the new method for on-field official review of replay calls – “which meant building a new system for 32 teams, 31 stadiums, and training all the officials” – as well as the launch of the NFL Now digital content site, McKenna-Doyle is back spending time with teams, counseling them on technology deployment resource management – “what to prioritize, and what to put on the back burner,” she said.

That includes technology ideas that might not work operationally, like a food-ordering service that isn’t staffed properly. For the in-seat food ordering feature at Levi’s Stadium, for instance, the 49ers said they did extensive research, hiring and training to make sure they had enough feet on the ground – runners carrying orders – to make the tech-inspired feature work.

If teams don’t do the human engineering behind the scenes, McKenna-Doyle said, the technology may not be that cool.

“We stress that it’s not about the technology, but about the fan experience,” McKenna-Doyle said. “It has to be operationally sound, and it has to be integrated with being at the game. If it’s not something that’s operationally sound, you might be better off not doing it.”