November 25, 2014

NFL’s CIO says teams need to share technology know-how

Michelle McKenna-Doyle, CIO, NFL

Michelle McKenna-Doyle, CIO, NFL

Editor’s note: the following interview with NFL CIO Michelle McKenna-Doyle was featured in our most recent long-form report, THE FOOTBALL ISSUE, which is available for free download. In addition to team-by-team capsules of technology deployments for all 32 NFL teams the issue also has several in-depth stadium technology profiles and an extensive look at the technology behind the San Francisco 49ers’ new Levi’s Stadium. Get your copy today!

In a league known for its intense rivalries, is it possible to get teams to work together and share information for the betterment of all? In the area of stadium technology, that task is on the to-do list for Michelle McKenna-Doyle, who is now in her third season as chief information officer for the National Football League.

In an interview with Mobile Sports Report, McKenna-Doyle outlined some of the league’s recent accomplishments in technology-related areas like instant replay and digital content, while also explaining how the league oversees stadium technology deployments. According to McKenna-Doyle, one of her office’s jobs is to act as a best-practices and lessons-learned clearinghouse, to better move the state of NFL stadium technology forward faster as a whole.

“One of our roles is helping teams help themselves” with technology deployment strategies, McKenna-Doyle said. “What we want is to provide a forum where clubs can share information with each other. If somebody’s done something and learned it doesn’t work, we can tell other clubs not to waste their time doing the same thing.”

The business use of technology

Focusing on the business uses of technology and not the 1s and 0s is somewhat of a natural fit for McKenna-Doyle, who spent 13 years at the Walt Disney Company in disciplines including finance and marketing before becoming a VP in IT for two years. During CIO stints at Centex Homes and Universal Orlando Resort, McKenna-Doyle said she focused on using technology to enhance the guest experience, a goal the NFL sought when it brought her aboard in September 2012.

Wi-Fi access point antennas visible on poles at CenturyLink Field, Seattle. Credit: Extreme Networks

Wi-Fi access point antennas visible on poles at CenturyLink Field, Seattle. Credit: Extreme Networks

“Part of my job is making sure our in-stadium experience for mobility meets the needs of our fans,” McKenna-Doyle said. While it is true that commissioner Roger Goodell said he wanted all NFL stadiums to have fan-facing Wi-Fi, and that the league does expect teams to meet a minimum set of connectivity standards, McKenna-Doyle said the NFL’s overall stadium-tech strategy is to be more of a guide than to dictate exactly which technologies or apps teams should deploy.

“People really are fans of their own team first, and we encourage clubs to have that engagement, and help them interact with fans,” McKenna-Doyle said. “There are minimum standards and we do grade their [technology] experience, and report that back to the club. But they manage it. We are more of a guide.”

Putting out a plan for stadium Wi-Fi

In the area of Wi-Fi, for example, McKenna-Doyle said that last year the league put together “a really deep-dive spec” that laid out all the basics necessary for stadium Wi-Fi deployments. “That was so teams didn’t have to start at square one for design,” McKenna-Doyle said.

The league also helped move Wi-Fi deployments forward faster by signing a preferred-supplier deal with Extreme Networks, under which teams get a discount on pricing in exchange for the league-wide sponsorship exposure. (Editor’s note: This week, the league announced that Extreme is now the official Wi-Fi technology supplier for the NFL.) Though teams are not required to use Extreme’s Wi-Fi gear, new Extreme-based Wi-Fi networks are in use this season at Seattle, Jacksonville, Tennessee, and Cincinnati, joining two previous Extreme installations in Philadelphia and New England.

“It’s a great option if teams choose Extreme, and it [the league deal] also creates a competitive environment for other suppliers like Cisco to step up,” McKenna-Doyle said.

Fans take pictures at Levi's Stadium. Credit: Paul Kapustka, MSR

Fans take pictures at Levi’s Stadium. Credit: Paul Kapustka, MSR

If necessary, the league can also play Wi-Fi matchmaker, as McKenna-Doyle said it did in bringing partner Verizon into Seattle, where the carrier deployed a Wi-Fi network at CenturyLink Field that is live this season.

At the writing of this report, 10 of the NFL’s 32 teams still had no fan-facing Wi-Fi services at their stadiums, a list that includes Green Bay, Baltimore, Cleveland, Buffalo, Houston, Oakland, San Diego, Washington, Minnesota and St. Louis. While McKenna-Doyle said that there are “a few more Wi-Fi announcements coming,” she also noted that some teams with lease uncertanties still don’t have firm plans to deploy Wi-Fi.

Help on the app side with partner YinzCam

McKenna-Doyle said the league also provides assistance in stadium application development via YinzCam Inc., a Pittsburgh-based company that has developed mobile apps for a number of pro sports teams. The NFL, which was an investor in YinzCam, uses the company’s technology in its league mobile apps, and McKenna-Doyle said YinzCam is also developing an app for the upcoming Super Bowl XLIX that will be “like nothing we’ve ever had before.”

Though teams are not required to use YinzCam – the San Francisco 49ers, for example, turned to newcomer VenueNext to develop their Levi’s Stadium app – McKenna-Doyle said that YinzCam may be a fit for other teams.

“For quality and speed to market, [YinzCam’s] product is very strong,” McKenna-Doyle said.

It’s about the fans, not the technology

Following a summer that saw her office overseeing the new method for on-field official review of replay calls – “which meant building a new system for 32 teams, 31 stadiums, and training all the officials” – as well as the launch of the NFL Now digital content site, McKenna-Doyle is back spending time with teams, counseling them on technology deployment resource management – “what to prioritize, and what to put on the back burner,” she said.

That includes technology ideas that might not work operationally, like a food-ordering service that isn’t staffed properly. For the in-seat food ordering feature at Levi’s Stadium, for instance, the 49ers said they did extensive research, hiring and training to make sure they had enough feet on the ground – runners carrying orders – to make the tech-inspired feature work.

If teams don’t do the human engineering behind the scenes, McKenna-Doyle said, the technology may not be that cool.

“We stress that it’s not about the technology, but about the fan experience,” McKenna-Doyle said. “It has to be operationally sound, and it has to be integrated with being at the game. If it’s not something that’s operationally sound, you might be better off not doing it.”

Is mobile access to live NFL games the next battleground for AT&T and Verizon Wireless?

NFL Mobile screen shot of server fail during Week 1. Photo Credit: Paul Kapustka, MSR

NFL Mobile screen shot of server fail during Week 1. Photo Credit: Paul Kapustka, MSR

Today’s news that DirecTV has signed a $12 billion deal with the NFL — priced at $1.5 billion a year for 8 years — to keep carrying its Sunday Ticket package has me thinking: Are we on the verge of a battle royale between the country’s two biggest cellular providers over mobile access to NFL games?

I don’t think it’s hyperbole to claim that mobile access to sports content and live NFL games are two of the hottest things going. Go look up any list of the most-watched live TV shows, and you will find various big NFL games dominating the list. Add that to stats like today’s news out of ESPN where the worldwide leader claimed it had 61.3 million unique mobile users during August and you can perhaps see another reason why AT&T might want to buy DirecTV: Because with Sunday Ticket, AT&T has a possible way to trump Verizon’s stranglehold on smartphone access to live NFL action, which it now shows via its exclusive contract with the NFL for its NFL Mobile app premium service.

Judging by traffic and search terms on our humble little site, people looking to find ways to watch live NFL action on their mobile devices is a pretty hot topic these days. Right now, the only way for most people to see any live action at all on a smartphone is to be a Verizon Wireless subscriber, and have the premium service for the company’s NFL Mobile app. Free to “More Everything” data plan customers and $5 a month for others, the premium NFL Mobile package provides access to Sunday night, Monday night and Thursday night games, as well as local Sunday games.

Sunday Ticket vs. NFL Mobile?

You can also watch the RedZone channel via NFL Mobile, but confusingly if you are on the More Everything plan you need to pay an additional $1.99 a month, a new process that helped mess up Verizon’s NFL Mobile access earlier this season. Non-Share Everything customers who pay the $5 a month fee have RedZone included for free. (For many true NFL fans, RedZone is often even better than having games streamed, since you get all the best action, even from blacked-out games or games not televised locally.)

Screen Shot 2014-10-01 at 2.37.08 PMWhere the battle might be joined is in the tablet space — since Verizon’s $1 billion agreement with the NFL only provides live game access to “phone-like devices” and not to tablets or computers, it will be interesting to see what happens if and when AT&T becomes the new owner of DirecTV. One of the more interesting options from DirecTV this year was the NFLSundayTicket.TV option, which allows a type of “cord-cutting,” providing all the options of Sunday Ticket without having to have satellite service. Right now the option (pricing starts at $199 for the season) is only available in a few cities, universities and apartment buildings, but with the heft of AT&T behind it who knows what might happen to both that deal and the regular Sunday Ticket package.

Though far pricer, the $329.94 Sunday Ticket Max plan offered by DirecTV currently allows for mobile viewing of all games, on “computer, tablet, phone or game console.” Anyone else see the possibility of AT&T offering free Sunday Ticket plans to purchases of new phones or tablets?

Verizon, which provides information about NFL Mobile subscriber stats just like Bill Belichick provides deep insights on the inner machinations of the New England Patriots, has not recently stated how many subscribers it has on the NFL Mobile premium package. But for $1 billion over 4 years you can bet the number of users is well into the millions, maybe even more than 10 million — and the exclusivity of NFL live action has certainly been a big selling point for Big Red. The good news for NFL fans is, if any battle begins, it will likely include more access for lower costs — that’s the kind of competition we can all cheer for.

Verizon, NFL fumble opening-day live video for NFL Mobile app — for the 2nd year in a row

Screen shot of the message NFL Mobile users are getting used to seeing on opening day. Credit: Paul Kapustka, MSR.

Screen shot of the message NFL Mobile users are getting used to seeing on opening day. Credit: Paul Kapustka, MSR.

It’s not a streak you want to continue, especially if you are the NFL, Verizon Wireless, or one of the frustrated customers who weren’t able to see live video using Verizon’s NFL Mobile application during Sunday’s season-opening games. But just like last year, when activation servers for a new version of the app got overwhelmed, issues arose again on Sunday, making live video — one of the premium features of NFL Mobile that makes the app and service so attractive — unavailable for most of the day, including the large slate of early games.

Verizon executives who spoke with MSR Monday declined to comment on the NFL Mobile issues, instead referring all questions to their partner, the NFL media team. Verizon wireless support was also apparently offline on Sunday, leaving the task of responding to the many irate messages and tweets to the small but valiant NFL Mobile Support team.

Alex Riethmiller, vice president of communications for NFL Media, said the issue once again was due to high traffic overwhelming the authentication servers, which (putting it simply) identify whether or not a customer is a Verizon subscriber, and whether or not that customer has access to the premium content. Some of the additional traffic may have been due to two changes Verizon made in NFL Mobile subscriptions from last season; one was to give free premium access (which lets you see live action) to all customers on a More Everything data plan, and the other was to charge all of those More Everything viewers another $1.99 a month if they wanted to also watch the NFL’s RedZone channel, one of the more attractive features of the app (since RedZone gives you lots of live action, even on games that might otherwise be blacked or not viewable for other reasons). Verizon customers who are not More Everything customers must pay $5 a month for access to the NFL Mobile premium content.

Theoretically, you should be able to order RedZone from this "settings" page of the app. But it didn't work when we tried to click the button.

Theoretically, you should be able to order RedZone from this “settings” page of the app. But it didn’t work when we tried to click the button.

For many NFL Mobile customers, yours truly included, any live video at all was not available for long stretches of Sunday, and after a brief 2-minute “preview” period the RedZone content was also unavailable, and the app was not allowing it to be purchased. As of Monday afternoon, the MSR official testing lab still couldn’t get the RedZone box to work — clicking it kept taking us to a page explaining the charges but not offering a way to add it to our bill.

“NFL Mobile requires a number of backend systems to ensure only Verizon customers can get live NFL video,” said Riethmiller, in a prepared statement. “We experienced issues with one of the systems that validates customers, and it took longer than anticipated to resolve because of tremendous demand. We are confident we have addressed the issue going forward.”

Kristin Rooney, Verizon Wireless director of sponsorships and branded entertainment, would not comment on the NFL Mobile outages in a phone interview Monday morning. However, Rooney was happy to talk about the new features in NFL Mobile this season, including the ability to watch local games live, even if they are out of the traditional prime-time window NFL Mobile has access to. In addition to the local games, NFL Mobile premium customers are supposed to be able to watch live games on Sunday nights, Monday nights and Thursday nights, and also have access to RedZone broadcasts during the day. That is, when the servers are working.

BUY... MORE... SERVERS

BUY… MORE… SERVERS

Another unclear option is what happens to RedZone access when NFL Mobile customers visit NFL stadiums — while Verizon blocks access to RedZone at some stadiums, at others it allows it, usually through a partnership with a team’s stadium app. Rooney, however, did not have a list of stadiums where NFL Mobile customers could watch RedZone broadcasts during a game; most famously, the service was blocked at last year’s Super Bowl in MetLife Stadium.

Rooney also declined to say just how many customers Verizon has for its NFL Mobile app, though the Google Play store says the app has between 10 and 50 million installs. However, since a non-premium version of the app is available to customers with contracts from other carriers, it’s unclear how many “premium” members were overwhelming Verizon’s servers. It’s worth noting that Verizon is no stranger to large numbers of single-day activations, as its systems support more than 100 million wireless subscribers, nation-wide.

In a related note, NFL Mobile wasn’t the only streaming product having issues Sunday. According to a report at TVPredictions.com, the DirecTV streaming service for NFL games also suffered outages Sunday, also for the second straight year.

NFL Mobile users watched 10.3 million video streams during Super Bowl week

Remember the Football on your phone video from last August? Pretty funny, right? Imagine, people wanting to watch football on their phones, no matter where they go. Well last week a whole bunch of them did just that. According to the NFL, more than 10 million video streams were watched on the Verizon NFL Mobile platform during Super Bowl week, a jump of more than 400 percent from the year before.

Football on your phone? You bet!

For some reason we can’t get the league’s media arm or Verizon Wireless to provide the one stat we really want — how many people watched live Super Bowl action on a smartphone? Verizon at least is consistent — they have never provided any kind of statistic on NFL Mobile usage. Today the NFL Media folks issued a press release with all kinds of “record” numbers (we will post the whole thing below if you want to dig through it), but no discrete number for the game itself. We’ll get to the reason for that in a bit. But for the overall stats, we can sum up the numbers quickly: If there was football programming available last week, a lot of people watched it. And if it was available online or to mobile devices, so much the better.

What really drove traffic across all NFL Media properties this year was the NFL Mobile package. Remember, this year the “NFL Mobile from Verizon” app was actually available for smartphones from any carrier; the catch was, you could only get live game action if you were a Verizon subscriber and paid $5 more a month. Everyone else, including Verizon customers, could see highlights and NFL Network video content, like features and reports from Super Bowl week. The opening up of the app is probably the biggest reason why unique users of NFL Mobile properties during Super Bowl week increased 88 percent this year versus last, 11.2 million users compared to 6.0 million, according to NFL statistics.

Here’s where the stats get interesting: While it’s impressive that video streams across all NFL Media properties during Super Bowl week increased 56 percent this year compared to last, 34.0 million to 21.8 million, what’s really mind-boggling is that 10.3 million of those streams this year were consumed via the NFL Mobile app, a 416 percent increase over last year.

To repeat: Almost ONE-THIRD OF ALL NFL MEDIA VIDEO STREAMS were watched last week… on a phone.

One possible reason why neither the league nor Verizon wants to release actual game-day live action viewing numbers for NFL Mobile is that they may not be that impressive. Remember, only Verizon customers who ponied up the $5 extra “premium” fee could have watched the Fox simulcast on their phones, so it’s a smaller subset to begin with. And really, for the big game itself, most likely you were on a couch watching a big screen. (We here at MSR HQ did find the NFL Mobile live feed effective for when we had to roam into the kitchen for more snacks, or for other “breaks” necessary during the game. But we didn’t watch more than a few minutes of the game on the phone.) Plus, the Fox stream was available to tablets using its app or for PCs or laptops watching online, so that probably took away some potential phone-watchers of live game action.

During non-game times, however, smartphones appear to be leading the mobile video explosion. Even though tablets seem to make more sense for watching sports while mobile, it’s pretty clear that people are watching a lot of NFL video on the thing that never leaves their pocket or purse — their phone. Are other sports taking note? And now do you know why the NFL is pushing toward NFL Now? Stay tuned. And keep your phone and checkbook handy.

(full press release content below)

COVERAGE OF SUPER BOWL XLVIII SETS VIEWERSHIP & TRAFFIC RECORDS ACROSS NFL MEDIA

Wall-to-wall coverage of Super Bowl XLVIII produces double-digit spikes for NFL Network, NFL.com, and NFL Mobile

NFL Media’s 11th year covering the Super Bowl produced record-breaking results across all platforms.

NFL NETWORK

Providing expert analysis, the latest news and reports, special guest appearances, and matchup-related programming leading up to Super Bowl XLVIII, NFL Network served up 140 total hours – including a record 86 live hours – of programming from 12 sets in eight locations across New York City and New Jersey, utilizing a roster of 40 on-air personalities with a combined 17 Super Bowl rings.

For Super Bowl week*, NFL Network averaged 175,000 viewers in Total Day and 226,000 viewers in Primetime – up +20% and +31%, respectively, compared to last year making this the most-watched Super Bowl week ever on NFL Network.

With more than 41 hours of Super Bowl coverage, SUPER BOWL LIVE averaged 141,000 viewers and up +60% over last year’s average audience (88,000 viewers).

On Super Bowl Sunday, NFL GAMEDAY MORNING averaged 657,000 viewers – up +10% over last year’s telecast (596,000 viewers), making this the most-watched NFL GAMEDAY MORNING ever.

NFL Network’s postgame audience peaked from 10:30PM-11:00PM with an average of 1.14 million viewers, up + 25% over last year’s highest postgame peak (906,000 viewers)

NFL Network averaged 313,000 viewers in Primetime on Super Bowl Sunday – up +52% compared to last year’s performance.

For the entire 2013 postseason, NFL Network averaged 148,000 viewers in Total Day – up +13% compared to last year’s performance (131,000 viewers) making this the most-watched postseason ever on NFL Network.

NFL DIGITAL MEDIA

Across all internet-connected devices, including PC’s, tablets, and smartphones, the official digital properties of the National Football League delivered original video programming, the latest news and information, as well as unprecedented access to players, celebrities and musicians. NFL Digital Media’s offerings included NFL Mobile from Verizon which provided fans access to live, streaming video of Super Bowl XLVIII and NFL Network, including exclusive Super Bowl content and commercials.

For Super Bowl week, visits to NFL Digital Media properties increased 24% versus last year, bolstered by a 149% increase in visits to NFL Mobile.

Unique users of NFL Digital Media properties rose 22% during the week versus last year (27.8M vs. 22.8M), driven by 88% growth in NFL Mobile Properties (11.2M vs. 6.0M).

Unique users and visits to NFL Digital Media properties were driven by an array of content around Super Bowl XLVII. The two highest performing long-form NFL Digital Media features were:

Ø Judy Battista’s piece on Super Bowl XLVIII’s impact on Peyton Manning’s legacy

Ø Mike Silver’s story on the Seahawks dominant defense being fueled by the tight bond amongst the ‘Legion of Boom’

Video Streams during Super Bowl week across all NFL Digital Media properties increased 56% versus last year (34.0M vs. 21.8M).

Video consumption to NFL Mobile from Verizon reached 10.3M streams for a 416% increase over last year’s Super Bowl Week.

The highest performing video of the week featured NFL Network analysts making predictions for Super Bowl XLVIII. The video was viewed more than 800k times.

Seen in 195 countries, NFL Game Pass consumption grew 49% year-over-year across all devices and 57% on desktop.

*Super Bowl Week is defined as January 27-February 2, 2014

– end press release —

And yes, we know the Football On Your Phone video was a promo for DirecTV’s Sunday Ticket, which is neither here nor there when it comes to Verizon NFL Mobile numbers. But the video is funny enough to watch again:

Fox: Super Bowl had 528K online viewers per minute

If you needed any more proof that online viewing doesn’t hurt TV audience numbers, the stats from Super Bowl should convince you: According to broadcaster Fox, there was an average of 528,000 online viewers per minute, even as the TV audience generated a record 111.5 million viewers.

We are still waiting on a total breakout of online views of the game — we apologize for misreading a Fox release that stated it had 5.5 million unique viewers all day Sunday, NOT specifically for the game. But even the 528K number is impressive, especially when you throw in the fact that it only represents desktops, laptops and Apple iPad tablets, since smartphones were blocked from Fox’s live streams due to Verizon’s deal with the league for its NFL Mobile app. Just imagine how much bigger those numbers could be if and when the NFL makes its digital access methods less confusing.

Overall it looks like Fox did a pretty solid job with its online efforts for both the playoffs and the Super Bowl. We are still waiting to hear from Fox about viewer numbers for the playoff streams, which unlike the Super Bowl were free only to viewers with pay-TV contracts that included Fox channel service.

We are also still waiting for numbers from Verizon and/or the NFL but don’t hold your breath, since both parties are typically close-mouthed about real numbers for online viewing. But judging from the number of pitches play by play man Joe Buck made during the game for the league’s new NFL Now service, it’s a good guess that online viewing is something you’re going to hear a lot more of from the NFL in the not too distant future.

AT&T Super Bowl Sunday stadium wireless traffic jumps 60 percent to 624 GB; Verizon claims 1.9 TB

Inside the AT&T head-end building at MetLife. Cables! Credit: AT&T

Inside the AT&T head-end building at MetLife. Cables! Credit: AT&T

Well, our prediction that this year’s Super Bowl might not set new wireless data records was about as solid as the Denver Broncos’ performance in the big game. According to AT&T, its wireless network in and around MetLife Stadium on Super Sunday saw 624 GB of traffic, a 60.8 percent increase from last year’s 388 GB total at the Superdome in New Orleans.

Part of the gain no doubt came from the fact that MetLife Stadium holds approximately 10,000 more fans during a football game than the Superdome, 82,566 to 72,003. But AT&T’s numerical evidence — including a new peak-time high of 119 GB during the hour before kickoff — shows that the desire to access mobile devices inside stadiums is still growing, especially at big games like the Super Bowl.

Verizon Wireless, which like AT&T spent more than a year getting MetLife’s network ready for Sunday’s game, also said it had experienced record traffic on its networks, claiming a total of 1.9 TB of cellular traffic inside MetLife Stadium. During the game Verizon Twitter accounts said new traffic records were set before halftime; however we still have no figures from Verizon for previous years’ data for comparison.

Verizon also claimed its peak hour usage was “800 percent” greater than last year’s, but again, we don’t have exact numbers so can’t confirm the accuracy of such claims. Still unkown is how much traffic was carried by the stadium Wi-Fi network which Verizon manages (and was free to all MetLife fans), since Verizon did not provide Wi-Fi traffic numbers. Our guess is it saw similar traffic leaps as AT&T’s and Verizon’s cellular nets.

Another interesting note from the AT&T data is a trend toward more mobile data use, and fewer voice calls. Fans at the game Sunday made 55,000 voice calls on AT&T networks, down from 73,000 calls in 2012. Some of that might have to do with the fact that last year’s game had a lengthy power outage, which no doubt prompted many “I’m OK” calls to loved ones. But still — a 20,000 call dropoff in the face of more than doubling data traffic seems to show that people are using their phones more, even if they are talking less.