April 25, 2015

Will Periscope and Meerkat swamp stadium networks?

Three thoughts to start your week off, of a completely unrelated nature. First one up is about a couple of live video-streaming services that you might have heard of or seen, Meerkat and Periscope. I successfully avoided watching any super-selfimportant types video themselves using Meerkat from SXSW, and I’ve been too wrapped up in March Madness to care yet about Periscope. So far I haven’t seen any coverage that details how much bandwidth the apps use up. Probably not much if you are livestreaming something all by yourself. But what if a bunch of people decide to livestream, and they’re all in the same place? So I do wonder how stadium networks will handle the idea of live video streams.

Will the Wi-Fi and DAS networks be able to handle the traffic? Anyone looking into this yet? Discuss. You can do so in the comments, or send me some longer thoughts via email and I will relay them to the crowd. Will Periscope and Meerkat be banned in-stadium? If so how can that happen? Will live video streams be the final straw that makes teams and leagues realize that Twitter may not be such a great content partner after all? I don’t have any answers yet but I assure you this is a question that will be asked the rest of the year in stadium IT shops — as well as in the lawyers’ offices where content and TV rights are negotiated and protected. Selfies may be fine, and Vine may be OK. But live streams of sports events are bound to get someone’s attention, fast.

Thought No. 2: Twenty-three years ago, I remember exactly where I was when I saw this:

I was in Beaver Creek, Colo., in a swanky hotel room that I normally couldn’t afford, watching the Duke-Kentucky game after covering pro ski racing during the day on the slopes of Beaver Creek. Because it was near the end of the ski season the still-new Beaver Creek wasn’t too full, so us members of the media got special rates to stay in the slopeside hotel rooms that now will cost you an arm, a leg and maybe a first-born. That is not important to this thought, though. What is important is that I remember watching the game on a nice TV. Which was the only way you could watch, 22 years ago.

Fast forward to Saturday night, when another classic NCAA tournament match involving Kentucky came down to the wire, and a last-second shot, on the exact anniversary of the Laettner shot. That Kentucky prevailed this time in another classic also doesn’t really matter here; what does is how I watched the second half — on my phone in my backyard while cooking dinner on the grill, over a Wi-Fi connection to a router inside the house. The thing I thought about afterwards was how completely normal it seemed to do something that was unthinkable 22 years ago, namely watch a live game via a handheld device through multiple connectivity junctures — and it all just worked. In the future I will probably remember the game more, and the key free throws and the crazy defense of the last play. But right now I’m still a little in wonder in how far the idea of watching sports on your phone has come.

Third thought: Some more history here — does anyone out there remember the 2009 version of SXSW, when Foursquare was launched and the huge influx of attendees using Twitter on their iPhones brought the AT&T network to its knees? Here’s another link to the historical moment when AT&T got pantsed publicly for not knowing how much bandwidth its customers would need at a gathering like SXSW.

Fast forward again to this year’s SXSW, and man, was AT&T ready for record network usage. Not only did it trot out the huge big-ball cellular antenna that it used at Coachella last year, it beefed up regular network connections and brought in a whole herd of COWs (cell trucks on wheels) to satisfy a mobile bandwidth demand that doesn’t seem to be able to stay flat or go down. According to AT&T, its network saw 37 terabytes of data used during the SXSW event — that’s like three-plus Super Bowls worth of traffic, and this is just on AT&T’s networks, so not counting other carrier traffic.

We concentrate a lot here on stadiums and the particular problems for wireless communications caused by a tight geographic grouping of device-holding people. But what about towns with festivals like SXSW, or other big gatherings? Is your event ready for massive wireless bandwidth needs? If not what is your plan going forward?

YinzCam’s Super Bowl stadium app will have instant replays, Super Bowl commericals, stadium maps and more

Screen shot of Super Bowl app for this year's game.

Screen shot of Super Bowl app for this year’s game.

We’ve been waiting for official word on what the YinzCam-developed app for the Super Bowl will look like, and though there’s no press release the page where we are guessing it will eventually be available is offering some details, like the availability of instant replays from different camera angles, video of Super Bowl commercials, and stadium maps.

On the Seahawks.com site we found a good how-to story for fans going to the game, which included a link to this page, where we are guessing the Super Bowl stadium app will be available for download. Here is the boilerplate:

New for Super Bowl XLIX, the Super Bowl Stadium App Presented by Verizon aims to take the fan experience inside University of Phoenix Stadium to the next level. Features that will enhance Super Bowl ticketholders’ experiences include exclusive in-stadium video content such as Super Bowl commercials and replays from four different camera angles, stadium seating and concession maps, once-in-a-lifetime gameday opportunities visible only to fans inside the stadium and the option to receive up-to-the-minute gameday notifications. Available on iOS, Android and Windows. Goes Live 23rd January 2015

(Looks like the app is already available in the App Store and in Google Play, but nothing is live; we downloaded the app and the only three buttons available, for highlights, commercials and memories, all say they will be available on Feb. 1 at the stadium, so no idea what the “goes live” on the splash page above means yet.)

Screen Shot 2015-01-23 at 11.40.41 PMYinzCam founder and CEO Priya Narasimhan had told us earlier this year that a Super Bowl app was in the making, and apparently it will contain features found in some of the latest YinzCam app deployments, such as the Seattle Seahawks’ new stadium app, which has multiple camera angle replays. The Super Bowl app is different from the Arizona Cardinals’ regular stadium app, which was also built by YinzCam, which also features instant replays.

We were able to download the app for iPhone (it’s free) and apparently you will need to be connected to the stadium Wi-Fi (which has the clever SSID of “Stadium WiFi”) in order to view highlights and other video options.

The good thing for fans at the big game, there will be plenty of networking horsepower to keep the app running, no matter where you are. If you’re inside the stadium there is a new Wi-Fi network and a refurbished DAS deployment to keep fans connected; stay tuned next week for our big breakdown of DAS deployments and carrier plans to keep the Super Bowl crowds super-connected.

College championship game at AT&T Stadium breaks 6 Terabyte wireless data mark, with almost 5 TB of Wi-Fi traffic

AT&T Stadium before the college football playoff championship game. (Click on any photo for larger image) Credit all photos: Paul Kapustka, MSR

AT&T Stadium before the college football playoff championship game. (Click on any photo for larger image) Credit all photos: Paul Kapustka, MSR

Not only did Monday night’s College Football Playoff championship game crown a new new national title team — it also broke the unofficial record for most wireless traffic at a single sporting event, with more than 6 terabytes of data used by the 85,689 fans in attendance at AT&T Stadium in Arlington, Texas.

John Winborn, chief information officer for the Dallas Cowboys, said the AT&T-hosted Wi-Fi network at AT&T Stadium carried 4.93 TB of traffic during Monday’s game between Ohio State and Oregon, a far higher total than we’ve ever heard of before for a single-game, single-venue event. AT&T cellular customers, Winborn said, used an additional 1.41 TB of wireless data on the stadium DAS network, resulting in a measured total of 6.34 TB of traffic. The real total is likely another terabyte or two higher, since these figures don’t include any traffic from other carriers (Verizon, Sprint, T-Mobile) carried on the AT&T-neutral host DAS. (Other carrier reps, please feel free to send us your data totals as well!)

The national championship numbers blew away the data traffic totals from last year’s Super Bowl, and also eclipsed the previous high-water Wi-Fi mark we knew of, the 3.3 TB number set by the San Francisco 49ers during the opening game of the season at their new Levi’s Stadium facility. Since we’ve not heard of any other event even coming close, we’ll crown AT&T Stadium and the college playoff championship as the new top dog in the wireless-data consumption arena, at least for now.

University of Phoenix Stadium, already with Super Bowl prep under way

University of Phoenix Stadium, already with Super Bowl prep under way

Coincidentally, MSR on Tuesday was touring the University of Phoenix Stadium and the surrounding Westgate entertainment district, which is in the process of getting the final touches on a new complex-wide DAS installed by Crown Castle. The new DAS includes antennas on buildings and railings around the restaurants and shops of the mall-like Westgate complex, as well as inside and outside the UoP Stadium. (We’ll have a full report soon on the new DAS installs, including antennas behind fake air-vent fans on the outside of the football stadium to help handle pre-game crowds).

The University of Phoenix Stadium also had its entire Wi-Fi network ripped and replaced this season, in order to better serve the wireless appetites coming for the big game on Feb. 1. At AT&T Stadium on Monday we learned that the network there had almost 300 new Wi-Fi access points and a number of new DAS antennas installed since Thanksgiving, in anticipation of a big traffic event Monday night. Our exclusive on-the-scene tests of the Wi-Fi and DAS network found no glitches or holes in coverage, which is probably part of the reason why so many people used so much data.

UPDATE: Here is the official press release from AT&T, which basically says the same thing our post does.

Stadium Tech Report: AT&T Stadium network a winner at CFP Championship game

Inside AT&T Stadium at the College Football Championship game. Credit all photos: Paul Kapustka, MSR

Inside AT&T Stadium at the College Football Championship game. Credit all photos: Paul Kapustka, MSR

It’s late here in North Texas and you know by now that Ohio State beat Oregon to win the first non-mythical college football championship. Behind the scenes at AT&T Stadium Monday night, the wireless network in AT&T Stadium was also a winner, standing up to the challenge of the 85,000-plus crowd on both the DAS and Wi-Fi front.

We’ll have a more thorough stadium report when we get time to digest all the info we gathered at the game (and get the network stats back from the AT&T Stadium tech crew) but one thing we learned before the game was that since November, the Wi-Fi network at AT&T Stadium grew by more than 280 access points, on top of a total somewhere in the 1,200 range. According to AT&T network folks the stadium here in Arlington, Texas, has been seeing game-day totals of 3.3 Terabytes of data carried on the Wi-Fi network — leading some here to believe that Monday’s championship game could well surpass 4 TB of data used at a single game, an unofficial record as far we know for a single-day, single facility network.

As guests of AT&T we also got a quick demonstration of LTE broadcast technology, which basically slices the available cellular spectrum into a channel that can provide live streams of video. We’ll have more on this new technology in another separate report, but it is something to watch for facilities that want video options but don’t want to go whole hog on Wi-Fi.

AT&T LTE Broadcast demo, showing a live streaming broadcast of the game

AT&T LTE Broadcast demo, showing a live streaming broadcast of the game

Even though we were housed in a field-level suite your intrepid MSR crew wandered all over the massive facility, and basically found great connectivity wherever we were. Two places stick out in my mind: At the very top of the nosebleed section in the south end zone the Wi-Fi dipped to just 1 Mbps, probably because the roof is so high there is no place for an access point. However, at that same spot the AT&T 4G LTE signal was around 7 Mbps, providing great connectivity in a tough to configure spot.

The other notable spot was in a “star level” suite (about the 6th level of the building), where we got a Wi-Fi signal of 28 Mbps download and 59 (no typo!) Mbps on the upload. Yes, suite people have it better but all around wherever we went we got consistent Wi-Fi signals in the high teens or low 20s, and LTE cellular signals (including Verizon 4G LTE) just under 10 Mbps. Like the Ohio State offense, the network at AT&T Stadium works really well and may have set a new record Monday night. More soon, and more images soon as well. For now, Elvis has left the building.

Outside in the frozen tundra of North Texas, aka Arlington

Outside in the frozen tundra of North Texas, aka Arlington

This place was humming all night long

This place was humming all night long

AT&T 4G LTE speedtest, from the top of the stadium

AT&T 4G LTE speedtest, from the top of the stadium

The view from the nosebleed section

The view from the nosebleed section

Some "suite" Wi-Fi speeds

Some “suite” Wi-Fi speeds

Stadium Tech Report: How VenueNext generates replays for Levi’s Stadium app

replay2During the final home game for the San Francisco 49ers this season, I was somewhat amazed to see a replay appear on the Levi’s Stadium app even before the next play had concluded. Clearly, the VenueNext team behind the app had progressed significantly from the early season, when the app’s replay function struggled somewhat.

During the Dec. 28 game against the Arizona Cardinals, the VenueNext team invited Mobile Sports Report to Levi’s Stadium for a behind-the-scenes look at how the replays are generated; here’s a quick take on how the video moves from playing field cameras to fans’ phones in a matter of seconds.

Step 1: Capturing what the cameras see

How do you make sure you have replays ready? You start by basically capturing all the action from all the cameras in the building. Those views are shuttled back to the main Levi’s Stadium video room, where 15 people have the somewhat enviable job of watching views from all the live cameras in the stadium. (We still haven’t heard a response to our offer of bringing a cooler in exchange for a spare seat next season.)

Inside the video room at Levi's Stadium. (Click on any photo for larger image) Credit all photos: Paul Kapustka, MSR

Inside the video room at Levi’s Stadium. (Click on any photo for larger image) Credit all photos: Paul Kapustka, MSR

According to VenueNext CEO John Paul, who led our tour at Levi’s, VenueNext has its own servers to cache video — here is a look at the Elemental gear used to store the video for the app’s replay function. Paul said that so far, VenueNext has only needed a little more than half of the capacity of the four servers pictured.

Elemental gear in data room next to video room at Levi's Stadium

Elemental gear in data room next to video room at Levi’s Stadium

Step 2: Labeling and thumbnailing the replay

While most of this kind of gear is not new to broadcast operations, the interesting part from an app perspective is the next layer, the choosing and human editing. One of my bigger questions was about how you exactly get a replay set up, labeled with a headline and a thumbnail, and into the app within seconds? The answer is: You write another app, and have two Android-based tablets with football-savvy operators in the press box so they can see the plays live and quickly attach the appropriate info to the replay clips.

In the two pictures below, you can see an over-the-shoulder view of the two tablet apps; the first one is a sort of play-by-play generator, which the operator uses to label the next replay clip as either a run, pass, punt, incomplete, etc., so that a fan looking at the app can quickly figure out what the play might be (“45-yard TD pass,” or something like that). On the second tablet app, another operator sees instant thumbnails gleaned from the video room from all the different camera angles, and when the play is over, that operator picks a thumbnail with a click and the replay is on its way to the app.

First replay tablet app, which adds info about the play

First replay tablet app, which adds info about the play

Second replay tablet app, which adds a thumbnail to the replay

Second replay tablet app, which adds a thumbnail to the replay

Since we only had a few minutes to watch the replay team in action (and since Paul was simultaneously hosting a group of representatives from a prospective future sports-team client), I didn’t get to ask detailed questions about how this all works but it does give you some idea. What’s impressive about the Levi’s Stadium app is that it offers four different camera angles of the same play, as well as a multi-cam view that shows all different angles at once. Though I’m not a fan of the all-at-once option I did find it enjoyable to watch a reply from multiple different perspectives, like watching a pass being caught from both a sideline and end-zone view.

Step 3: Educating fans how to use replay

Initially one of the most-hyped features of the planned stadium app, the replay function has seen only limited use this season, making it somewhat of a disappointment to Niners CEO Jed York. From our perspective, there were two big things holding back extended use of the replay function: The first was the ever-changing appearance of the app itself, which felt like a beta project for most of the season, with a different interface almost every single game; the second thing holding people back from using the app to view replays are the two excellent humungous video boards at Levi’s, which not only show live game action but also quickly show replays, sometimes from those multiple angle views as well.

There might even be a third thing holding fans back from trying out the replay function — that would be just getting fans to use the stadium app in the first place. While usage of the cellular and Wi-Fi networks started at record levels and remained strong from game to game, most fans at Levi’s Stadium were using their devices for other apps, like Twitter, Instagram, Facebook and good old email. As the season progressed, the team started promoting the app and the network more aggressively, adding more-frequent messages to the main video boards and dressing its stadium tech support crew (the “NiNerds”) in neon vests for easier recognition.

What’s next for next season? My guess is that with a year under their belt, the Niners and VenueNext will have a more-aggressive campaign in 2015 to steer more fans to the replay function, which is truly a wonderful enhancement to the game-day experience.

All NFL playoff games available online; Verizon only for smartphone watching

vzn_playoffWe’ve come a long way from the days when it was a struggle to even find NFL live action online. This season, all NFL playoff games, including the Super Bowl, will be available for online viewing, via a desktop computer, laptop or tablet, no matter which network is carrying the games. The league has even created a handy single web page to find instant access to the live streams, no small matter since network pages usually make you jump through several clicks to find the actual live stream.

Depending upon the broadcast network, you may need to have a qualifying cable or satellite contract to view the games. For this morning’s CBS game there was no confirmation process but for the afternoon Fox game I needed to submit cable provider info. Let me know what you see.

However, if you want to watch playoff games on your smartphone, your only option is to be a Verizon Wireless customer since that provider is the exclusive NFL live action host for smartphone devices. To view the live action, Verizon customers need to have either a More Everything plan or pay the $5 monthly premium fee for the NFL Mobile application. To be clear, customers of any wireless provider can download the NFL Mobile app to get all the stats and archived video it provides, but only Verizon customers can purchase access to live action on phone platforms.

UPDATE: Congrats to FOX for having the first playoff online broadcast fail:

Screen Shot 2015-01-04 at 3.01.07 PM

No word from Fox yet on when its stream will come back, or why it was down.