DAS, Wi-Fi get a workout at NCAA men’s hoops venues

AT&T cell tower on wheels (COW) with an ‘eyeball’ antenna

Stadium DAS and Wi-Fi networks at venues hosting NCAA men’s basketball tournament action saw plenty of usage the past two weeks, confirming that in-stadium wireless device use continues to grow.

According to AT&T, fans on its networks across all NCAA men’s venues used a total of 11.6 terabytes of data, with Bankers Life Fieldhouse in Indianapolis seeing the most of any single venue, with 1.8 TB of data used in games there from March 17-19. The Amway Center in Orlando was close behind, with 1.075 TB used on the AT&T DAS networks for games on March 16 and 18.

On the Wi-Fi side, San Jose’s SAP Center saw almost 1.7 TB of data used this past weekend for regional finals games. Allison Aiello, director of IT for Sharks Sports & Entertainment, said the venue saw 1.1 TB of data used for Thursday’s two games, with a peak concurrent user number of 3,041 and a peak bandwidth utilization of 1.1 Gbps. On Saturday Aiello said the SAP Center’s new Wi-Fi network saw 592 GB of data used, with a peak concurrent user total of 2,802 and peak bandwidth utilization of 451 Mbps. (Any other venues or carriers with stats, send them in and we will update this post.)

AT&T also said that it has — once againupdated its coverage in and around the University of Phoenix Stadium in Glendale, Arizona, home of this year’s men’s Final Four. With a 60 percent increase in capacity for the stadium’s DAS and additional equipment for other Phoenix venues and outside areas, AT&T continues to invest in infrastructure for what it sees as a still-climbing demand for bandwidth at sporting events. According to AT&T:

Over the last five NCAA Men’s Final Four tournaments (2012-2016), total data usage on AT&T’s venue-specific mobile network alone has increased more than 900 percent, setting records on our network year after year. Next weekend’s tournament will surely be no different.

Comcast brings new 3-Gig backbone to Memphis Grizzlies’ FedExForum

Comcast Business, which has sponsored the backbone bandwidth to many sports arenas, announced that it has installed a new 3-gigabit fiber backbone to the NBA’s Memphis Grizzlies’ FedExForum, which will support the already-existing fan-facing Wi-Fi network at the stadium.

According to our past research, the FedExForum Wi-Fi used to be supported by a wide-area wireless link provided by Ubiquiti, the firm that Grizzlies owner Robert Pera is CEO of. We are guessing here but we suspect that the fan-facing Wi-Fi will soon have a new SSID name of xfinitywifi, the SSID used by Comcast in other arenas where it provides backbone services.

Though we haven’t updated our specific information since our original report, news reports today claiming that Comcast is supplying Wi-Fi to FedExForum for the first time are incorrect. According to Comcast, under the new deal Comcast will also provide internet service and other communications services for the team’s front office operations.

A building for the future: Tech shines through at Sacramento’s new Golden 1 Center

Golden 1 Center, the new home of the Sacramento Kings. Credit: Sacramento Kings

If you’re building for the future, it’s best to start with a building for the future.

That’s what has happened in downtown Sacramento, where the Sacramento Kings have built a technology-laden future-proof arena, a venue designed not just to host basketball games but to be the centerpiece of a metro revival for years to come.

Now open for business, the Golden 1 Center is a living blueprint for the arena of the future, especially from a technology perspective. And while some technology inside the venue is impossible to ignore — starting with the massive 4K scoreboard that overhangs the court — there’s also a wealth of less-apparent technology woven throughout the building’s core and pervasive in its operating functions.

Led by Kings majority owner and former software company founder Vivek Ranadive, the technology-focused direction of the new arena is a blend of the latest thinking in venue experiences and operations. Among the many got-to-have staples: High-quality wireless connectivity and multiple mobile device-based services, including food ordering and delivery, map-based parking, wayfinding help, and digital ticketing. While its already-available options easily place Golden 1 Center among the top tier of connected stadiums today, what may be more impressive is the internal planning for future technologies and services, a sign that its owners and operators clearly understand the ever-changing nature of digital systems.

The purple lights are on in the Golden 1 Center data room. Credit all following photos: Paul Kapustka, MSR (click on any photo for a larger image)

While the arena is open today, it’s still somewhat of a diamond in the rough, as planned surrounding structures, including adjacent hotel and retail outlets, are still in the concrete-and-cranes phase, with “coming soon” signs on the area’s many construction fences. As they wait for their team to show signs of on-court improvement, Sacramento citizens must also be patient for the full plan of the downtown arena to emerge, along with its promise to revive an area once stuck in the past.

The good news? With Golden 1 Center Sacramento fans already have a winner, in a venue that will provide fans with some of the best digital-based services and amenities found anywhere, for now and for the foreseeable future. What follows are our first impressions from an early December 2016 visit to a Kings home game, hosted by representatives of the Kings’ technical staff along with representatives from Wi-Fi gear provider Ruckus and cellular DAS deployment firm DAS Group Professionals.

Showing off the data center

Editor’s note: This profile is from our latest STADIUM TECH REPORT, the ONLY in-depth publication created specifically for the stadium technology professional and the stadium technology marketplace. Read about new networks at the Indiana Pacers’ Bankers Life Fieldhouse and the new Wi-Fi network used for the Super Bowl in our report, which is available now for FREE DOWNLOAD from our site!

Data center guards. Small, but well armed.

If you had any doubts about how proud the Kings are of their stadium technology, those are erased the moment you enter the stadium via the VIP doorway; after the metal detectors but before you hit the new-wave ticket scanners, you see a set of floor-to-ceiling glass walls and doors to your left, showing off the impressive racks of the venue’s main data equipment room.

How can gear racks be impressive? How about if they are impeccably encased in their own white metal and glass enclosures, a technique that allows the Kings to refrigerate each rack separately, leaving the rest of the room at a temperature more suitable to human bodies. You don’t have to be a network equipment operator to recognize an over-the-top attention to detail here; even the exposed fiber cabling that stretches out up and across the ceiling is color-coded in the Kings’ main team purple; another level of coolness appears when the main lights in the room are turned off, and more LEDs come on to bathe the room in a completely purple hue.

This room is also where you see the first hints of how the team is preparing for the future, with two 100 Gbps incoming bandwidth pipes (from Comcast), as well as two full rows of racks left empty, waiting for whatever innovation needs arise next. While the backbone bandwidth will eventually also support the nearby hotel and retail locations, twin 100-Gbps connections should provide adequate throughput for now and the foreseeable future.

Walk a few steps past the mini-sized Imperial Stormtroopers who guard the facility and you are in a hallway that separates a “mission control” room with monitors for a huge number of operational services, and the video control room. The innovation here starts simply with the side-by-side proximity of network, operations and video administration rooms, a rarity especially in older stadiums where coordination between people working in such rooms often meant walkie-talkies and lots of running around.

Multiple live video inputs in the “control room” at G1C.

While the video control room and its need to supply coordinated content to more than 800 monitors in the building (as well as to the app) is impressive, what’s really interesting is the “mission control” room, where Kings employees, network types and public safety personnel can track multiple inputs on a wall of monitors. In addition to security and public service video monitoring (Kings reps talk about seeing fans spill a drink and hustling to deploy clean-up services before anyone can ask for them), there are also displays for real-time social media mentions and live traffic information, which the Kings can monitor and respond to as needed.

Another “unseen” technology innovation is an operational app that provides real-time access to a huge list of game-day business statistics, like live ticket-scan numbers and real-time updates to concession purchases. This app is also available to Kings execs on their mobile devices, and it’s addicting to watch the numbers update in real time, especially the fast-moving alcoholic beverage purchase totals; according to the Kings, during a Jimmy Buffett concert at the arena, adult-beverage purchases were pushing the $1,000-per-minute mark.

When it comes to the fan experience, such “hidden” technologies may be the services that provide the best examples for how high-quality networks can bring real ROI to stadiums and large public venues. Fans may never know the guts of the system, but when a stand doesn’t run out of hot dogs or a clean-up squad arrives quickly to mop up a spilled beer, it’s a good bet that customer satisfaction will keep increasing. With massively connected systems and attached real-time analytics, such services become easier to deploy and manage; at Golden 1 Center, it’s easy to see how multiple stakeholders in the venue benefit from the decision to make networked technology a primary core of the building’s operations.

The huge scoreboard dominates the view at Golden 1 Center.

A scoreboard that stretches from hoop to hoop

Taking an elevator up to the main concourse floor, the initial impression of Golden 1 Center is its openness — it is built so that the main or ground level entrance is at the top of the bottom bowl of seats, with court level below. Open all the way around, the ability to see across the venue gives it an airy feeling, more like a bigger enclosed football stadium than a basketball arena. On the night we toured the venue its unique glass entryway windows were closed, but they can be opened to let in the breeze during milder days — adding another degree of difficulty for wireless network administration, since LTE signals can both enter and leave the building when the windows are open.

The next thing to catch your eye is the main scoreboard, which the Kings bill as the biggest 4K screen for a permanent indoor arena, with 35 million pixels. If it were lowered during a game, the Kings folks claim the screen would touch both baskets, so without any other numbers you get the idea: This thing is huge.

New entry kiosks from SkiData move more fans inside more quickly, the Kings claim.

It’s also incredibly clear, thanks in part to the 4K resolution but also in part to the fact that it is tilted at just the correct angles so that it’s easy to glance up from live action for a look at either the main screens or the bordering screens on both sides. Just citing clarity or size for scoreboards, I think, is missing a critical factor for video boards — what really matters is whether or not the screen is a positive or negative factor for during-game viewing, a subjective measurement that may take time to sink in. First impressions, however, during the live action between the Kings and Knicks during our visit, were incredibly positive, with the screen not interfering with live action views but incredibly clear for replays and live statistics.

The next part of our tour was to see if we could spot any of the 931 Ruckus Wi-Fi APs that are installed inside the venue. With the clear emphasis on clean aesthetics it was hard to spot any of the wall- or ceiling-mounted units, but we were able to locate several of the many under-seat AP enclosures, including some on retractable seats. According to the Ruckus folks on hand the retractable-seat APs took a little extra engineering, to allow the devices to be disconnected during seat movements.

The JMA Wireless DAS equipment was a little easier to spot, since like at Levi’s Stadium there are a number of antenna placements around the main concourse, pointing down into the lower bowl seating. The DAS Group Professional representatives on hand also pointed out more antennas up in the rafters, as well as some specially designed “antenna rocks” that hide cellular equipment outside the stadium in the open-air plaza. According to DGP and the Kings there are 136 DAS remote placements housing 213 antennas; right now only AT&T and Verizon Wireless are active on the DAS, with T-Mobile scheduled to join before the end of the NBA season. Negotiations with Sprint are still under discussion.

Blazing Wi-Fi in the basement of the building… and the rafters

When we dropped back down to the court-level to see the locker room entrances and one of the premium-seat club areas, we took our first Wi-Fi speed test at Golden 1 Center, and almost couldn’t believe the result: We got 132 Mbps for the download speed and 98 Mbps for upload. Greeted a few minutes later by owner Ranadive himself, we congratulated him on getting what he wanted in terms of connectivity, a theme he relentlessly promoted during the arena’s construction phases.

That’s good Wi-Fi. Taken in the Lexus Club on court level at Golden 1 Center.

The Wi-Fi connectivity was superb throughout the venue, with readings of 51.35/22.21 on press row (located at the top of the main lower bowl, just in front of the main concourse) and 42.14/38.83 in the crowded Sierra Nevada brewpub club at the top level of the arena. In section 220 in the upper deck we got Wi-Fi readings of 53.39 Mbps for download and 36.27 for upload. Throughout the stadium the Verizon LTE signal was in low teens to 20 Mbps range on the download side and usually between 20-30 Mbps on the upload side.

One of the decisions the Kings made on the Wi-Fi side was to drop 2.4 GHz coverage for fan devices in the main bowl area. According to both Ruckus and the Kings, fan devices now are almost 90 percent 5 GHz capable, meaning that it makes administrative sense to take 2.4 GHz out of the main fan Wi-Fi equation (while still keeping it for back-of-house operations like POS and wireless wristbands and cameras, which all still use 2.4 GHz technology). Other teams in the NBA, including the Indiana Pacers (who also recently installed a Ruckus Wi-Fi network) have also said that they are getting rid of 2.4 GHz coverage for fans since most devices used today have 5 GHz connectivity.

While we didn’t have time during this visit to explore all the numerous services available through the team’s app — including a game that lets fans bet loyalty points on predictions about which players will score the most points — it was clear that many fans were taking advantage of the connectivity, especially in the brewpub area where handy lean-up railings with small shelves made it easier to operate a mobile device while still being somewhat engaged with the court action below.

Team execs can get live feeds of fan-related stats on their internal app.

According to the Kings, during the first regular-season home game on Oct. 27, 2016, there were 8,307 unique users of the Wi-Fi network, out of 17,608 fans in attendance. The connected fans used a total of 1.4 terabytes of data on the Wi-Fi network that night, with a top peak concurrent connection number of 7,761 users. The highest sustained traffic to the Internet that night was a mark of 1.01 Gbps for a 15-minute period between 7:45 to 8:00 p.m., according to the Kings.

Another technology twist we saw in the brewpub was the use of Appetize’s flip-screen POS terminals, which allows for faster order taking simply by letting fans sign on screens with their fingers. Back at the front gates, the new ticket-scanning kiosks from SkiData may take some time for fans to get used to, but even obvious first-timers seemed to quickly understand the kiosk’s operation without much help needed, thanks to the helpful instructions on the wide screen that greets fans as they encounter the device. According to the Kings, tests of the new kiosks at other venues have shown that they can be as much as three times faster than previous technologies, good news to anyone who’s ever had to wait in line just to have their ticket checked.

A building for the future, whenever it comes

While we here at MSR clearly focus on venue technology, it was clear even during our brief stay at Golden 1 Center that while Sacramento fans may be immediately enjoying the amenities, they are still first and foremost concerned about the product on the court. In the upper deck two men spent several minutes questioning why Kings star DeMarcus “Boogie” Cousins (who has since been traded to the New Orleans Pelicans) didn’t seem to get the kind of refereeing treatment alloted to other NBA leaders; on an escalator another fan interrupted one of my speedtests by loudly requesting a fan-to-fan fistbump while he simply said, “Kings basketball, right baby?”

A view outside the stadium’s main entrance, with one of the two large vertical video boards visible.

Even in the face of mulitple years without playoff teams, Sacramento fans still turn out for the Kings; the point here in regards to technology is that it may take time for fans to notice and embrace the finer points of all the technological attributes of their new arena, which should become more than just an NBA venue as more concerts and civic events are held in and around its environs.

Our quick take is that fans may turn faster to services like the traffic, parking and seat-wayfinding features in the app, simply due to the newness of the building to everyone, as well as its tightly sandwiched downtown location. Like in other new arenas, the jury is still out on other app-based services like the loyalty-points voting game, and in-seat concessions ordering and delivery; the Kings declined to provide any statistics for in-seat ordering and delivery, a service which became available to the entire stadium on the night of our visit. The Kings, like many other teams, also offer instant replays via the app, but with the numerous high-quality big-screen displays (including two arena-sized screens outside the main entryway) it will be interesting to see if fans ever see an overwhelming need to check their devices for live action while attending a game.

The good news for the Kings is that they based their stadium and team app on a new flexible platform from a company called Built.io, which the Kings say allows for easier addition (or deletion) of services through an API layer. Like the future-proof parts of the building itself, the app also shows the Kings’ dedication to building something now that will almost certainly change going forward. As we look to the future it will be interesting to see which parts of the technology base contribute most to the fan experience and business operations at Golden 1 Center — and to see how many other existing or new arenas follow the lead.

More photos from our visit below!

Under seat Wi-Fi AP on a moveable section of stands.

The view from upper-deck seats.

A Wi-Fi speed test from those same seats.

One of the “rocks” hiding DAS antennas on the outside walkway.

Extreme bids $100 million for Avaya’s networking business

In a move that could expand its reach into the stadium networking business, Extreme Networks said Tuesday that it had entered into an “asset purchase agreement” to acquire the networking business of Avaya, which recently filed for bankruptcy. The price of the deal is $100 million, which is still subject to the possibility of other companies submitting higher bids for Avaya’s networking business assets.

We have calls in to get some more information, such as what might possibly happen to Avaya stadium networks in venues like Avaya Stadium, Montreal’s Bell Centre and Denver’s Pepsi Center. More as we hear more.

College Football Playoff championship sees 2.4 TB of Wi-Fi — big decline from 2016

We finally have numbers for the Wi-Fi usage at the most recent College Football Playoff championship game, and in somewhat of a first the total data used during the event was much lower than the previous year’s game, with just 2.4 terabytes of data used on Jan. 9 at Raymond James Stadium in Tampa, Fla. — compared to 4.9 TB of Wi-Fi used at the championship game in 2016, held at the University of Phoenix Stadium in Glendale, Ariz.

The reason for the decline is probably not due to any sudden dropoff in user demand, since usage of in-stadium cellular or DAS networks increased from 2016 to 2017, with AT&T’s observed network usage doubling from 1.9 TB to 3.8 TB in Tampa. More likely the dropoff is due to the fact that the Wi-Fi network at the University of Phoenix Stadium had been through recent upgrades to prepare for both the college championship game and Super Bowl XLIX, while the network in Raymond James Stadium hasn’t seen a significant upgrade since 2010, according to stadium officials. At last check, the Wi-Fi network at University of Phoenix Stadium had more than 750 APs installed.

Joey Jones, network engineer/information security for the Tampa Bay Buccaneers, said the Wi-Fi network currently in use at Raymond James Stadium has a total of 325 Cisco Wi-Fi APs, with 130 of those in the bowl seating areas. The design is all overhead placements, Jones said in an email discussion, with no under-seat or handrail enclosure placements. The total unique number of Wi-Fi users for the college playoff game was 11,671, with a peak concurrent connection of 7,353 users, Jones said.

Still tops among college playoff championship games in Wi-Fi is the first one held at AT&T Stadium in 2015, where 4.93 TB of Wi-Fi was used. Next year’s championship game is scheduled to be held at the new Mercedes-Benz Stadium in Atlanta, where one of the latest Wi-Fi networks should be in place and operational.

Seahawks see big jump in Wi-Fi usage at CenturyLink Field for 2016-17 season

Screen Shot 2016-09-12 at 1.11.24 PMThe Seattle Seahawks saw almost every metric associated with the Wi-Fi network at CenturyLink Field just about double from the 2015-16 to the 2016-17 NFL regular season, according to statistics provided by the team.

Chip Suttles, vice president of technology for the Seahawks, sent us over some excellent season-long charts of Wi-Fi activity, including unique and concurrent-user peaks, top throughput, and a couple of comparison charts mapping this most recent season’s activity compared to that a year before.

With a capacity crowd attendance total of 69,000, the always sold-out CenturyLink saw a take rate nearing 50 percent for most of the season, with a top unique-user number of 35,808 for a Nov. 7 31-25 win over the Buffalo Bills. Interestingly, the biggest day for total data usage wasn’t the Bills game (3.259 terabytes) but a 26-15 win over the Philadelphia Eagles on Nov. 20, when the Wi-Fi network saw 3.526 TB of data used. If you look at the comparitive graphs, both peak usage and total usage numbers pretty much doubled down on what was seen the year before.

According to Suttles, there wasn’t much in the way of upgrades to the Extreme Networks-based network before this past season — just some firmware and software updates, and “about a half-dozen” new APs to support additional seating added in the south end zone area. “Overall, I think it [the data increases] is more to do with familiarity,” Suttles said. Thanks to Chip and the Seahawks for sharing the numbers!

sea2

sea1