When the lights went off at the Super Bowl during the big game's infamous power outage, not everything went offline.
The stadium-wide Wi-Fi network kept working—or at least it did in some areas and for part of the outage. But it was a very old-school technology—a landline phone—that saved the day for Andrew Stern, a broadcast engineer for a San Francisco radio station.
After coming home from New Orleans, Stern gave me a rundown on how the Super Bowl's technology held up. Was the Wi-Fi network as good as promised? Answer: yes. Did the 35-minute power outage cause problems? Answer: yes—but Stern and team found a way to manage.
The outage set off a "mad scramble" in the press box, Stern said: "It wasn't just the lights—it was everything. I mean everything. There were folks stuck in elevators. The broadcast booth—everybody and everything lost power."
Stern was exaggerating a bit. Amazingly, "the Wi-Fi did not go out," he said. "The telephone circuits did not go out. The network did not go out. It's my working assumption that the network switches were powered off some kind of UPS (uninterruptable power supply) system and the access points are POE (power over Ethernet)." The CBS broadcast stayed up, too—after the game, Stern noticed that "every single TV truck had its own UPS to keep it powered up."
Testing the media's Wi-Fi network on his iPhone 5, Stern consistently got more than 20Mbps down and 40Mbps up, specifically 23 down and 44 up during the game, 28 down and 44 up afterward.
That's impressive enough, but what was really important was a landline. Stern and team's broadcast gear lost power, but the announcer was able to quickly get back on the air with an old-school phone. "We dialed the station to have them take the call and put it on the air," Stern said. The station's announcer "literally hopped on the phone for about a half hour to fill time before the game was resumed."
It was one of those reminders that for all the usefulness of smartphones and wireless Internet access, sometimes the most traditional technology is the best in an emergency. Perhaps the radio crew could have used a cell phone when its standard broadcast equipment lost power, but the call quality on a landline is often better. And the reliability is top-notch, as landlines pull power through phone lines and can work during emergencies.
"Telephone switch rooms are government-mandated that they have banks of batteries to run phone systems for 911 service in case of an emergency," Stern said. "150-year-old POTS (plain old telephone service), regular phone service will keep running during an emergency. This is something that hasn't been figured out by VoIP (Voice over Internet Protocol)."
About that Wi-Fi network
Since we reported the Super Bowl's plan to provide Wi-Fi to up to 30,000 simultaneous users, we've been trying to get a post-game rundown of exactly what kind of usage the network got—and how it performed during the outage. We haven't been able to get any official word from the Superdome and network provider Verizon Wireless yet, but we have some clues.
As noted, Stern found the Wi-Fi performance excellent, even during the outage. However, he was in the press box, so perhaps it was different elsewhere.
Steve Jenson, an ex-Twitter engineer, said he sat in section 116 during the Super Bowl, near the 20-yard-line. "Wi-fi worked for the first 10 minutes or so of the power outage but then went down. I don't remember it coming back online until after the lights came back on fully," Jenson told us. There was also about 10 minutes during the power outage when he couldn't get on the Verizon cellular network, but for the rest of the game it worked fine. It's possible the cellular got congested during that outage with people having more time to use their cellular devices. Additionally, if Wi-Fi was spotty during the outage, the positive impacts Wi-Fi has in offloading traffic from the cellular network would have been lost.
When not watching the game, Jenson said he used the Wi-Fi network to upload photos and view photos on Twitter. He didn't try to access any streaming video, but he said he was "really impressed with the Wi-Fi speed overall." Jenson accessed the network from an iPhone 4S, which can only get on the 2.4GHz Wi-Fi band and not the less-crowded 5GHz band. If his experience was typical, we can assume the network worked well for people regardless of which Wi-Fi bands their devices support.
Another Super Bowl attendee told us on Twitter that the Wi-Fi "worked well in the tunnel/maze on the way in, but was sporadic at best during the game."
Stern mostly used the media Wi-Fi network. Even though the goal was Wi-Fi for everyone, the Superdome might have provided more bandwidth to media members who were working in a dense environment, filing stories, and uploading videos and photos (granted, much of that might have happened over wired connections). Stern did test out the general-access Wi-Fi network used by fans at one point and said it worked just fine, allowing him to use Twitter, Facebook, Internet, and e-mail. "I hopped onto it and it sped right along," he said.
Since we know the Wi-Fi network was at least partially online during the outage, how did that happen? Well, the Superdome and Verizon Wireless haven't told us, but we can make a guess.
All the network gear was made by Cisco. While the wireless access points—all 700 of them—were not battery-powered, a Cisco spokesperson told Ars today that they "draw their power from the switch. So as long as the switch(es) are drawing/have power, the AP’s will have power." The network relied upon two Cisco Nexus 7000 Series switches, so as long as those had some sort of backup power supply, that could explain the Wi-Fi network's resilience during the outage. The power supply modules for the Nexus 7000 switches support "environments with unreliable AC power, with battery backup capability," a Cisco data sheet notes.
(One commenter points out that the access points were likely plugged into additional Power over Ethernet switches distributed throughout the building, meaning the Superdome would have needed a large number of backup power supplies to cover all the switches.)
(Mostly) smooth sailing on game day
As we reported before the Super Bowl, the NFL tightly controlled what devices were allowed into the stadium to prevent interference with the Wi-Fi network. Stern experienced the security firsthand.
Scanning of equipment that came into the stadium happened with the help of the Society of Broadcast Engineers. "They weren't doing any testing on phones or pads or laptops," Stern said. "They were doing frequency coordination with anything that was broadcast gear or any kind of two-way radio, anything that has an antenna on it and looked obvious, or any piece of broadcast gear."
Antennas approved for use got a little sticker denoting they had been cleared:
The only major wireless trouble Stern ran into was in the days leading up to the Super Bowl, in which media outlets were jammed into the nearby conference center.
"There were 150 radio stations there," Stern said. "Every single radio station sets up its own access point and has five or six things connected to it. We're talking a small, dense space and it was right next to the writers wing too, where all the writers were doing their work. It was a dense network and it fell apart."
iOS devices became unresponsive because of the congestion, he said. But on game day, in the Superdome, it was a completely different story.
"All I know is that in the stadium with tens and tens of thousands of people, the system held up great," Stern said. "Whatever they did, they did an amazing job."
reader comments
41