Biz & IT —

FiOS customer discovers the limits of “unlimited” data: 77TB a month

Average 50TB a month—30,000 percent higher than average—and Verizon will call.

Yes, Virginia, there is a limit to what Verizon will let you do with FiOS' "unlimited" data plan. And a California man discovered that limit when he got a phone call from a Verizon representative wanting to know what, exactly, he was doing to create more than 50 terabytes of traffic on average per month—hitting a peak of 77TB in March alone.

"I have never heard of this happening to anyone," the 27-year-old Californian—who uses the screen name houkouonchi and would prefer not to be identified by name—wrote in a post on DSLreports.com entitled "LOL VZ called me about my bandwidth usage Gotta go Biz." "But I probably use more bandwidth than any FiOS customer in California, so I am not super surprised about this."

Curious about how one person could generate that kind of traffic, Ars reached out to houkouonchi and spoke with him via instant message. As it turns out, he's the ultimate outlier. His problem is more that he's violated Verizon's terms of service than his excessive bandwidth usage. An IT professional who manages a test lab for an Internet storage company, houkouonchi has been providing friends and family a personal VPN, video streaming, and peer-to-peer file service—running a rack of seven servers with 209TB of raw storage in his house.

Just another home network

I asked what exactly he does with that hardware to generate so much traffic. "Lots of stuff," houkouonchi replied. " I do some VPN stuff for people and Web/FTP/SFTP servers. A lot of friends and family stream stuff off me from my huge media collection. And I also do some P2P and Usenet stuff." Most of the storage space is taken up by videos and other media.

He's always had heavy storage requirements—in 2006, houkouonchi said he already had about 8TB of disk with no RAID. "In 2007 I went to 20x 1TB, then not too much later 20x 2TB, and then I started adding more disks and more chassis… but all the chassis that have a lot of hot-swap disk bays are rack-mount. And after having several rackmount machines, I finally decided to just buy a rack when I bought a house and got FiOS back in 2010."

This Verizon FiOS customer's home network generated 77TB of traffic in one month.
This Verizon FiOS customer's home network generated 77TB of traffic in one month.

Here's what's in houkouonchi's personal data center, from top to bottom:

  • A 1u server acting as router and VPN server with 4 1.5TB disks.
  • A 1u testing server with two 1.5TB disks.
  • A 2u server—formerly a "colo box"—with eight 750GB disks
  • A 4u Solaris/ZFS backup machine with 24 1TB disks
  • Another 4u server—houkouonchi's main server with 24 2TB disks and two 3u storage expansion units, each with 15 3TB disks.
  • A 2u "Windows/miscellaneous" server with eight 1TB disks
  • Two 2u uninterruptible power supplies
  • Another 4u Solaris/ZFS server for backups with 24 1TB disks

That's just on premises. Houkouonchi also owns a 2U server running in a colocation facility with 12TB of disk on dual gigabit connections, "which I push quite a bit from as well. It runs game-servers and hosts what used to be the only LA SpeedTest.net server and a bunch of other stuff."

Saving a few bucks

Houkouonchi switched from dual 150 megabit business class connections to a 300 megabit downstream/65 megabit upstream residential plan in January (though he said he was getting 150 megabits upstream). "I only switched to residential simply because business pricing wasn't in line with residential anymore," he said. (Verizon prices the residential service at more than $200 a month for 300/65; business service for dual 150 megabit lines runs $340 a month, and 300 megabit service costs $259 a month.)

Since the switch, he's used nearly 200TB in bandwidth—an average of 50TB a month. However, traffic most months is in the 30TB range. That sort of bandwidth would cost thousands of dollars per month in most colocation facilities—with the 77TB peak month jumping into the tens of thousands based on pricing plans I looked at from a variety of hosting vendors.

Houkouonchi's network traffic—blue is outbound, green is inbound.
Houkouonchi's network traffic—blue is outbound, green is inbound.

But his bandwidth hogging eventually drew the attention of Verizon's engineers, who monitor usage for signs of unusual patterns in traffic. This practice is to watch for both abuse of the network (such as spam and denial of service attacks, for example) and for violations of the FiOS terms of service. Those terms exclude the use of FiOS for "high volume purposes" and forbid customers to "host any type of server. Violation of this section may result in bandwidth restrictions on your Service or suspension or termination of your Service."

Houkouonchi got a call from a Verizon representative this week. "Basically he said that my bandwidth usage was excessive (like 30,000 percent higher than their average customer)," houkouonchi said. "[He] wanted to know WTF I was doing. I told him I have a full rack and run servers, and then he said, 'Well, that's against our ToS.' And he said I would need to switch to the business service or I would be disconnected in July. It wasn't a super long call."

"I don't mind upgrading to business if that's really all the problem is," houkouonchi told me. "It just surprises me they would bother going after people just to get them to pay a little bit more per month. I know that when I switched to GPON (FiOS' Gigabit-capable Passive Optical Network) about six months after a serving hub came up in my area—it took six months fighting with Verizon to get switched over—there was only one other guy on the GPON ports on the serving hub. A lot of people in my area go with Time-Warner because it's cheaper."

Channel Ars Technica