Verizon disrupted part of the internet due to misrouting of traffic
Verizon disrupted part of the global internet Monday morning as the American company routed traffic to the networks of an Internet provider and metal company in the state of Pennsylvania. However, these networks could not handle this traffic.
According to Cloudflare, the internet problems started with an internet provider in the US state of Pennsylvania, DQE Communications. This company used a bgp optimizer from Noction, which can optimize network traffic thanks to redirects. This optimizer forwarded the routing to a customer of DQE, a metal company in the same state. DQE Communications then also automatically forwarded the routing information to its transit provider Verizon, which then announced the “optimum route” to the Internet. As a result, Verizon, Allegheny, and DQE had to deal with more traffic than they could handle.
As a result of the incident, Cloudflare said it had fifteen percent less worldwide internet traffic than normal. According to BGPmon.net, 20,000 prefixes for 2,400 networks via the metal company’s network and Facebook and Amazon went offline due to the outage. Discord also suffered from the outage.
In the end, the routing issues were resolved as Cloudflare contacted DQE Communications. DQE then ensured that the ‘fast routes’ were no longer communicated via the steel company’s network to other network providers.
Cloudflare points the finger of blame at Verizon in its blog post and says there are multiple ways to stop such network traffic routing errors. For example, bgp sessions can be given a maximum of prefixes, which means that such a session will be stopped when that maximum is reached. IRR-based filtering could also have worked, according to Cloudflare, which works on the basis of a database containing prefix lists. According to Cloudflare, Verizon does not use filtering, while this could have prevented the routing problem.
In the blog post, Cloudflare writes that it uses an rpki framework, which among other things ensures that prefixes that are more specific than a size specified by Cloudflare are not accepted. Cloudflare therefore calls on Verizon to use this framework, so that such a failure does not have to occur again.
Cloudflare contacted Verizon, but says it has not yet received a response about the incident. Verizon told The Register that there was a “sporadic outage” but that the company has since rectified the disruption.