Internet Traffic Disruptions Imminent As Routing Tables Approach Limit

Brief disruptions in Internet traffic routing on Tuesday have been attributed to a temporary surge in the number of registered IPv4 network routes which pushed the total number beyond the capacity of some old but still-widely-used routers that form the backbone of the Internet.The problem might have been solved, but experts are warning that it was only a precursor to widespread disruption in the very near future.

Massive-scale data-centre Internet routers, specifically some of the older ones still in use, have a hard-coded limit to the amount of TCAM (Ternary Content-Addressable Memory) allocated to storing records of Internet routes which are broadcast by all kinds of servers from time to time. These routes are then used to send data such as Web site requests from any corner of the world to any other.

Thousands of these routers typically mark the entry and exit points, known as gateways, between Internet service providers' networks and the massive backbones that ferry data packets around the world. They are governed and synchronised by what is known as the Border Gateway Protocol (BGP).

The practical limit on the number of storable routes is 524288 (2 raised to the power of 19 in binary, or 512,000), and this was surpassed on Tuesday, leading to congestion and blocked routes. BGP was unable to determine routes, resulting in packet loss.

The number of routes has been hovering at around 500,000 for several months and is creeping up slowly. Stakeholders have been preparing to fix the router memory allocation issue but were not prepared for the sudden surge in published route records. The problem has since been attributed by BGPMon to an error in Verizon's systems, and has been fixed. Traffic has therefore returned to normal, but the 512,000 record limit will eventually be reached, much like the Y2K limit was.

Cisco published information in May about how to re-allocate TCAM on its widely-deployed Catalyst 6500 and 7600 routers to 1,000,000. The fix is relatively simple to apply, but will require each router to be taken offline briefly. Doubling the capacity should alleviate the problem for a long time, but it is not a permanent fix. According to packetfix.net, the milestone of 300,000 routes was passed in August 2009, indicating a rapid acceleration in growth.

Users can expect widespread disruption to traffic, including reduced reliability of cloud-based services and apps that need to connect to a remote server, which are all commonplace today (and in many cases, indispensible). It is not known how many routers still need to be patched or replaced, and whether their owners are doing so proactively.