The day Facebook withdrew itself from the internet
A command meant to measure backbone capacity disconnected every Facebook data centre. The DNS servers, built to withdraw their BGP routes when they lose the data centres, all did so at once, and Facebook, Instagram and WhatsApp vanished from the internet for nearly six hours, along with the tools engineers needed to fix it.
Times are approximate, in UTC, from the sources at the end. The diagram is simplified.
At 15:39 UTC on Monday 4 October 2021, Facebook, Instagram, WhatsApp and Oculus stopped working for everyone, everywhere. Apps spun, browsers said facebook.com didn’t exist, and for nearly six hours a company claiming 3.5 billion users was missing from the internet. Its servers were running the whole time. The rest of the internet had been told, by Facebook’s own machines, that there was no way to reach them.
How the internet finds facebook.com
Two systems have to work before your phone can load a feed.
The first is DNS, the internet’s address book. Your phone asks a resolver, usually your ISP’s or a public one such as 1.1.1.1 or 8.8.8.8, for the address of facebook.com. If the resolver has no recent answer cached, it asks the domain’s authoritative name servers, which Facebook runs and which hold the real answers. One of them, a.ns.facebook.com, lived at 129.134.30.12. The resolver caches the answer for a set time, its TTL, and then asks again.
The second is BGP, the Border Gateway Protocol. The internet is a network of networks, each called an autonomous system, and BGP is how they tell each other which blocks of addresses (prefixes) they can deliver. Facebook’s network is AS32934. On the morning of the outage, one public route collector saw it announcing 133 IPv4 prefixes and 216 IPv6 prefixes. If nobody announces a prefix, routers have no path to it, and packets for those addresses go nowhere.
Facebook’s own network has two kinds of site. Data centres are huge buildings where the apps run and the data lives. Smaller edge facilities sit close to users and connect Facebook to everyone else. A backbone of tens of thousands of miles of fibre joins them, so a request that reaches an edge facility travels over the backbone to a data centre and back.
The authoritative DNS servers run in the edge facilities, and each facility announces the DNS prefixes over BGP. The same addresses are announced from many places, so a resolver’s question lands at whichever site BGP picks, usually a nearby one. This is called anycast. Each DNS server also checks that it can reach the data centres, and if it can’t, it stops announcing. That is a sensible rule: a site cut off from the data centres can’t serve anyone, and withdrawing its route lets BGP send users to a healthy site instead.
What happened
During routine maintenance on the backbone, an engineer issued a command meant to assess how much global backbone capacity was available. Facebook hasn’t published the command itself. It took down every connection in the backbone instead, disconnecting Facebook’s data centres from each other and from the internet. Facebook’s systems are designed to audit commands like this and stop the dangerous ones, but a bug in the audit tool let this one through.
At 15:39 Kentik, which measures traffic across many networks, saw Facebook’s traffic collapse and a burst of BGP updates from AS32934. In every edge facility, the DNS servers had run their health check, found no data centre, and withdrawn their routes. The rule did exactly what it was written to do, in every site at the same moment. From one collector’s view, Facebook withdrew 8 of its 133 IPv4 prefixes and 14 of its 216 IPv6 prefixes around 15:40. That was enough, because those were the prefixes holding the DNS servers. Most of Facebook’s other addresses stayed announced, and were useless without names that pointed at them.
For a few minutes, people whose resolvers still had facebook.com cached could reach an edge server. It had nothing behind it, and answered HTTP 503 with a no_server_available status. Then the cached answers expired. By about 15:50 facebook.com had stopped resolving on 1.1.1.1, and resolvers everywhere were returning SERVFAIL, the DNS error that means the resolver couldn’t get an answer.
At 15:51 Cloudflare opened an internal incident, worried that its own resolver was broken. At 15:58 it saw the real reason: Facebook had stopped announcing the routes to its DNS prefixes. Asked for 129.134.30.0/23, a router at the public Route Views project replied that the network was not in its table.
Then came the retries. Apps don’t take an error for an answer; they retry, some of them aggressively, while people reload pages and relaunch apps. Public resolvers were soon handling 30 times their usual number of queries. Kentik saw Facebook’s TCP traffic disappear entirely, leaving a thin stream of UDP: DNS retries from phones trying, and failing, to reconnect. People went elsewhere too, and DNS queries for Twitter, Signal and other platforms rose.
Why it took six hours
Inside Facebook, the engineers had two problems at once. They couldn’t reach the data centres through their normal means, because the network those means ran over was the one that was down. And the loss of DNS broke many of the internal tools they would normally use to investigate and fix an outage like this. Both the primary and the out-of-band network access were down.
So Facebook sent engineers to the data centres to debug the problem and restart systems. That took time, because the facilities are designed to be hard to get into, and once inside, the hardware and routers are designed to be hard to modify even with physical access. Activating the secure access procedures that let people work on site took extra time on top. The hardening that keeps attackers out slowed down the people fixing a mistake Facebook had made itself. Facebook’s view afterwards was that the trade is still worth it: much better security every day, against a slower recovery from a rare event.
Facebook’s postmortem gives no times for the hours on site. It is clear about the last risk, though, which came after the fix. Individual data centres had seen their power draw drop by tens of megawatts. Reversing a dip like that all at once could put everything from electrical systems to caches at risk, just as billions of apps retried together.
How they got back
Around 21:00, Facebook’s routes started coming back, and BGP activity from its network peaked at 21:17. facebook.com resolved on 1.1.1.1 again at 21:20, and by 21:28 Facebook looked reconnected to the internet from outside. As WhatsApp came back, Kentik measured a surge in latency: upwards of a billion users were likely trying to reconnect at the same time.
Facebook brought services back carefully, and here it had practised. Its “storm” drills simulate a major failure by taking a service, a data centre or a whole region offline, stress-testing the infrastructure and software involved. That experience let the team manage the returning load, and services came back without another system-wide failure. Facebook had never run a storm that took out the global backbone, and said it would look for ways to simulate one.
Facebook’s public account ends there, without a list of fixes. It said the job from here was to strengthen testing, drills and overall resilience, which leaves the specific changes to the audit tool and the DNS health check unpublished.
Other systems with the same rule
“Take yourself out when you’re unhealthy” is one of the most common rules in infrastructure. Load balancers drop backends that fail health checks, service meshes eject hosts, DNS services stop returning records whose checks fail. The mature versions all have a floor for the day every check fails together. Envoy has a panic threshold: when fewer than half the hosts in a cluster look healthy (the default), it ignores health and spreads traffic across all of them, on the theory that the check is more likely wrong than every host. Amazon Route 53 does something similar, answering as if every record were healthy when all of them fail.
A floor wouldn’t have brought Facebook back, because its data centres really were unreachable. But the names would have kept resolving, users would have seen Facebook’s own error pages, and the outage would have looked like what it was: a broken backbone, not a company that had vanished.
What I'd take from it
- Give health checks a floor. Withdrawing a route when you are unhealthy is right for one site and fatal for all of them. If most sites fail the same check at the same moment, suspect the check or a shared dependency, and keep announcing from the last few.
- Out-of-band has to be out of band. Separate network, separate name resolution, separate credentials, and used often enough that you know it works. Facebook had out-of-band access, and it was down too.
- Test the guardrail with the commands it must stop. The audit tool was the only thing between a capacity check and the whole backbone. Feed it known-dangerous commands in CI and fail the build if one gets through.
- Rehearse the cold start. Facebook's storm drills are why the restart went smoothly. Include the case where everything is off at once, and plan how to bring load back in steps.