AWS outage 2021: us-east-1 and the retry storm inside AWS

At 07:30 PST (15:30 UTC) on Tuesday 7 December 2021, AWS’s Northern Virginia region, us-east-1, began to fail in an odd way. Most websites and apps that ran there kept running. What stopped was anything that needed to change: launching servers, signing in to the console, getting fresh credentials. Disney+, Netflix, Slack, Ticketmaster, Robinhood and Coinbase reported problems, and so did Amazon itself. In the middle of peak season, warehouse and delivery workers couldn’t use the Flex and AtoZ apps to scan packages or see their routes. The cause was a traffic jam on a network most AWS customers never see.

How AWS’s two networks fit together

Most AWS services, and every customer application, run on what AWS calls the main AWS network. A set of foundational services runs on a separate internal network: monitoring, internal DNS, authorization services, and parts of the EC2 control plane. Because those services matter so much, AWS joins the two networks through multiple geographically isolated networking devices, with capacity scaled well beyond normal needs. The devices do routing and network address translation (NAT), so services on one side can talk to the other.

It helps to split any cloud service in two. The control plane is the machinery that makes changes: launching instances, creating load balancers, issuing credentials. The data plane is the daily business of the resources that already exist: an instance running your code, a load balancer forwarding requests. AWS designs its data planes to keep working when the control plane is impaired, and calls this static stability. A physical EC2 host already has what it needs to route its instances’ packets, so it doesn’t have to ask the control plane anything minute to minute.

Customer workloads live on the main network. Much of what manages them, and watches them, lives on the internal one, behind one set of devices.Open in topo ↗

Why retries turn a slowdown into a jam

Networks and services fail in small ways all the time: a dropped packet, a slow reply. The standard answer is to retry, and that works well when failures are rare and random. Under overload it works against you. Amazon’s own guidance describes retries as selfish: a client that retries spends more of the server’s time to raise its own chance of success. When the server is failing because it is overloaded, every retry adds load, and the extra load can keep the system down long after whatever started it has gone.

Retries also multiply across layers. If one customer request passes through five services and each tries three times, a failing database at the bottom sees 3⁵, or 243, times the load. So the usual advice is to retry at a single layer, back off exponentially between attempts, and add jitter, a random delay, so that thousands of clients don’t all retry at the same instant. In one simulation of 100 clients competing for the same resource, adding jitter to exponential backoff more than halved the number of calls. Since 2016 the AWS SDKs have also capped retries with a token bucket: a client may retry while it has tokens, and after that only at a fixed rate. All of this assumes the clients actually back off.

What happened

At 07:30 an automated activity began scaling capacity for one AWS service on the main network. It triggered behaviour nobody had seen before in a large number of clients on the internal network, and they produced a surge of connection activity that overwhelmed the devices between the two networks. Traffic across them slowed. Rising latency and errors led to more connection attempts and retries, which kept the devices congested. AWS hasn’t said what the clients were doing, only that they have well-tested back-off and that a latent issue stopped it working. The code path had been in production for many years.

Each attempt died in the congested devices, and each client tried again the moment its timeout fired, with no back-off. Many clients doing this kept the devices congested. Timings are illustrative.Open in ack ↗

The congestion immediately cut off real-time monitoring data for AWS’s internal operations teams, so the people who most needed to see the network were working from logs. Route 53’s API, used to change DNS records, was impaired from 07:30. From 07:33 EC2’s APIs for launching and describing instances were failing, and so was opening a support case. Signing in to the console failed. STS, which hands out temporary credentials, was slow to issue them for identity providers using OpenID Connect, so logins to services such as Redshift failed. Running instances, existing load balancers and answers to DNS queries carried on as normal.

The Service Health Dashboard, AWS’s public status page, should have failed over to a standby region, and the congestion stopped that too. AWS was updating it by 08:22.

In the logs, operators found elevated internal DNS errors. Internal DNS underpins every service, and its traffic seemed to be adding to the congestion, so the teams concentrated on moving it off the congested paths. That was done at 09:28, and DNS resolution errors fully recovered. It helped: several services improved as load on the devices fell. But it didn’t clear the congestion, and monitoring was still dark.

What followed was slow, deliberate load shedding: finding the heaviest sources of traffic and isolating them onto dedicated network devices, switching off some services that moved a lot of data, and bringing more networking capacity online. At 12:35 operators even disabled event delivery for EventBridge to take load off the devices. EC2 API errors began to ease at 13:15, congestion improved significantly by 13:34, and every network device had fully recovered by 14:22, when console access came back too. New EC2 launches recovered at 14:40.

Why it took seven hours

Every timeout became another connection attempt, and the monitoring that would have shown it sat behind the same devices.Open in topo ↗

The operators were blind. Monitoring ran on the internal network, so it failed at the very start of the event it should have diagnosed. Finding the sources of congestion without real-time data meant working from logs, one hypothesis at a time.

The repair tools sat on the broken network. AWS’s internal deployment systems also run on the internal network, so pushing the changes that would relieve the congestion was itself slowed by the congestion. The status page’s failover and the support system depended on the same network, which is why customers heard so little in the first hour.

Caution cost time. Most of the main network and most customer applications were still working, and a hasty change to relieve the devices could have broken those too.

And some recovery needed the control plane. Many API Gateway servers, cut off from the internal network early on, ended up in a state where they had to be replaced. An automated process normally recycles them, but it needs EC2 launches, so it couldn’t start until the EC2 APIs recovered. API Gateway began recovering at 13:35 and was largely back by 16:37, and Amazon Connect, which used it for calls and chats, returned to normal at 16:41. Containers on Fargate, ECS and EKS kept running, but any that failed couldn’t be replaced; afterwards Fargate had a backlog to start and ran short of capacity, and its API errors only returned to normal around 17:00. STS fully recovered at 16:28, and EventBridge worked through its backlog of events until 18:40.

S3 and DynamoDB themselves were fine, but reaching them through VPC endpoints, the private connections from inside a customer’s own network, was impaired throughout.

How they fixed it

AWS disabled the scaling activities that had triggered the event straight away, and said it wouldn’t resume them until every fix was deployed. It started work on the bug that had stopped the clients backing off, expecting to roll the fix out within two weeks, and added network configuration to protect the devices even if a similar surge happened again.

For communication, AWS promised a new Service Health Dashboard early the following year, to make it easier to see which services were affected, and a support system that actively runs across multiple regions, so that opening a case wouldn’t depend on one region’s internal network. The API Gateway team began work on keeping its servers healthy even when the internal network is unavailable, and on making the recycling process faster.

What I'd take from it

Retries need a budget, at one layer. Three tries at each of five layers is 243 times the load on the bottom one. Retry at a single layer, back off with jitter, and cap retries with a token bucket, so a slowdown can't become congestion that feeds itself.
Test the back-off under the failure it exists for. AWS's clients had well-tested back-off, and a latent bug stopped it working in exactly this event. Run congestion drills that check clients really slow down, not just that the code path exists.
Monitoring must not share fate with what it watches. Operators lost real-time monitoring the moment the incident began and worked from logs for hours. Keep a thin, independent path for the few signals you need most in an incident.
Keep the control plane out of your recovery path. Running instances were fine; anything that had to launch one to recover was stuck. Pre-provision what you need to recover, so recovery doesn't depend on the part that is broken.