One stale check, and DynamoDB's endpoint resolved to nothing
A latent race between two copies of DynamoDB's DNS automation put an old plan on the us-east-1 endpoint and then deleted it, leaving dynamodb.us-east-1.amazonaws.com with no addresses. DNS was repaired in under three hours, but EC2's server-lease system collapsed under its own recovery work, and new instances, load balancers and the services built on them took until the afternoon.
Times are approximate, in PDT (UTC−7), from the sources at the end. The diagram is simplified.
At 23:48 Pacific time on Sunday 19 October 2025 (06:48 UTC on the 20th), every system that reached DynamoDB in us-east-1 through its public endpoint started failing to connect. Customer applications failed, and so did the AWS services that keep their own state in DynamoDB. The database was running. Its name, dynamodb.us-east-1.amazonaws.com, pointed nowhere. DNS was repaired within three hours, but the event did not end until 14:20 the next afternoon, because of what those three hours did to EC2.
How DynamoDB keeps its DNS current
A client that wants DynamoDB asks DNS for the addresses behind the endpoint name, gets back a list of IP addresses, and connects to one of them. It may cache that answer for as long as the record’s time to live (TTL) allows. Behind DynamoDB’s regional name sit many load balancers, and DNS is how traffic is spread across them: put a load balancer’s address in the record and it gets traffic, take it out and it stops.
Those records change constantly. DynamoDB maintains hundreds of thousands of DNS records to run a very large, mixed fleet of load balancers in each region, and capacity is added, hardware fails and traffic shifts all day. Nobody edits them by hand. Automation keeps the regional endpoint current, along with variants such as the FIPS endpoint, the IPv6 endpoint and per-account endpoints.
The automation is split in two, for availability. The DNS Planner watches the health and capacity of the load balancers and periodically produces a new plan: for each endpoint, a set of load balancers and their weights. There is one plan for the whole region, which keeps capacity management simple when endpoints share capacity, as the regional and the new IPv6 endpoint do. The DNS Enactor turns plans into records in Route 53, AWS’s DNS service. It is deliberately built with minimal dependencies so it can help recover from anything, and it runs as three fully independent copies, one in each of three Availability Zones. Each copy looks for new plans and applies them endpoint by endpoint, replacing the old plan with the new one in a Route 53 transaction, so an endpoint always holds one complete plan even when two Enactors update it at the same time.
Several writers for the same data is how a control plane survives losing a zone. It is also how you get races. A transaction makes each write all or nothing. It does not make writes land in the order they were meant to. For ordering, each Enactor relied on one check, made before it started on a plan: that the plan was newer than the one already applied.
What happened
Just before 23:48, one Enactor (call it A; AWS doesn’t name them) ran into unusually high delays and had to retry its updates on several endpoints. While A crawled through its list, other things kept moving:
- The Planner kept running and produced many newer generations of plans.
- A second Enactor, B, picked up one of the newest plans and worked through every endpoint quickly.
- B finished and started its clean-up, which deletes plans significantly older than the one it has just applied.
- At that moment A, still carrying its much older plan, reached the regional endpoint and applied it, overwriting B’s newer plan. A’s check that its plan was newer had been made at the start, and by now it was stale.
- B’s clean-up then deleted A’s plan, because it was many generations old. It was also the plan now live on
dynamodb.us-east-1.amazonaws.com, and deleting it removed every IP address from the record.
Each step was reasonable on its own. Together they left the system in a state it couldn’t leave: with the active plan deleted, no Enactor could apply any later plan. It needed people.
From 23:48, every lookup of the endpoint came back empty. Customers with DynamoDB global tables could still use their replicas in other regions, but replication to and from us-east-1 fell far behind. Inside AWS the list of things that depend on DynamoDB turned out to be long: console sign-ins with IAM users saw increased failures from 23:51 until 01:25, STS returned errors from 23:51 until 01:19, and Lambda couldn’t create or update functions while its SQS and Kinesis event sources fell behind. The AWS Support Center failed over to another region as designed, but a subsystem for account metadata started returning invalid responses that locked legitimate users out of support cases until 02:40. And a Redshift defect reached beyond the region: until 01:20, customers in every region couldn’t run queries with IAM user credentials, because Redshift resolved user groups through an IAM API in us-east-1.
By 00:38 engineers had identified DynamoDB’s DNS state as the source. At 01:15, temporary mitigations let some internal services connect to DynamoDB again and repaired key internal tooling, which the rest of the recovery needed. By 02:25 all the DNS information was restored, and as cached answers expired between 02:25 and 02:40, clients resolved the endpoint and connected. Global tables were caught up by 02:32.
Why it took until the afternoon
DynamoDB was back. EC2 was not, and the reason is a lease.
EC2 runs its physical servers, which it calls droplets, with a system called DropletWorkflow Manager (DWFM). Each DWFM host manages a set of droplets in a zone and holds a lease on each one, which is how it tracks the droplet’s state and makes sure actions such as a reboot from inside an instance are reflected correctly across EC2. To keep a lease, the DWFM host completes a state check with the droplet every few minutes, and that check depends on DynamoDB.
From 23:48 the checks failed. Running instances noticed nothing, since nothing touched them. But leases started timing out one after another, and a droplet without an active lease is not a candidate for new instances. When DynamoDB returned at 02:25, leases across the fleet had been expiring for more than two and a half hours, and EC2 was answering launch requests with insufficient capacity errors.
Then the recovery collapsed under its own weight. DWFM set about re-establishing leases across the whole fleet. There were so many droplets that each attempt took long enough to time out before it completed, the failed work was queued to try again, and the queue only grew. AWS calls this congestive collapse: the system was busy all the time and made no progress. There was no established procedure for it, so engineers moved carefully to avoid making it worse. After several mitigation attempts, at 04:14 they throttled incoming work and began restarting DWFM hosts selectively, which cleared the queues. By 05:28 every droplet in the region had a lease and launches began to succeed, though the throttling now turned many requests away with request limit exceeded errors.
Networking was next. A new instance needs its network configuration, the state that lets it talk to its VPC, other network appliances and the internet, pushed to it by a system called Network Manager. Hours of launches and terminations had been held back, and from 05:28 the backlog arrived together. At 06:21 propagation times began to climb. Instances launched, but without working networks.
Those instances then broke load balancing. The Network Load Balancer’s health checker brought new instances into service while their network state had not yet propagated, so checks failed on targets that were fine, then passed, then failed again. Nodes and targets were removed from DNS and returned on the next good check. Some load balancers had rising connection errors from 05:30, and monitoring caught the pattern at 06:52. The alternating results also loaded the health check subsystem itself, which slowed down and triggered automatic Availability Zone failover, taking capacity out of service for multi-AZ load balancers. At 09:36 engineers disabled automatic health check failover for NLB, which let all the healthy nodes and targets back into service and ended the connection errors.
Every service built on those layers had a second wave. From 07:04, NLB health check failures caused instance terminations that left some of Lambda’s internal systems under-scaled, and with launches still impaired AWS throttled event source mappings and asynchronous invocations to protect synchronous calls. STS errors returned between 08:31 and 09:59. Amazon Connect callers heard busy tones and dead air.
Network propagation was back to normal at 10:36. Engineers began relaxing the EC2 throttles at 11:23, all EC2 APIs and launches were normal at 13:50, Lambda had processed its backlogs by 14:15, and ECS, EKS and Fargate were recovered by 14:20, which AWS gives as the end of the event. Some Redshift clusters, stuck waiting for replacement hosts while launches were impaired, took until 04:05 on 21 October.
Two things made the tail this long. The first is ordering: each layer could only recover after the one beneath it, so the times added up rather than overlapping. The second is that recovery is load. Every system had deferred work (leases to renew, network state to push, messages to process) and all of it arrived at once, at a scale none of them met on a normal day.
How they fixed it
The DNS record was repaired by hand; everything after that was throttling and restarts, as above. In the summary AWS listed what it is changing:
- The DynamoDB DNS Planner and DNS Enactor automation is disabled worldwide. Before it is turned back on, the race will be fixed and extra protections added to stop incorrect DNS plans being applied.
- NLB gets a velocity control that limits how much capacity a single load balancer can remove when health check failures cause Availability Zone failover.
- EC2 gets a new test suite, on top of its existing scale tests, that exercises the DWFM recovery workflow to catch regressions.
- EC2’s data propagation systems get better throttling, rate-limiting incoming work by the size of the queue already waiting.
The pattern is older than DynamoDB: check, then act, with time in between. A check made before a slow operation describes the world as it was, and a second writer, here a clean-up job, can change it in the gap. The fix is to make the write itself conditional on what it expects to replace, so the stale writer’s transaction fails instead of succeeding late.
What I'd take from it
- Make the write check the version, not the writer. The Enactor checked once, before a long run of retries, that its plan was newer. Put the check inside the write itself, comparing a generation number in the same transaction, so a stale writer fails instead of winning.
- Clean-up is a writer too. Deleting plans by age is safe only if the live plan can never be old. Look up what is currently applied in the same step as the delete, and never delete that.
- Refuse the obviously wrong state. An empty record for a regional endpoint is never a valid plan. The code that applies changes should reject it outright, whatever the plan says.
- Rehearse the recovery where everything comes back at once. DWFM coped with leases expiring one by one, not with renewing all of them together, and there was no runbook. Test that moment, and throttle incoming work by the size of the queue already waiting.
- Put a brake on automatic failover. NLB's health checks took missing network state for dead targets and removed healthy capacity. Cap how much capacity a health signal can take out of service in one go.