A mistyped input, and an S3 index nobody had restarted in years
While debugging a slow billing system, an S3 engineer ran an established playbook command with one input wrong and removed far more servers than intended, including capacity behind S3's index and placement subsystems. Both needed a full restart, which had not been done in S3's larger regions for years, so S3 in us-east-1 stopped serving requests for nearly three hours and new uploads for over four. EC2 launches, Lambda and AWS's own status dashboard went down with it.
Times are approximate, in PST (UTC−8), from the sources at the end. The diagram is simplified.
At 09:37 PST on Tuesday 28 February 2017, Amazon S3 in Northern Virginia (us-east-1) stopped answering. Requests to read, list, write and delete objects failed for nearly three hours, and uploads for more than four. Everything in the region that keeps its data in S3 went down with it: new EC2 instances would not launch, Lambda failed, and AWS could not update the dashboard it uses to tell customers what is broken, because that depended on S3 too.
How S3 finds an object
S3 stores objects: a name (the key, inside a bucket) and some bytes. Its API is small. PUT writes an object, GET reads one, LIST returns the keys under a prefix and DELETE removes one.
Storing bytes is the easier half of an object store. The harder half is remembering where they are. With an enormous number of objects spread over a huge fleet of disks, the service needs a map from every bucket and key to the machines holding its bytes, plus the object’s size and other metadata. That map is kept separately from the data, and in S3 it belongs to the index subsystem, which AWS describes as managing the metadata and location of every object in the region.
Look at the API with that in mind and the index is in every call. A GET has to ask it where the bytes are. A LIST is a query on the index alone. A DELETE removes an entry. A PUT adds one, but first the bytes need a home, and choosing it is the job of the placement subsystem, which allocates new storage. Placement needs a working index to do that correctly, so writes depend on both.
Both subsystems are fleets of servers, and the fleets change all the time. Disks fail, machines are retired, capacity moves around. AWS built S3’s subsystems to survive losing a significant share of their servers with little or no effect on customers, and removing capacity was, in its own words, one of its core operational processes.
That tolerance has a limit. Take away enough servers and a subsystem cannot keep serving; it has to be brought back from scratch. For a metadata service that takes time: the servers have to come up, and the service has to check that the metadata is intact before trusting it. Serving wrong locations would be far worse than serving none.
What happened
On the morning of 28 February, the S3 team was debugging its billing system, which was progressing more slowly than expected. To help, an authorised member of the team used an established playbook to remove a small number of servers from one of the subsystems that S3 billing uses.
At 09:37 they ran the command. One of its inputs was entered incorrectly, and a much larger set of servers was removed than intended. AWS never said which input or by how much. Those servers also supported the index and the placement subsystems, and losing that much capacity at once meant both needed a full restart. Until they were back, S3 in us-east-1 could not serve requests.
The effects spread through the region straight away. The S3 console failed. New EC2 instances could not launch. EBS volumes that needed data from an S3 snapshot were affected, and so was AWS Lambda.
The AWS Service Health Dashboard, the page customers check during an incident, has an administration console that engineers use to change each service’s status. That console depended on S3. From the start of the event until 11:37, AWS could not update the status of individual services on its own dashboard. It posted updates on its Twitter account, @AWSCloud, and in a banner at the top of the page instead.
The index had to come back first, because placement needs it. The restart took longer than anyone expected. AWS had not completely restarted the index or placement subsystems in its larger regions for many years, and S3 had grown enormously in that time. Bringing the index up and running the safety checks that validate its metadata simply took hours at that size.
By 12:26 the index had brought enough capacity back to serve GET, LIST and DELETE again. At 13:18 it was fully recovered and those calls were working normally. Uploads were still failing: PUT needs placement, and placement could only begin its own recovery once the index was working. It finished at 13:54, and S3 was operating normally, four hours and 17 minutes after the command. Other AWS services that had queued work during the disruption then needed extra time to clear their backlogs.
The engineer was not improvising on a production box. They were authorised, following an established playbook, and doing an operation S3 had relied on since its launch. The command did exactly what it was told. AWS’s own summary puts the fault in the tool, which let too much capacity be removed too quickly.
Why it took four hours
Several things lined up, and none of them was the typo itself.
Nothing knew the floor. The tool had no idea what each subsystem needed to keep running. A mistyped value went straight through, and, as AWS put it, the tool let too much capacity go too quickly.
The rare path had never been rehearsed at full size. Removing a few servers was practised constantly and worked. A full restart of the index in a large region was something S3 had not done in many years, during which the amount of metadata had grown massively. The integrity checks scale with that metadata, so the recovery took far longer than it would have when anyone last did it. A system can be excellent at the operations it runs daily and have no measured answer for the one it hasn’t run since it was much smaller.
Recovery was serial. Placement could not start until the index was working, and PUT needed both. Reads came back at 12:26, but writes had to wait another 88 minutes.
Everything leaned on one region’s S3. S3 is a building block for other AWS services, not only for customers. When it stopped, launching a virtual machine, restoring a disk from a snapshot and running a function stopped too. A storage outage became a compute outage.
The status page shared its fate. The dashboard’s admin console lived on the very service that was down. For the first two hours, the page customers were refreshing could not be updated service by service, and the most reliable status source was a Twitter feed. Nobody sets out to make a status page depend on the thing it reports on. It is an easy dependency to acquire when S3 is the default home for everything.
How they fixed it
The recovery itself was mostly waiting: restart the index and let its checks run, then restart placement, then let dependent services drain their backlogs. There was no shortcut that would not have risked the integrity of the metadata.
The changes AWS announced afterwards were aimed at the tool and the recovery, not at the person:
- The capacity removal tool was changed to remove capacity more slowly, and it now refuses to remove capacity when that would take any subsystem below its minimum required level. An incorrect input can no longer do what this one did.
- AWS started auditing its other operational tools for the same kind of safety check.
- To make recovery faster, S3 leaned harder on cells: splitting a service into small partitions that fail and recover independently. Cells shrink the blast radius of a mistake, and they make recovery of even the largest subsystem small enough to test properly. S3 had already been refactoring parts of the service into cells, and further partitioning of the index had been planned for later in 2017. AWS moved that work up to start immediately.
- The Service Health Dashboard’s admin console was changed to run across multiple AWS regions.
The postmortem has no retraining plan and no call to be more careful. It treats the operator as part of a system that should have stopped a bad input, and the fixes went into the tooling.
What I'd take from it
- Give every capacity tool a floor. AWS's first fix was a check that refuses to take any subsystem below its minimum capacity, plus a slower removal rate. Any command that drains, deletes or scales down should know that number and stop there, whatever the operator typed.
- Rehearse the cold start, not only the routine change. Removing a few servers happened all the time. Restarting the whole index at its current size had not happened in years, and its integrity checks took far longer than expected. If a system can ever need a full restart, time one regularly.
- Status tooling must not depend on what it reports on. The dashboard's admin console depended on S3. I'd host status pages and incident tools in another region or with another provider, and update them from there during every drill.
- Smaller cells make restarts smaller. Splitting a subsystem into independent partitions means one mistake, or one restart, covers a slice of the keyspace instead of a whole region. It also makes a full recovery small enough to test.