Outages
Famous outages, replayed one step at a time.
Each one is rebuilt from the public postmortem as a live diagram. Press play and watch it fail: components go red, data stops moving, and the timeline shows how long it hurt. Below the replay is the story of how it happened and what I would take from it. The replays are drawn with topo, and the timing figures come from ack, my sequence diagram tool.
Cloudflare
Duplicate rows in ClickHouse, and a proxy that panicked on them
A ClickHouse permissions change doubled the Bot Management feature file past a hard limit in the core proxy.
~5.5 hours
AWS
One stale check, and DynamoDB's endpoint resolved to nothing
A race between two copies of DynamoDB's DNS automation left the us-east-1 endpoint with an empty record.
~14.5 hours
CrowdStrike
A content update that crashed 8.5 million Windows machines
A content update checked a 21st input that the sensor never supplied, and the kernel read past the end of the array.
78 min, then days
Kubernetes retired a label, and Reddit's route reflectors vanished
A Kubernetes 1.24 upgrade removed the node label that selected Calico's BGP route reflectors, and pod networking stopped.
314 minutes
Datadog
A systemd patch that took every Datadog region offline at once
An automatic systemd security update restarted systemd-networkd, which deleted the routing rules Cilium had installed.
~27 hours
Atlassian
The wrong IDs, a permanent delete, and two weeks to put it back
A clean-up script was given site IDs instead of app IDs and permanently deleted 883 customer sites.
up to 14 days
AWS
us-east-1 and the retry storm inside AWS
An automated scaling job set off a connection surge that congested the devices between two internal networks.
~7 hours
Roblox
73 hours, two bugs inside Consul, and monitoring that ran on it
A new Consul streaming feature and a BoltDB freelist problem slowed the one Consul cluster every Roblox service depended on.
73 hours
The day Facebook withdrew itself from the internet
A maintenance command took down the backbone, and DNS servers pulled their own BGP routes.
~6 hours
Cloudflare
One regular expression, every CPU on the edge
A WAF rule with a backtracking regex pinned CPU at 100% on every edge server.
27 minutes
GitHub
A 43-second network cut, and 24 hours to make the data agree
A 43-second network partition let automated failover move MySQL primaries across the country, and both coasts took writes.
24 hours 11 min
AWS
A mistyped input, and an S3 index nobody had restarted in years
A mistyped input to a playbook command removed servers that S3's index and placement subsystems needed.
~4 hours
GitLab
rm -rf on the wrong database, and five backups that weren't
An engineer wiped the primary database instead of the replica, and the backups turned out not to work.
~18 hours
Knight Capital
Seven servers got the new code, and the eighth ran Power Peg
A deploy missed one of eight servers, and a reused flag switched on order-routing code retired nine years earlier.
45 minutes
Linux
One leap second, and Linux timers started firing a second early
The kernel set the clock back a second without telling its timer code, so short waits returned at once and processes spun.
until reset
Diagrams are simplified from the postmortems linked on each page. Built with topo.