rm -rf on the wrong database, and five backups that weren't
While rebuilding a replica that heavy load had knocked out of sync, an engineer deleted the data directory on GitLab.com's primary database instead. The team then found that none of their backups worked. A snapshot taken six hours earlier for staging saved GitLab.com, after about 18 hours of downtime and six hours of lost data.
Times are approximate, in UTC, from the sources at the end. The diagram is simplified.
At 23:27 UTC on Tuesday 31 January 2017, a GitLab engineer stopped a command a second or two after starting it. It had been deleting the data directory of GitLab.com’s primary PostgreSQL database, and when it stopped, about 4.5 GB was left of roughly 300 GB. GitLab.com went down for about 18 hours. When the team went looking for the backups, they found that none of them worked.
How GitLab.com kept its data
GitLab.com ran on one PostgreSQL primary, db1.cluster.gitlab.com, and one secondary, db2.cluster.gitlab.com, in hot-standby mode. The secondary existed only for failover, so the primary carried all the load, and it had been a single point of failure in earlier incidents.
The link between the two was streaming replication. PostgreSQL records every change in its write-ahead log (WAL) before applying it. A standby connects to the primary, streams that log and replays it, so it stays moments behind. The primary only keeps a limited amount of WAL. If a standby falls so far behind that the segments it needs have already been removed, it can never catch up by streaming, and the fix is to start over: empty the standby’s data directory and copy a fresh base backup from the primary with pg_basebackup. GitLab wasn’t archiving WAL anywhere else, so there was nowhere to fetch missing segments from, and no way to restore the database to an arbitrary point in time.
Around that sat several ways to get data back, at least on paper: a pg_dump of the database every 24 hours, uploaded to Amazon S3; an LVM snapshot of the database disk every 24 hours, used to refresh the staging environment; Azure disk snapshots every 24 hours for some servers; and the replica itself, which was meant for failover, not for recovering from mistakes.
What happened
On the evening of 31 January an engineer was setting up extra PostgreSQL servers in staging, to try pgpool-II for spreading queries across several databases. At about 17:20, before starting, they took a fresh LVM snapshot of production and loaded it into staging, because the automatic one from 01:00 was too old for accurate load tests.
At 18:00 GitLab saw spammers hammering the database by creating snippets. Part of the load, the postmortem found later, came from a background job deleting a GitLab employee and all their data: a troll had reported them for abuse, and the account had been scheduled for removal by mistake. Another user was using a repository as a kind of CDN, with 47,000 IP addresses signing in to the same account. At 21:00 the load caused a lockup on writes and some downtime. The team blocked spammers by IP address and removed the abusive users.
At 22:00 they were paged: replication was lagging badly. db2 was about 4 GB behind db1, and around 23:00 replication failed outright, because the WAL segments db2 needed were already gone from db1. db2 would have to be rebuilt from scratch.
An engineer wiped db2’s data directory and ran pg_basebackup. It hung, printing nothing, even with the verbose flag. After a few tries it complained that the primary didn’t allow enough replication connections, so the team raised max_wal_senders from 3 to 32. PostgreSQL then refused to restart, reporting too many semaphores, because max_connections was set to 8,000: far too high, although it had run that way for almost a year. They lowered it to 2,000 and PostgreSQL started. pg_basebackup still hung, and strace showed it waiting in a poll call and nothing more. The engineer had said earlier that he was about to sign off for the night, at around 23:00 his time, and was still there because of this.
He wondered whether pg_basebackup was refusing to start because the earlier attempts had left files in db2’s data directory, and decided to remove the directory. He ran the command in the terminal for db1. A second or two later he noticed, and at 23:27 he stopped it. The Register, working from GitLab’s live notes, reported that it was an rm -rf. About 4.5 GB of roughly 300 GB was left.
It later turned out that pg_basebackup had been fine all along. It waits silently for the primary to start sending data, which another engineer said can take up to 10 minutes. That wasn’t written down in GitLab’s runbooks, or clearly in the PostgreSQL documentation.
Five backups, none working
GitLab took the site down for emergency maintenance and went looking for backups, asking for help on Slack.
The daily pg_dump had never worked. The S3 bucket was empty, and the dump files that existed were only a few bytes. The job ran on an application server, not the database server. GitLab’s Omnibus package chooses its PostgreSQL binaries by reading the version file in the local data directory, and an application server has no data directory, so it fell back to 9.2. pg_dump 9.2 refuses to dump a 9.6 server and exits with an error, every night. The cron job did send failure emails, but DMARC wasn’t enabled for them, so the receiving mail server rejected them, and nobody ever saw one.
Azure disk snapshots were enabled for the NFS servers holding Git data, but not for the database servers, on the assumption that the other backups were enough. Restoring one from another storage account could take hours or days anyway; once it had taken more than a week.
The replica was already empty. Emptying it was what the engineer had been doing.
That left the LVM snapshots: the daily one, almost 24 hours old, and the one taken by hand at 17:20. They existed to refresh staging, not for disaster recovery, and the refresh process deletes webhooks from the staging copy so they can’t fire by accident. GitLab’s own tally, written that night, was five backup or replication techniques, none of them working reliably or even set up.
The five whys in the postmortem end on one line about why nobody had noticed: there was no owner for backups, so nobody was responsible for testing them.
How they fixed it
The 17:20 snapshot was the only way to keep the loss to six hours instead of nearly 24, so they used it. The plan: copy the staging data directory, which had no webhooks, to production; bring up a separate copy of the snapshot to recover the webhooks from; move every database sequence on by 100,000 so no ID from the lost hours could be handed out twice; then re-enable GitLab.com gradually.
At 00:36 on 1 February they backed up the staging database’s data, and at 00:55 they mounted db1.staging on db1.cluster. Then the copy ran for about 18 hours. Staging lived on Azure classic without Premium Storage, to save money, on network disks throttled to around 60 Mbps, with no way to move to faster storage.
GitLab worked in the open throughout. The live notes were in a public Google Doc, the recovery was streamed on YouTube with a peak of around 5,000 viewers, the second most-watched live stream on the site for several hours, and updates went out on Twitter. The engineer had added his own name to the notes and didn’t mind it being public; GitLab said it would redact names in future anyway.
At 17:00 on 1 February the database was restored without webhooks. The webhooks came from the separate copy, the final steps were done around 18:00, and GitLab.com was back online at 18:14. Database changes made between 17:20 and the deletion were gone: an estimated 5,000 projects, 5,000 comments and 700 new user accounts. Git repositories and wikis, stored separately, were untouched, and so were self-managed GitLab installations.
Afterwards GitLab opened issues for prompts that make hosts and environments look different, backup monitoring in Prometheus, a sane max_connections, hourly LVM snapshots, Azure snapshots for the databases, point-in-time recovery with continuous WAL archiving, automated tests that restore database backups, better replication runbooks, more secondaries behind pgpool-II, a softer delete for users flagged as spam, and an owner for data durability. Its stated aim was not to stop engineers running dangerous commands. Aliasing rm protects against one mistake and not against disk corruption. The aim was to make mistakes cheap to recover from.
What I'd take from it
- A backup you haven't restored is a hope. Restore to a scratch machine on a schedule, check the result, and alert when it fails or takes longer than it should. GitLab's dump files were a few bytes each, and nobody knew.
- Alert on silence, not only on failure. The failure emails were rejected by the mail server, so every night looked fine. Watch the age and size of the last good backup from a system that doesn't rely on the backup job speaking up.
- A replica is not a backup. Replication copies mistakes as faithfully as data, and here the replica had already been emptied. Keep continuous archiving (for PostgreSQL, WAL archiving with point-in-time recovery) somewhere one command can't reach.
- Make the dangerous host look dangerous. A different prompt colour for production, the host name written into every destructive runbook step, and a second person reading along during surgery on a primary. They cost nothing.
- Know your restore time before you need it. The only good copy sat on disks throttled to around 60 Mbps, which turned a six-hour-old snapshot into 18 hours of copying. Where the copy lives is part of the backup.