A 43-second network cut, and 24 hours to make the data agree

At 22:52 UTC on Sunday 21 October 2018, engineers replacing failing optical equipment disconnected GitHub’s primary US East Coast data center from its network hub. The link was back 43 seconds later. What followed was 24 hours and 11 minutes in which GitHub.com showed out-of-date and inconsistent information, webhooks stopped arriving and GitHub Pages sites stopped building. Git repositories themselves were never affected, and in the end no user data was lost.

How GitHub’s MySQL failover works

MySQL is GitHub’s main datastore for everything that isn’t Git: issues, pull requests, authentication, the coordination of background jobs. In 2018 that meant multiple clusters, functionally sharded (each holding one area of the product), ranging from hundreds of gigabytes to nearly five terabytes, each with up to dozens of read replicas.

Each cluster uses classic primary-replica replication. One server, the primary, accepts writes and records every change in its binary log. The replicas pull that log and replay it, and serve reads. The replay is asynchronous: the primary confirms a write to the application before any replica has it, so replicas are always a little behind. GitHub also used semi-synchronous replication, where the primary waits up to 500 ms for at least one replica in the same data center to acknowledge a change. So a failover within a data center was expected, though not guaranteed, to lose nothing. GitHub’s own write-up said plainly that it did not expect lossless failover when a whole data center was lost.

When a primary fails, a replica has to be promoted in its place, quickly, and every application has to start sending writes to the new one. GitHub’s machinery for that had three parts:

• Orchestrator detects the failure, picks the best replica and promotes it. It runs as a small cluster that agrees on decisions using Raft, a consensus protocol: the nodes elect a leader, a leader needs the support of a majority, and only the leader acts. GitHub placed one or two Orchestrator nodes in each site so that no single site held a majority, but any group of all the other sites did.
• Consul, used here as a key-value store, holds the identity of each cluster’s current primary. Orchestrator writes the new one there.
• Proxies (GLB on HAProxy) in each data center watch Consul and send all writes for a cluster to whichever server Consul names. Applications never know which machine is the primary.

GitHub had published this design four months earlier. Failovers took between 10 and 13 seconds in most cases. The same post also named an unhandled case: if a data center is cut off while it holds a primary, applications inside it can keep writing to that primary, and the data may be inconsistent once the network comes back. GitHub said it was working on a way to fence the old primary from inside the isolated site. That is close to the scenario that arrived in October.

What happened

At 22:52 UTC, routine maintenance to replace failing 100G optical equipment cut the connection between GitHub’s US East Coast network hub and its primary US East Coast data center. Connectivity came back in 43 seconds.

Those 43 seconds were enough. Orchestrator’s leader, in the primary data center, lost contact with its peers and, as Raft requires, began stepping down. The nodes in the US West Coast data center and in an East Coast public cloud still had a majority between them, elected a leader, and began failing clusters over so that writes would go to the West Coast. When the link came back, the application tier did what the proxies told it: it started sending writes to the new primaries in the West.

The East Coast primaries held a brief period of writes, several seconds’ worth, that had never replicated to the West. Now the West was accepting writes the East did not have. Each side held data the other lacked, which meant neither could simply be made to follow the other.

At 22:54 monitoring began alerting on numerous faults. By 23:02 the first responders had worked out that many database clusters were in an unexpected state: querying Orchestrator showed replication topologies containing only West Coast servers. At 23:07 they locked the deployment tooling to stop any other changes, at 23:09 they set the site status to yellow, and at 23:13 the incident coordinator turned it red.

The database engineering team was paged. Moving the primaries straight back to the East was not safe: the West clusters had been taking the application’s writes for close to 40 minutes, by GitHub’s count, and the East still held its few unreplicated seconds, which blocked replicating the new writes back. Discarding either set would lose users’ data. GitHub decided to keep both and fail forward, knowing the price: applications in the East were not built for a cross-country round trip on most database calls, and the site would be unusable for many people until the topology was fixed.

At 23:19 the team stopped the background jobs that write metadata, such as those processing pushes, and paused webhook delivery and GitHub Pages builds. Data integrity came before usability and before a fast recovery.

The plan, settled by 00:05, was to restore each affected cluster from backup in the East, let the restored servers replicate the new writes from the West, move back to a stable topology and then work through the queued jobs. Backups ran every four hours and were kept for years, but they lived in a public cloud blob store, and restoring multiple terabytes meant hours of transfer, then decompressing, checksumming, preparing and loading the files onto freshly provisioned servers. The restore procedure was tested daily. What had never happened was rebuilding an entire cluster from backup; until then, delayed replicas had always been enough. By 00:41 restores were running for every affected cluster.

At 06:51 several clusters had been restored in the East and were replicating from the West. Pages that wrote to the database were slow, because writes still crossed the country, but reads from the restored replicas were current. The team had found a way to restore directly from the West Coast to get around the download limits, and, extrapolating linearly from replication telemetry, told users to expect recovery in about two hours. At 07:46 GitHub published a blog post explaining the incident. That took extra effort: GitHub uses Pages internally, and Pages builds had been paused for hours.

At 11:12 every primary was back in the East Coast data center, next to the application, and the site became far more responsive. But dozens of read replicas were still hours behind, and requests were spread across the replica pool, so any page could come from a server showing data several hours old.

Why it took 24 hours

Failover worked exactly as configured. Orchestrator’s job was to keep every cluster writable through the loss of a site, and it did that in seconds. The problem was that a primary on the other side of the country was a topology the application could not run on. GitHub’s own analysis says Orchestrator behaved as configured; nobody had seen a partition of that size inside their network, so the gap between what the failover system allowed and what the application could survive had never been exposed.

Asynchronous replication made the fork certain. Any primary that accepts writes in the seconds before a failover holds data its replicas do not. Promote a replica elsewhere and resume writes there, and history has split. On one of the busiest clusters, those few seconds held 954 writes. MySQL replication has no way to merge two histories, so the fix was a rebuild.

The restore path had never run at this size. Restoring one cluster was routine and measured. Restoring many clusters at once, terabytes each, over the link to a cloud blob store, was not. Transfer and preparation took most of the night.

Catch-up did not follow the estimate. Replication lag shrank like a decaying curve, not a straight line, and it slowed further as Europe and then the US started their Monday and write load climbed. By 13:15, close to peak traffic, the delays were growing. What saved it was read capacity: the team had started adding MySQL read replicas in the East Coast public cloud earlier in the incident, and once they came online the read load spread thin enough for replication to catch up.

Protecting data built a backlog. Pausing jobs kept the data safe, but everything they would have done piled up: over five million webhook events and 80,000 Pages builds. When processing resumed, about 200,000 webhook payloads turned out to have outlived an internal TTL and were dropped. The team paused again and raised the TTL for the rest of the backlog.

How they fixed it

At 16:24 on 22 October the replicas were in sync and the team failed back to the original topology, which dealt with the immediate latency and availability problems. GitHub deliberately kept its status red while the backlog ran, rather than declare victory and then report new trouble. At 23:03 every pending webhook and Pages build had been processed, the systems had been checked, and the status went green, 24 hours and 11 minutes after the cut.

The few seconds of East Coast writes that never reached the West were captured from each cluster’s MySQL binary logs. At the time of the analysis GitHub was still reconciling them, working out which could be applied automatically and which needed contacting users; some turned out to have been repeated by users already.

The changes GitHub listed:

• Configure Orchestrator so it cannot promote a database primary across regional boundaries. Leader election within a region is generally safe; the sudden cross-country latency was the major contributing factor.
• Move to a status page that shows individual components, instead of one green, yellow or red for the whole site.
• Accelerate an effort, begun weeks before, to serve GitHub from multiple data centers in an active/active/active design, so that losing a whole site has no user impact.
• Test assumptions more actively, including investment in fault injection and chaos engineering, so failure scenarios are validated before users find them.

What I'd take from it

1. Failover has to respect what the application can survive. Orchestrator was allowed to promote a primary on the other coast, and the application tier could not live with a cross-country round trip on every write. I'd define which replicas may become primary from the application's point of view, and keep automatic promotion inside a region.
2. Asynchronous replication plus automatic failover means forks. If the old primary accepted even a second of writes the replica never received, promoting the replica splits the history in two. Fence the old primary before promoting, or plan for someone reconciling binary logs by hand.
3. Time the whole restore, not a sample of it. GitHub tested restores daily, but had never rebuilt an entire cluster from backup, let alone many at once. The number that matters is every affected cluster, terabytes over the real network path, while production is busy.
4. Replication catch-up is a curve, not a line. Lag shrinks slowly when the primary keeps taking writes, and more slowly as traffic rises. A linear estimate from the first hour will be optimistic; publish a range or no estimate at all.
5. When you pause consumers, extend the queue's TTL in the same change. About 200,000 webhook payloads expired while they waited out the incident. A TTL that suits a healthy queue quietly drops work after a long, deliberate pause.

Sources

• GitHub: October 21 post-incident analysis
• GitHub: October 21 incident report
• GitHub engineering: MySQL High Availability at GitHub (June 2018)