73 hours, two bugs inside Consul, and monitoring that ran on it
Roblox ran its backend on a single Consul cluster for service discovery, health checks, secrets and scheduling. A day after streaming, a new Consul feature, was switched on for a busy routing service, contention in the streaming code slowed the cluster until the platform went down, and a pathology in BoltDB, the store behind Consul's Raft log, kept some leaders slow even after that was fixed. With the telemetry that would have pointed at both running on the same cluster, finding them took most of the 73 hours.
Times are approximate, in Pacific time, from the sources at the end. The diagram is simplified.
On the afternoon of Thursday 28 October 2021, Roblox began to fail. By 16:35 Pacific time the number of players online had fallen to half of normal, and soon the whole platform was down. It stayed down until Sunday afternoon: 73 hours, by Roblox’s count, for a service that some fifty million people play every day. When Roblox published its postmortem in January, it was written together with HashiCorp, the company behind the software at the centre of it.
How Consul and Raft work
Roblox runs its backend in its own data centers: over 18,000 servers and 170,000 containers, running hundreds of internal services. For those services to talk to each other, each one has to know where the others are right now. That is service discovery, and at Roblox it was one piece of a set of tools commonly called the HashiStack:
- Nomad schedules work: it decides which containers run on which machines and on which ports, and checks that they are healthy.
- Consul keeps the registry of where every service instance is and whether it is healthy. Services ask it where their dependencies live. It also provides session locks for high-availability systems built on top, and a key-value (KV) store.
- Vault stores and hands out the secrets services use to authenticate. Both Nomad and Vault rely on Consul.
Consul’s servers form a small cluster that has to agree on one version of its data. Roblox ran ten: five voters, which hold the authoritative state, and five non-voters, read-only copies that help serve the enormous read load. The voters agree using Raft, a consensus protocol.
Raft works like this. One voter is elected leader. Every write goes to the leader, which appends it to its log and sends it to the other voters. Once a majority (three of the five) have written it down, the change is committed and applied. If the leader stops responding, the others elect a new one; at Roblox that normally happened several times a day. The important consequence: every write in the whole cluster waits on the leader writing its log to disk, and on the network between the voters.
The log cannot grow forever, so Consul regularly takes a snapshot of its current state and deletes the oldest log entries. It keeps the log in BoltDB, an embedded key-value database that stores everything in a single file made of 4 kB pages.
One more piece. Thousands of clients want to know the moment a service’s list of healthy instances changes. The old way is long polling: the client asks, Consul holds the request open until something changes or a timeout passes, answers, and the client asks again. Consul 1.10 added streaming, where the client subscribes and the servers push each change as it happens. It was designed to cut the CPU and network bandwidth that large clusters spend distributing updates. Roblox upgraded from 1.9 to 1.10 for it, and in the following months switched it on for more and more backend services. It worked as advertised.
What happened
On 27 October at 14:00, the day before the outage, Roblox enabled streaming on the backend service responsible for traffic routing. To prepare for the traffic it usually sees at the end of the year, it also increased the number of nodes supporting traffic routing by 50%. Everything ran normally for a day.
At 13:37 on 28 October, Vault’s performance degraded and a single Consul server was running at high CPU. Players were not affected yet. The Consul metrics showed KV writes taking 2 seconds at the median, where they were normally under 300 ms. At Roblox’s scale hardware problems are routine, and Consul copes well with a machine that dies but badly with one that is merely slow, so the team suspected a degraded server and began replacing one. HashiCorp engineers joined around then and stayed for the rest of the incident.
New hardware didn’t help. At 16:35 the number of players online fell to half of normal, and the degradation turned into a complete outage. With Consul unhealthy, services could not reliably find each other, Nomad could not schedule new containers and Vault could not hand out the production secrets services use to authenticate. Consul had become a single point of failure for everything.
The next theory was load: perhaps the system had simply reached a tipping point. The team replaced every node in the cluster with more powerful machines, 128 cores (twice as many) and faster NVMe SSDs. By 19:00 most of the cluster had moved, and it was no healthier: most nodes could not keep up with writes, and the median KV write was still around 2 seconds.
By 02:00 on Friday there was also a symptom nobody could explain: the leader was regularly out of sync with the other voters. The team shut the whole Consul cluster down and restored it from a snapshot taken a few hours earlier, at the start of the outage. That risked losing a little system configuration (not user data), which they judged acceptable. Worried that the internal services’ own traffic would knock it straight over, they blocked access with iptables and planned to let the load back in gradually. The reset went smoothly and the metrics looked good. Then the block was lifted, the discovery and health-check traffic came back, writes returned to 2 seconds, and services began marking themselves unhealthy again. It was 04:00, more than 14 hours in, and the cause was still unknown.
The third theory was a reconnection storm: thousands of containers hammering a freshly restored cluster. So the team cut Consul’s load to the bone. They listed every service that used Consul and rolled out config changes to disable all non-essential use, which took hours given the variety of systems. Services that normally ran hundreds of instances were scaled down to single digits. Health checks went from every 60 seconds to every 10 minutes. At 16:00 on Friday, more than a day into the outage, they tried again to bring Roblox back. It looked good at first, and by 02:00 on Saturday Consul was unhealthy again, this time under far less load.
So the load was not the whole story. The team turned from Consul’s clients to Consul’s internals: debug logs and operating system metrics showed KV writes blocked for long periods. That is contention, many threads waiting on the same shared resource. One idea was that the move to 128 cores had made it worse, so the cluster went back to 64-core servers. That didn’t help either.
The answer came from perf reports and flame graphs of the Consul servers, which showed most of the time being spent in kernel spin locks along the streaming subscription code path. The team disabled streaming for every Consul client, including traffic routing. The change finished propagating at 15:51 on Saturday, and the median KV write fell to 300 ms.
It was not over. Leaders still changed from time to time, which was normal, but some of them showed the same slow writes as before, which was not. Without a root cause, and seeing that the cluster was healthy as long as certain servers were not leader, the team stopped those servers from staying elected.
Why it took 73 hours
Two unrelated bugs had the same symptom. Both showed up as slow KV writes. Fixing the first left the second visible only when particular servers led, which looks like bad hardware or bad luck.
The first was in streaming. HashiCorp explained that it used fewer concurrency controls (Go channels) than long polling. Under very high read and very high write load at once, the updates contended on a single Go channel, and writes blocked. The new 128-core machines were dual-socket servers with a NUMA memory layout, where contention on shared resources gets worse, so the move to more powerful hardware probably made things worse. HashiCorp had benchmarked streaming at similar scale, but this behaviour needed a combination of a large number of streams and a high churn rate.
The second, found by HashiCorp in the days after the outage, was in BoltDB. When Consul deletes old log entries, BoltDB’s file never shrinks: freed pages are listed on a freelist and reused for later writes. Normally updating that list costs almost nothing. Roblox’s workload turned it pathological:
The log store was 4.2 GB but held only 489 MB of data; 3.8 GB was empty. Its freelist held nearly a million free page IDs and took up 7.8 MB, and it was being written to disk again for every log append, even though the data appended was 16 kB or less. The back-pressure filled TCP buffers, and unhealthy leaders took 2 to 3 seconds per write.
The telemetry ran on the thing that was broken. Roblox’s monitoring had a circular dependency on Consul. The systems that would have shown what was wrong were degraded along with it, and the team spent two days testing theories with limited visibility.
Every theory was reasonable, and each took hours to test. Slow hardware is common at that scale. Load was plausible: the routing fleet had just grown by half. A reconnection storm after a reset is a classic. But replacing a cluster’s hardware, resetting its state or rolling config changes out to every service each costs hours, and each one ruled a theory out without pointing to the real cause.
The start from cold was slow too. By 20:00 on Saturday, 54 hours in, Consul was finally stable, but the caching layer, which normally handles a billion requests a second across its layers, was unhealthy and had to be redeployed. That ran into its own problems: scheduling data the caches stored in Consul’s KV was wrong, probably because of the snapshot reset; the scheduler saw one unhealthy node as completely free and kept placing cache jobs there, where they failed; and the cache deployment tool had been built to adjust large deployments already serving traffic, not to build one from nothing.
How they fixed it
The team worked through the night on the caches. At 05:00 on Sunday Consul and the caching system were both healthy. Services that had been shut down or scaled down were restarted at the right capacity and checked, and by 10:00 Roblox was ready to let players back.
It did that slowly on purpose. With cold caches and a system nobody fully trusted yet, a flood of returning players could have tipped it over again. Roblox used DNS steering to let in a random percentage of players and send everyone else to a static maintenance page, raising the share in steps of about 10% while watching database load, cache performance and stability. Some dedicated players worked out the DNS scheme and shared it on Twitter to get in early. At 16:45 on Sunday, 100% of players had access.
Afterwards, HashiCorp and Roblox:
- compacted BoltDB with existing tooling, which resolved the slow leaders, and planned to move Consul to bbolt, a successor to BoltDB without the unbounded freelist growth, deliberately waiting until after the end-of-year peak;
- kept streaming off until a redesigned implementation had been tested at Roblox’s scale, with HashiCorp building benchmarks to reproduce the contention;
- removed the circular dependency, so the telemetry systems no longer depend on the systems they monitor, and added targeted alerts for Consul and BoltDB;
- split some critical services onto their own Consul clusters, started moving data that had been kept in Consul’s KV store to more suitable storage, and deleted a large amount of obsolete KV data, which improved performance;
- built out an additional, geographically separate data center, with plans for multiple availability zones;
- redesigned cache deployment so the caches can come up from a standing start, and worked with HashiCorp on Nomad changes for bringing up large jobs after a long outage.
Roblox usually sees a traffic surge at the end of December. In January it reported that the surge had passed without a single significant production incident.
What I'd take from it
- Monitoring must not depend on what it monitors. Roblox's telemetry depended on Consul, so the outage hid its own cause. I'd give observability its own discovery, storage and access path, and prove it by switching the main platform off in a drill.
- Treat the heaviest client as a new rollout. Streaming had run for months on other services before it was switched on for traffic routing, alongside 50% more routing nodes. A feature that is fine at one load can fail at another; roll it out to the biggest consumer on its own, with a fast way back.
- One coordination cluster for everything is one failure for everything. Discovery, health checks, locks, key-value data, Nomad and Vault all shared one Consul cluster. Split critical workloads into their own clusters, and keep application data out of the coordination store.
- When latency ignores capacity, profile before you scale. Doubling the cores on dual-socket NUMA machines made the contention worse. Flame graphs found the cause after several rounds of hardware changes had not.
- Rehearse the cold start. Caches, their deploy tool and the scheduler were all built for a system already running. Practise bringing the platform up from nothing, and let users back in gradually, as Roblox did with DNS steering in steps of about 10%.