Kubernetes retired a label, and Reddit's route reflectors vanished
Upgrading Reddit's most important legacy Kubernetes cluster from 1.23 to 1.24 removed the old master node label, and the Calico route reflectors that every node learned its routes from were selected by that label. Every node dropped its pod routes at once, and with no supported downgrade the team restored the control plane from a backup whose procedure had never been run in production. Reddit was down for 314 minutes on Pi Day.
Times are approximate, in UTC, from the sources at the end. The diagram is simplified.
Just after 19:00 UTC on 14 March 2023, an engineer on Reddit’s Compute team started upgrading one of the company’s most important Kubernetes clusters, the one that runs the legacy part of the stack the community knows as Old Reddit. For a couple of minutes everything looked fine. Then the site came to a halt. For the next 314 minutes, on Pi Day of all days, Reddit showed error pages or a home page with no posts. The cause was a single node label, and nobody found it until well after the site was back.
How pods find each other: Calico, BGP and route reflectors
Every pod in a Kubernetes cluster gets its own IP address. When a pod on one node talks to a pod on another, the sending node has to know which node owns the destination address. Some network plugins hide this behind an overlay, wrapping pod traffic inside packets addressed to the other node. Calico, the plugin Reddit uses (its CNI, or container network interface), can instead put real routes in each node’s routing table, such as a route saying that pod addresses in 10.1.2.0/26 are behind node B.
Calico learns those routes with BGP, the Border Gateway Protocol, which routers use to tell each other which addresses they can reach. Each node runs calico-node, and inside it a small BGP daemon called BIRD. Each node announces its own block of pod addresses to its BGP peers and installs the routes it hears from them.
By default every node peers with every other node, a full mesh. That is simple and works well for small clusters, but the number of sessions grows with the square of the node count: 100 nodes need 4,950 sessions, 1,000 nodes nearly half a million, and every route change has to reach all of them. Calico’s documentation suggests route reflectors beyond about 100 nodes. Reddit’s post says its clusters add zeros to that.
With route reflectors, a few nodes peer with everything and pass each node’s routes on to all the others. Every other node peers only with the reflectors, and the full mesh is switched off. It scales well. It also means that if the reflectors disappear, every node loses every route it learned from them, all at once.
In Calico you make a node a reflector by giving it a cluster ID in Calico’s own node resource, which is normally edited with Calico’s CLI rather than through the standard Kubernetes API. Then BGPPeer resources, which use label selectors, say who peers with whom. Reddit’s reflectors were the control plane nodes, which made sense: in an autoscaled cluster they were the only nodes guaranteed to be there. The BGPPeer that Reddit published in its postmortem looks like this, trimmed:
apiVersion: crd.projectcalico.org/v1
kind: BGPPeer
metadata:
name: rr-mesh
spec:
nodeSelector: has(node-role.kubernetes.io/master)
peerSelector: has(node-role.kubernetes.io/master)The label node-role.kubernetes.io/master was the whole connection between Calico and the reflectors. Kubernetes had been retiring that word for two years. Since 1.20, kubeadm, the tool Reddit built these clusters with, had applied node-role.kubernetes.io/control-plane alongside the old label. In 1.24 it stopped applying the old one, and the upgrade command removes it: when kubeadm upgrade apply runs, it finds every node carrying the old label and deletes it, not only the node being upgraded.
What happened
The team had just finished the postmortem of a previous upgrade of this same cluster, which had gone badly in a mild way and for a cause that was fixed. Upgrades at Reddit are tested on dedicated clusters and rolled out from the least critical production clusters to the most. This was one of the most critical.
Responders were in a call three minutes after the site stopped. The first thing they found was that the cluster had lost all of its metrics. Reddit’s graphs of traffic at the CDN edge, kept separate on purpose, showed the collapse, but inside the cluster they were blind. The one clear symptom was DNS: names in Consul, which Reddit runs for cross-environment dynamic DNS, and in-cluster names would not resolve, yet public DNS names resolved fine. Nobody had seen this in any earlier upgrade or in tests.
Rolling back is normally plan A. Kubernetes has no supported downgrade: an upgrade runs schema and data migrations automatically, and there is no path back, so a downgrade means restoring the control plane’s state from a backup. Reddit takes a backup before every upgrade. But the restore procedure had been written years earlier, had fallen behind changes in the environment, and had never been used on a production cluster, let alone this one. Early estimates were hours of guaranteed downtime. The team chose to keep fixing forward.
Half an hour in there was still no clear lead. About a half-dozen engineers worked hands-on while dozens watched. After another half hour, a group from the Compute team split off to prepare the restore, just in case. The others kept digging, and the symptoms kept piling up. Pods took a very long time to start and stop. Container images under 100 MB took minutes to pull over a multi-gigabit link. Calico’s own pods were unhealthy: calico-kube-controllers was stuck in ContainerCreating, which looked like a known, low-severity CRI-O bug, until deleting the pod changed nothing. They deleted the calico-typha pods, and new ones didn’t appear. They restarted the control plane components, then turned the whole control plane off and on. Nothing.
Then someone noticed that API server writes were timing out, not on the writes themselves but on calls to admission webhooks. The only one watching all resources on this cluster was Open Policy Agent (OPA), which was down anyway, so they deleted its webhook configurations. The timeouts vanished instantly. The cluster stayed broken.
More than two hours in, the team made the call to restore. They first terminated every worker node, since the restore would invalidate them anyway; on their largest cluster that took about 20 minutes of API calls. Then they ran a procedure nobody present had ever performed: terminate two of the three control plane nodes, downgrade the remaining one, restore the data onto it, and join new nodes to it. It had been written for an end-of-life Kubernetes version and for Docker, before Reddit moved to CRI-O, so commands and arguments had changed and the procedure was rewritten as they went.
The first node came back, and within moments new nodes began joining as the cluster autoscaler woke up, a sign that networking worked again. The team paused the autoscaler, because one control plane node would not survive a large cluster’s load. The replacement control plane nodes were delayed because AWS had run out of capacity for their instance type. When they did launch, they couldn’t join: each got stuck connecting to etcd on the restored node. The backup procedure could run on any control plane node, but the restore had to happen on the same node, and it hadn’t, so the TLS certificates the restored node presented carried the wrong hostname. With some fumbling, and little documentation, they generated new certificates and the control plane was whole again.
Meanwhile the rest of the team brought traffic back very carefully, starting at about 1%. Reddit relies on many caches, and idle services scale down during an outage, so turning everything on at once invites a thundering herd. They stepped through 5, 10, 20, 35, 55 and 80% to 100%, and switched some touchy legacy services back on by hand.
Finding the needle
The site was back, and nobody knew why it had broken. Metrics were no help, since they had died with the cluster. The logs, though, were collected by a low-level agent that had kept working, and they were all there, some 3.9 billion lines of them.
API server log volume jumped fivefold at 19:04:49. The OPA logs stopped about five seconds before that, which looked like a dead end. Calico’s logs showed the rest. Two seconds before the flood, calico-node across the cluster started reconfiguring BIRD and shutting down BGP sessions. Dropping routes to the control plane node being upgraded was expected. Dropping the routes to every other node was not.
One engineer happened to remember that Reddit used route reflectors. They had been set up years earlier by the team that preceded Compute, everyone who knew about them had since moved on, and only the largest and most legacy clusters used them. Worse, part of their configuration had been fetched with Calico’s CLI, edited by hand and uploaded back, so it was committed nowhere. There was nothing for anyone to find.
Put together: the upgrade of the first control plane node removed the label from all three, the selectors matched nothing, and every node’s BGP daemon dropped its sessions to the reflectors in the same second. Pods lost their routes to every other node. That fits everything the responders saw: in-cluster DNS failing, the API server unable to reach the OPA webhook, Calico’s components unable to talk to each other. Every symptom they chased was downstream of one missing label.
It also fits why the restore worked, though the post doesn’t say so: it put the cluster back in its 1.23 state, where the control plane nodes carry the old label, so the selectors matched again and the reflectors returned. Nobody knew yet that the rollback had fixed the cause as well as the symptoms.
Why it took 314 minutes
The instruments died with the patient. Metrics were Kubernetes-native, so they depended on the pod network. The one independent view was the CDN edge, which could say that the site was down but not why.
Every symptom was real, and none was the cause. DNS, slow pods, stuck Calico pods and webhook timeouts all pointed somewhere plausible. Fixing the webhook proved each fix could clear a symptom without touching the fault.
Nothing in testing looked like this cluster. Only the largest, oldest clusters used route reflectors, so the upgrade path that was tested elsewhere didn’t exercise them. Reddit calls the deeper cause inconsistency: nearly every critical cluster was bespoke in some way, which it describes as having raised pets rather than cattle.
The way back was one-way and untested. With no downgrade, the only rollback was a restore, and the restore carried years of drift: Docker commands on a CRI-O cluster, an assumption about which node it ran on, and certificates tied to hostnames. Add an AWS capacity shortage for the control plane instance type, and every step of the rollback found its own problem.
How they fixed it
The outage ended with the restore and the stepped return of traffic described above. The fix for the future is broader, and Reddit framed it as not wasting a good crisis. It committed to making cluster upgrades safe and to the longer project of making its clusters alike: more components standardised and shared, more pre-production clusters to test against, tooling to manage the lifecycle of whole clusters so they can be recreated, and everything codified where it can be, with the rest documented in detail and treated as an exception to replace. The backup and restore process is on the list too: Reddit said it was full of pitfalls and must be improved, even though it worked.
Reddit’s newer service API layers stayed up throughout, though they couldn’t help much when the node they all depended on went down.
What I'd take from it
- Every label selector is a dependency. Before an upgrade that changes node labels, search every selector in the cluster for the old name: workloads, webhooks, and CNI resources such as Calico's BGPPeer, which a normal
kubectl get allwon't show. - If it isn't in git, it doesn't exist. The route reflector setup lived only in Calico's datastore, edited by hand years earlier by people who had left. Export anything managed through a vendor CLI into version control, even if it takes a small script, so the next engineer can find it.
- Rehearse the restore on the cluster that matters. The backup was fine. The procedure was written for Docker, an end-of-life Kubernetes version and one particular node. Run the restore against a production-shaped cluster on a schedule, and time it, so the decision to roll back isn't a leap in the dark.
- Make backup and restore agree on where they run. The backup could run on any control plane node; the restore assumed the same node, and certificates tied to hostnames made the difference fatal. Write down the assumptions of both halves in one place and test them together.
- Bring traffic back in steps. Reddit returned traffic at 1%, then 5, 10, 20, 35, 55, 80 and 100%, so caches and services that had scaled down could warm up. After a long outage, full load in one go can cause the next one.