Datadog outage 2023: A systemd patch that took every Datadog region offline at once

At 06:00 UTC on 8 March 2023, Ubuntu’s unattended upgrades woke up on Datadog’s Kubernetes nodes, as they did every morning, and installed a security fix for systemd. Within the hour tens of thousands of nodes had dropped off the network. By 06:31 so much of the fleet was gone that customers could see it: in five Datadog regions, on three cloud providers with no network connection between them, the web app, the APIs and monitors failed together. It was Datadog’s first global incident, and getting everything back took more than a day.

How a Kubernetes node routes a packet

Linux doesn’t have one routing table; it has many. A routing table maps destinations to where packets should go next: this subnet on this interface, everything else via that gateway. Which table a packet consults is decided first by routing policy rules, the list ip rule prints, checked in priority order from the lowest number up. A fresh Ubuntu machine has three:

0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

The local table holds the machine’s own addresses. The rule at priority 0 is what makes a packet addressed to the node’s own IP get delivered to the node, instead of being routed somewhere else.

Datadog gives pods real addresses from the cloud network (the VPC), with no overlay. That is efficient, but it means the network plugin, Cilium, has to manage the node’s routing. On AWS, Cilium adds extra network interfaces to each node for pod traffic, and the cloud network only accepts a pod’s packets if they leave through the interface that owns the pod’s address. So Cilium adds rules: traffic to a local pod stays on the host, traffic from a pod uses a separate table whose default route points at the pod interface, and a few more handle its L7 proxy. For an edge case in that proxy, Cilium also moves the lookup local rule from priority 0 to 100. A node with one pod looked like this in Datadog’s example:

9:      from all fwmark 0x200/0xf00 lookup 2004   # L7 proxy
20:     from all to 10.132.126.226 lookup main    # to the pod
100:    from all lookup local                     # moved by Cilium
109:    from all fwmark 0x80/0xf80 lookup main
111:    from 10.132.126.226 lookup 11             # from the pod
32766:  from all lookup main
32767:  from all lookup default

Azure works much the same way. On Google Cloud, pods share the node’s main interface through alias IP ranges, so there are no rules for pod egress, but Cilium still moves the lookup local rule.

Each packet walks the rules in priority order until one picks a table. Remove Cilium's rules and everything falls through to main, out of the wrong interface.Open in topo ↗

A change from 2020, waiting for a restart

In December 2020, systemd v248 changed systemd-networkd, the systemd component that manages network configuration on a host: when it starts, it flushes every routing policy rule it doesn’t know about. Five months later v249 added a setting to opt out, ManageForeignRoutingPolicyRules=, which defaults to yes. Ubuntu 20.04 ships systemd v245 and never had the behaviour. Ubuntu 22.04 ships v249 with the default.

Datadog began testing Ubuntu 22.04 in June 2022 and rolled it out progressively from November; by 8 March it ran on more than 90% of the fleet. The flush had been there all along, and it never fired, because systemd-networkd only started when a node booted, before Cilium had installed any rules. A fresh node was fine. A rebooted node was fine. Only restarting systemd-networkd on a running node would show the problem, and nothing did that.

Datadog doesn’t patch nodes in place. It replaces them, region by region and cluster by cluster, with a week to a month between stages. But its base image had never switched off Ubuntu’s unattended upgrades. They were configured with Ubuntu’s defaults: download updates twice a day, and install security updates once a day at 06:00 UTC with a random delay of up to 60 minutes. With a minimal image, they had rarely had anything to install, and never anything disruptive.

What happened

On 7 March a fix for a systemd CVE was published in Ubuntu’s repositories, version 249.11-0ubuntu3.7. Installing a systemd update restarts all of systemd’s components, including systemd-networkd. According to Ubuntu’s publishing history there had been no systemd security patch since at least September 2022, before Datadog began deploying 22.04, so this had never happened on those nodes.

From 06:00 on 8 March, every 22.04 node that had already downloaded the new package installed it. (Nodes that hadn’t were spared: Datadog estimates the package became available to Google Cloud nodes at about 18:50 on 7 March, so Google Cloud nodes whose evening download ran before then weren’t hit.) systemd-networkd restarted and flushed the rules. The listing shrank to two lines:

32766:  from all lookup main
32767:  from all lookup default

On AWS and Azure, traffic from pods lost its rule and left through the node’s main interface, where the cloud network dropped it. On all three providers, the lookup local rule at priority 100 was deleted and the original one at 0 was never put back, so traffic addressed to the node’s own IP was no longer local. Replies arriving for the node followed the default route straight back out of the interface and were dropped. The node lost the network entirely.

Nothing connected the regions: no network, no shared control plane, no common deploy. The image and its default timer did, and they fired in the same hour everywhere.Open in topo ↗

Internal monitoring flagged a problem at 06:03. At 06:08, about 1 a.m. for the first responders, two teams were paged: an APM team whose pods were not restarting properly, which reasonably assumed a local problem, and the team that receives Datadog’s out-of-band alerts, monitors that run entirely outside Datadog’s infrastructure and use its APIs like a customer. The second team saw trouble across products and called a high-severity incident, which took about ten minutes to open because the incident tool itself was failing intermittently. The first status page update went out at 06:31.

Diagnosis meant unlearning what everyone knew. Datadog’s regions share no infrastructure, and every change is rolled out one region at a time, so nobody in the room knew of anything that could touch all of them at once. Responders pulled in the networking team, the compute team, because pods were failing to start or stuck in restart loops, and the web UI team, which had made the last change to a shared configuration utility that might conceivably have acted globally. At 07:20 they established that Kubernetes nodes were failing everywhere; by 08:00 they had confirmed it had stopped spreading. Unattended upgrades weren’t identified as the trigger until 11:36.

The damage also depended on the cloud. On Google Cloud and Azure, an instance with no network simply keeps running, and because Datadog relies on Kubernetes rather than cloud health checks to replace nodes, nothing replaced them. A reboot through the cloud API brought them back with their disks intact. On AWS, failed status checks told the Auto Scaling groups to terminate and replace the instances: more than 8% of US1’s instances every ten minutes between 06:00 and 07:00, and about 60% of the region, tens of thousands of instances, by 08:00. Many of Datadog’s data stores, coordination systems and caches keep their state on local disks, which are fast and cheap, with replication handled by the data stores themselves. On AWS those disks went with the instances.

At first the AWS region looked healthier, because its stateless services came back on fresh instances while web traffic on Google Cloud and Azure was still down. Later it became clear that AWS was the harder case: the other regions needed reboots, but US1 had lost a large amount of data that had to be rebuilt, reprocessed or restored from backups.

Why it took 27 hours

The monitors went down with the monitored. Datadog watches Datadog with Datadog. The out-of-band checks raised the alarm, but most of the tools responders normally debug with were in the same broken regions, and so were the CI/CD and automation tools.

Recovery had to go in order. Datadog runs parent clusters whose nodes host the control planes of child clusters, where the applications run. In EU1 the parent control planes had to be rebooted first (done by 08:45), then the parent nodes hosting the child control planes (09:30), then thousands of application nodes, prioritised by workload and paced so the control planes weren’t overwhelmed: 60% by 10:20 and all by 12:05. Web access started coming back at 09:13.

Replaced nodes are new nodes. In US1 every replacement had to go through registration: authenticate to Vault, which checks the instance with AWS APIs and signs its certificate, join the cluster, then wait for the Cilium operator to attach a network interface with pod addresses. Tens of thousands of nodes did this at once. AWS rate-limited the calls, and more than 99% of the operator’s interface requests were throttled between 06:00 and 15:00. A bug in the operator made it worse: it bypassed its own rate limiter and retried every few milliseconds, seven to eight thousand calls a minute per cluster. Three metrics clusters of more than 4,000 nodes each filled with nodes that had registered but had no network. After deleting unready nodes and a limit increase from AWS at 14:55, US1’s compute capacity was back at 15:15.

Catching up needed more than twice the capacity. To process live data and the backlog together, services scaled beyond anything Datadog had tested; it kept headroom for 50% growth, not 100%. In EU1, instance creation stopped at 14:18 at Google Cloud’s documented limit of 15,500 instances in a VPC peering group, which nobody had checked; Google raised it at 18:00, and three clusters then filled their subnets until 00:45. In US1 Vault became unstable from 14:30 under the load and the AWS rate limits, so new nodes couldn’t get certificates, until AWS raised the limit, Datadog put rate limits in front of Vault, and it steadied at 19:00.

Quorum systems stop; sharded ones sag. Datadog noticed that its statically sharded stores, such as those holding point values and raw logs, lost capacity roughly in proportion to the nodes lost. Its metadata stores, which need a quorum, worked normally until quorum was gone and then refused to work at all. Much of the service recovery went into getting those back to accepting writes.

How they fixed it

The order of work was fixed early: in each region, recover enough compute to process incoming data (Datadog scaled above normal, with help from each cloud provider), then recover every service in parallel, putting live data and alerts ahead of history. The first major service was declared operational at 16:44 on 8 March, every service in every region at 08:58 on 9 March, and the incident was closed at 06:25 on 10 March, once historical data had been backfilled.

Mitigations against a repeat were rolled out from 15:54 on the first day. Datadog disabled the legacy security update channel in every affected region (security patches keep arriving through its own node replacement), configured systemd-networkd to leave the routing table alone on restart, and audited its infrastructure for similar update channels. A few days later the Cilium team made sure systemd could not remove the rules Cilium configures, and Datadog worked with them upstream on the rate limiter bug.

The deeper lessons Datadog drew were about correlation. Separate regions on separate clouds hadn’t been enough, because they ran the same operating system, container runtime and orchestrator. It now plans for global failure, wants the platform to keep serving live data and alerts in a degraded mode rather than going fully dark, and intends to test chaos and scaling at the size this day demanded.

What I'd take from it

Switch off every update path you don't use. Datadog patched by replacing nodes region by region, but its base image still had Ubuntu's unattended upgrades on, a second path that skipped every stage. Audit images for background updaters, timers and agents that can change a running node.
The same default everywhere is a global dependency. Isolated regions on separate clouds still ran the same image with the same timer, set to the same hour. Treat a shared image and its defaults as a global component, and stagger anything scheduled by the clock.
Tell the host's network manager what it doesn't own. When a CNI installs routes and rules, configure systemd-networkd to leave them alone (ManageForeignRoutes=no, ManageForeignRoutingPolicyRules=no). Test by restarting the manager on a busy node, not only by booting a fresh one.
Know what auto-repair does to stateful nodes. On AWS, failed status checks replaced instances and discarded their local disks; on Google Cloud and Azure a reboot brought them back intact. Decide which you want for each workload before an incident decides for you.
Plan recovery for double, not one and a half. Datadog kept headroom for 50% growth; clearing the backlog needed more than twice normal capacity, and a peering limit, full subnets, cloud API throttling and Vault all failed at once. Rehearse the recovery at that scale, cloud quotas included.