CrowdStrike outage 2024: A content update that crashed 8.5 million Windows machines

At 04:09 UTC on Friday 19 July 2024, just after nine on Thursday evening in California, CrowdStrike sent a routine content update to the Falcon sensor on Windows machines around the world. Machines that received it hit a blue screen, restarted, and hit it again. Microsoft later estimated 8.5 million devices, less than 1% of all Windows machines, but they were the ones running airlines, hospitals, payment systems and emergency call centres. The bad file was live for 78 minutes. Getting the machines back took days, because most of them needed a person.

How Falcon decides what to look for

Falcon is an endpoint detection and response (EDR) agent: software on each machine that watches what programs do and stops the ones that behave like malware. To see everything, part of it runs inside the Windows kernel as a driver, csagent.sys, which loads early in boot so it can watch malware that starts before normal programs do.

That placement matters. Programs in user mode each get a private address space, so when one crashes, Windows cleans up after it and carries on. All code in kernel mode shares a single address space with Windows itself, so a driver that reads or writes the wrong address can’t be contained. If a kernel-mode driver crashes, the whole operating system goes with it. Windows stops the machine, and the blue screen is how it says so: a bug check.

CrowdStrike updates the sensor in two ways. Sensor Content ships with new versions of the sensor itself: code, machine-learning models, and Template Types, which define what a kind of detection can look at. Sensor releases go through automated testing, internal dogfooding and early adopters before general availability, and customers choose whether each group of machines runs the latest version or one or two behind.

Rapid Response Content is different. It is data: Template Instances, sets of matching criteria that threat detection engineers write for an existing Template Type, bundled into channel files and pushed from the cloud to every sensor, so new attack techniques can be caught without changing any code. On the machine, the sensor’s Content Interpreter, a regex-based matching engine, evaluates them.

Think of a Template Type as a form with numbered boxes that the sensor fills in from what it observes, and a Template Instance as a rule saying which boxes to check and what patterns to look for. A wildcard in a box matches any value, so the interpreter never needs to look inside it.

The definition said 21 inputs and the sensor supplied 20. The validator checked content against the definition, never against the sensor.Open in topo ↗

What happened

Sensor 7.11, generally available from 28 February 2024, added a new Template Type for interprocess communication (IPC), to catch attacks that abuse named pipes, a mechanism Windows programs use to talk to each other. Its definition declared 21 input fields. The sensor code that feeds the interpreter supplied only 20. Nobody noticed, for a simple reason: nothing ever asked for the 21st.

On 5 March the new type passed a stress test in CrowdStrike’s staging environment, and the first IPC Template Instance went to production. Three more followed between 8 and 24 April, and all behaved as expected. Every one of them, like the 12 automated test cases written for the type, had a wildcard in the 21st field.

On 19 July two more IPC Template Instances went out, and one of them was the first to put a real matching value in the 21st field. The Content Validator checked it and passed it, because it compared the instance against the definition, and the definition said there were 21 inputs. The March stress test, the earlier successful instances and trust in the validator’s checks all pointed the same way, and at 04:09 the file shipped to every Windows sensor, version 7.11 or later, that was online.

The sensor wrote the new channel file to disk and loaded it. At the next named-pipe notification from Windows, the interpreter evaluated the new instance and went to fetch input number 21.

CrowdStrike’s crash dump analysis shows what happened next. The inputs were an array of 20 pointers, each to a string. The code asked for index 0x14, the 21st element, and got whatever sat in memory just past the end of the array: 0xffffd6030000006a, which is not a valid address. The code checked that the pointer wasn’t null, and it wasn’t, so it read through it. The processor raised a page fault in kernel mode, and Windows stopped with PAGE_FAULT_IN_NONPAGED_AREA, bug check 0x50, in csagent.sys.

The interpreter was designed to handle exceptions from bad content gracefully. This one couldn’t be handled: an invalid memory reference like this in the kernel can’t be caught with a try-except block, as the bug check’s own description says. Then the machine restarted, the driver loaded early in boot, read channel file 291 from disk, and crashed again. Machines were in a boot loop.

At 05:27, 78 minutes after release, CrowdStrike reverted the content. Machines that had been offline during the window never received the bad file. At 05:30 CrowdStrike published a tech alert, and at 06:30 it added workaround steps.

Why it took days

The bad file lives on the disk, so every boot replays the crash. The only automatic way out is to win a race with the network.Open in topo ↗

Reverting the file in the cloud didn’t reach the machines that needed it most. A crashed machine could only fix itself if the sensor connected and downloaded the reverted file before the bad one crashed it again. CrowdStrike’s first advice was to reboot and try, preferably on a wired network, because a machine gets online much faster over Ethernet than over Wi-Fi. Microsoft had reports of machines that recovered after several restarts.

For the rest, the fix was simple and slow: boot into Safe Mode or the Windows Recovery Environment, go to the CrowdStrike folder under System32\drivers, delete the file matching C-00000291*.sys, and restart. It had to be done by a person, at the machine, one machine at a time: laptops, check-in kiosks, servers.

BitLocker disk encryption made it slower. On an encrypted machine, recovery mode can’t touch the disk without that machine’s recovery password, a 48-digit number unique to each device. Microsoft’s instructions had people look the key up on their phone, because the machine in front of them couldn’t. Virtual machines in the cloud had their own version of the chore: detach the disk, attach it to a healthy machine, delete the file and put it back, or roll back to a snapshot from before 04:09.

Customers who had been cautious got no protection from it. Falcon lets customers keep groups of machines one or two sensor versions behind the latest. That control applied to sensor releases, not to Rapid Response Content, which reached every online sensor from 7.11 up at the same moment.

And the damage was concentrated. As Laura Nolan pointed out in USENIX ;login:, whole sectors, aviation and healthcare among them, relied on one software stack, so neighbouring organisations couldn’t cover for each other. Those that coped best could work without their IT: Ryanair boarded planes with paper manifests.

Help arrived over the weekend. Cloud providers published ways to repair VM disks, and by 00:05 UTC on 20 July CrowdStrike’s alert linked to automated recovery for Google Cloud. On 20 July Microsoft published a tool that builds bootable USB recovery media, adding a safe-mode option the next day. On 22 July CrowdStrike introduced automated remediation techniques. On 23 July it added the bad channel file to Falcon’s known-bad list in its cloud, which could let a looping machine with a strong network connection recover by itself. By the evening of 24 July, Pacific time, more than 97% of Windows sensors were back online, and by 29 July about 99%, within the usual week-to-week variation of about 1%.

How they fixed it

CrowdStrike’s root cause analysis, published on 6 August, names three problems together: the mismatch between 21 inputs validated and 20 supplied, the missing bounds check in the interpreter, and the lack of any test with a real value in the 21st field. The fixes, with their dates:

The sensor content compiler now checks how many inputs a Template Type provides. The patch was written on 19 July and in production on 27 July, and it confirmed that no other Template Type, on any platform, supplied the wrong number of inputs.
The Content Interpreter got a bounds check on input reads on 25 July, plus a check that the input array matches what the content expects. Both were backported to sensors from 7.11 up in a hotfix due by 9 August, and the IPC Template Type now supplies all 21 inputs.
Automated tests now put non-wildcard values in every field of every Template Type, and every new Template Instance is tested, not only the first.
The Content Validator gained checks for content that matches more fields than the interpreter receives, due on 19 August. Until then it only allows a wildcard in the 21st field.
Rapid Response Content now goes out in stages: a canary first, then successively larger rings, each with bake time and telemetry, and customers can choose where and when it deploys.
Two independent security firms were engaged to review the sensor code and the quality process from development to deployment.

What I'd take from it

Content is code if code executes it. CrowdStrike's review stresses that channel files are configuration, not code. But an interpreter in the kernel ran them, so they needed a code-grade rollout: a canary ring, bake time, then the world.
Validate where the data is used. The validator checked the content against a definition file, and the interpreter trusted it. A bounds check at the point of use, now added, turns a kernel crash into a rejected file.
Test the values you ship, not just the shape. Every test and every earlier instance used a wildcard in the 21st field, so nothing ever read it. Generate tests that put a real value in every field.
Anything that loads at boot needs a way to skip bad content. If a component crashes twice after loading new content, fall back to the last content that worked. Akamai built this long ago and called it crash rejection.
Plan the recovery for when the machine can't help. Keep recovery keys reachable from a phone, outside the systems that might be down, and practise the manual fix on a few machines before you need it on thousands.