fin1te

Outages/CrowdStrike··78 min, then days

A content update that crashed 8.5 million Windows machines

A Falcon sensor content update defined 21 input fields where the sensor supplied 20. The content interpreter read past the end of the array in kernel mode, and every Windows machine that picked it up crashed and kept crashing on boot. The file was pulled after 78 minutes, but each crashed machine needed hands-on repair.

Fig.CrowdStrike, 19 July 20248 components · 7 linksOpen in topo ↗
7 stepsPress play, or step through with → and ←
Impact 10d
T+0mT+10d
passedchannel file 291safe mode, delete the fileWindows hosts · Falcon sensor 7.11+Content Validatorhad a bugFalcon cloudRapid Response ContentSensorkernel driverContent Interpreterexpects 21 inputs, gets 20Windows8.5 million machinesAirlines, hospitals, bankscritical systemsIT teamshands on every machineBitLocker keysone per machine

Times are approximate, in UTC, from the public postmortem. The diagram is simplified.

What happened

CrowdStrike's Falcon sensor runs as a Windows kernel driver. Besides code updates, it takes frequent "Rapid Response Content" updates: configuration that tells it what behavior to look for. At 04:09 UTC a new channel file, 291, went out with new template instances for spotting a certain kind of named-pipe abuse.

The new template instances used 21 input fields. The sensor code that fed the interpreter supplied 20. The content validator, which should have caught the mismatch, had a bug and passed it. When the interpreter reached the 21st field it read memory it did not own, in the kernel, and Windows crashed. The sensor loads early at boot, so machines went into a crash loop.

CrowdStrike reverted the file at 05:27 UTC, 78 minutes later. Machines that were offline in that window, or rebooted fast enough to fetch the fix, recovered. The rest needed someone to boot them into safe mode or recovery and delete the file. With BitLocker on, that also meant finding each recovery key. Airlines, hospitals, broadcasters and banks were among the 8.5 million machines, and by 29 July about 99% of Windows sensors were back online.

What I'd take from it

  1. Content is code if code executes it. A config file that drives a kernel interpreter deserves the same staged rollout as a driver update: canary rings first, then the world.
  2. Validate at the edge that consumes it. The validator ran upstream and had a bug. A bounds check in the interpreter itself would have turned a crash into a logged error.
  3. Plan the recovery for when the machine can't help. Anything that loads at boot needs a way to be skipped automatically after repeated crashes.

Sources