One regular expression, every CPU on the edge
A new WAF managed rule contained a regular expression that backtracks badly on some inputs. It was pushed to every edge server within seconds, CPUs everywhere hit 100%, and visitors to Cloudflare sites got 502 errors until the WAF was switched off globally.
Times are approximate, in UTC, from the public postmortem. The diagram is simplified.
What happened
An engineer deployed a new rule to the WAF managed rules to catch a class of cross-site scripting. It went out in log-only mode, which was meant to make it safe: it would record matches and block nothing. But a rule in log-only mode still runs its regular expression on every request.
The expression contained .*(?:.*=.*). On some inputs a backtracking regex engine tries an exponential number of ways to match that, and the proxy spent all its CPU doing so. The change went out through Quicksilver, the global configuration store, which reaches every server in seconds. There was no staged rollout for WAF rules, because fast rollout was the point: new attacks need new rules quickly.
Cloudflare's own dashboard and API sat behind the same edge, and some internal tools did too, which slowed the response. The team used a global kill switch for the WAF managed rules at 14:07, traffic recovered by 14:09, and the rule was reverted and the WAF turned back on later that afternoon.
What I'd take from it
- "Log only" is not "off". Anything that runs on the hot path can hurt it. Treat a dry run like a real deploy.
- Fast global config needs a staged path too. Even a few minutes on one data center first would have caught this. Speed for emergencies, stages for everything else.
- Use a regex engine with guarantees. Engines like RE2 or Rust's regex run in linear time and can't backtrack. Where that isn't possible, cap the CPU a single match can use.