Duplicate rows in ClickHouse, and a proxy that panicked on them
A permissions change on a ClickHouse cluster made a metadata query return more than twice as many rows, and the Bot Management feature file built from it went past a 200-feature limit that Cloudflare's core proxy failed on everywhere. The file was rebuilt every five minutes on a cluster being updated node by node, so the network failed and recovered in waves and at first looked like an attack. Core traffic was back at 14:30 and every service by 17:06.
Times are approximate, in UTC, from the sources at the end. The diagram is simplified.
At 11:28 UTC on 18 November 2025, visitors to sites behind Cloudflare started getting error pages that blamed Cloudflare’s own network. For the next three hours most of the traffic Cloudflare carries failed, recovered, and failed again. It was the company’s worst outage since the regex of 2019, and the first in more than six years to stop most core traffic flowing through its network. It was not an attack. It was a metadata query in ClickHouse that started returning more rows than it used to.
How a request meets Bot Management
Every request to Cloudflare takes the same path. It terminates at the HTTP and TLS layer, passes through the core proxy (internally FL, for Frontline), and then reaches Pingora, which serves it from cache or fetches it from the customer’s origin. The core proxy is where the products live: it applies each customer’s configuration through modules for the WAF, DDoS protection, Workers routing and the rest.
One of those modules is Bot Management. It runs a machine learning model on every request and produces a bot score, a number for how likely the request is to come from automation. Customers write rules on that score: block likely bots, challenge them, let good crawlers through.
A model needs inputs. Each input is a feature, one trait of the request that the model uses to make its guess, and the list of features lives in a feature file. Cloudflare rebuilds that file every few minutes and pushes it to every server, because bots change tactics quickly and the model has to keep up. Speed is the whole point of this pipeline.
The module keeps a hard limit on how many features it will hold. For performance it preallocates the memory for them, sized for 200. The file normally held about 60. A limit more than three times the normal size looks like plenty of headroom, and for years it was.
Two proxies were in service that day. Cloudflare was part way through moving customer traffic from FL to FL2, a rewrite in Rust. Both read the same feature file, and they failed in different ways.
How the same query got a different answer
The feature file is generated from a query against a ClickHouse cluster. ClickHouse splits data into shards across many servers. To query every shard at once you create a table with the Distributed engine, which stores nothing itself: it forwards each query to the real tables on the shards and merges what comes back. At Cloudflare the distributed tables live in a database called default, and the per-shard tables underneath them in a database called r0.
Until that morning, the subqueries a distributed table sends to the shards ran under a shared system account. Cloudflare was changing that so they would run as the user who issued the original query. Query limits and access grants could then be checked per user, and one user’s bad subquery couldn’t affect everyone else. Users already had implicit access to the tables in r0; at 11:05 a change made that access explicit, and it reached the cluster’s nodes gradually.
ClickHouse shows a user the metadata of the tables that user can access. Before the change, system.columns listed the default tables. After it, the r0 tables appeared as well. That was correct. The trouble was a query written on the old assumption:
SELECT name, type
FROM system.columns
WHERE table = 'http_requests_features'
ORDER BY name;There is no filter on database. On an updated node the table name matched in both databases, so each column came back twice, once from default and once from r0; the postmortem says the response more than doubled. The job turns each row into a feature, so the file went past 200.
What happened
At 11:28 a bad file reached the network and the first errors hit customer traffic. (The postmortem’s opening says 11:20; its timeline and its chart put the first errors at 11:28.) In FL2, adding the features to the preallocated buffer returned an error, and the caller handled it like this:
let (feature_values, _) = features
.append_with_names(&self.config.feature_names)
.unwrap();Calling unwrap() on an Err panics. The worker thread died with called Result::unwrap() on an Err value, and the request got a 5xx. Every request that depended on the bots module failed the same way, on every server that had the new file.
Customers still on the old FL proxy saw no errors. Their requests went through, but the module failed to score them and every request got a bot score of zero. Customers with rules that block bots saw large numbers of false positives; customers who didn’t use the score noticed nothing.
Then the strange part: the errors stopped, started again, and stopped. A deterministic bug in a proxy does not do that. The feature file job ran every five minutes, and it ran on whichever part of the ClickHouse cluster answered while the permissions change was still rolling out. A run on an updated node produced a bad file, a run on an old node a good one, and each new file reached every server within minutes. The whole network swung between failing and working. Cloudflare’s chart of 5xx responses shows several sharp spikes that drop back almost to zero, until about 13:00, when every node had the change, every file was bad, and the failure held steady.
An automated test caught the problem at 11:31, an engineer was investigating by 11:32, and the incident call opened at 11:35. The first thing responders saw was Workers KV, Cloudflare’s key-value store, returning a lot of errors. Its front-end gateway runs through the core proxy, so it was failing with everything else, but it looked like the source. From 11:32 to 13:05 the team worked on KV, trying traffic manipulation and account limits to bring it back.
Then Cloudflare’s status page went down. It is hosted entirely off Cloudflare’s infrastructure, and its failure turned out to be a coincidence. But it came during a run of very large DDoS attacks from the Aisuru botnet, and on a network flapping on and off, an attacker hitting both Cloudflare and its status page was the obvious reading. In the incident chat, Matthew Prince, the chief executive, wrote that he worried it was “the big botnet flexing”.
The damage spread past the proxy. Turnstile, Cloudflare’s CAPTCHA replacement, failed to load, and because the dashboard’s login page uses Turnstile, customers without a live session couldn’t log in to change anything. Cloudflare Access, the Zero Trust login product, failed authentication for most users. Latency rose across the CDN too, because the debugging and observability systems, which automatically attach extra detail to uncaught errors, were burning large amounts of CPU on the flood of panics.
At 13:05 the team moved Workers KV and Access onto a bypass through the older proxy. The bug existed there too but did less harm, and their error rates fell. By 13:37 the team was confident the Bot Management file was the trigger and concentrated on restoring a last known good version. At 14:24 they stopped the generation and propagation of new feature files, and a test with the old file worked. At 14:30 a known good file went into the distribution queue, reached every server, and the core proxy was restarted. Most traffic flowed normally again.
Why it took three hours
The flapping pointed the wrong way. An internal fault that clears itself every few minutes doesn’t look internal. It suggested something outside Cloudflare, and the status page going down at the same time made that story feel confirmed.
The first symptom was a victim. Workers KV failed because the proxy failed, but it was the first thing anyone noticed, and about ninety minutes of mitigation went into it.
A failure mode chosen by accident. Bot scoring enriches a request; the request can be served without a score. But one unwrap() turned a bad input into a crash, so a problem in one module became a 5xx for every request that touched it. Cloudflare’s follow-up names two places where assuming failure would have kept customers unaffected: reading the file, which should have fallen back to validated defaults, and the proxy’s call into the module, which should have let traffic through with a passable classification.
Trusted because it was internal. The file came from Cloudflare’s own pipeline. The first fix in the postmortem is to validate Cloudflare-generated configuration the way it validates user input.
The escape hatches ran on the broken thing. Turnstile on the login page kept customers out of the dashboard at the moment they most needed it. Error reporting, built to help debugging, took the CPU the proxy needed.
Six years earlier, Cloudflare’s last outage this size had the same shape: a change designed to reach everywhere fast, consumed on the request path, with no stage in between. In 2019 it was a WAF rule. This time it was a data file that nobody thought of as a deploy.
How they fixed it
The long tail after 14:30 was teams restarting services that had got stuck in a bad state. The dashboard had a second dip from 14:40, when a backlog of login attempts and their retries overwhelmed it; scaling up control plane concurrency brought it back at about 15:30. At 17:06 every service was back to normal.
In the postmortem Cloudflare committed to four things: validate internally generated configuration files like user input, add more global kill switches for features, stop core dumps and other error reports from overwhelming system resources, and review the failure modes of every core proxy module.
A second global outage followed on 5 December, again from a configuration change pushed everywhere at once. Cloudflare declared a “Code Orange”, its name for work that takes priority over everything else, with three goals: roll out configuration changes in stages with health checks, as it already did for software; give every system on the traffic path well-defined behaviour under errors; and fix the break-glass procedures and circular dependencies, like Turnstile on the login page, that slowed the response.
When Cloudflare reported the work complete, it described a new system, Snapstone, that packages a configuration change and releases it gradually with health monitoring and automatic rollback. Proxy modules now fall back to the last known good configuration, and when there is none they fail open or closed, as reviewed case by case. Cloudflare says the same Bot Management change today would be refused by the system and caught early in the rollout, affecting a small percentage of traffic at most. Its engineering rules now ban .unwrap() outside tests and build scripts.
What I'd take from it
- Treat your own generated config as untrusted input. The feature file came from Cloudflare's own pipeline, so the proxy trusted it. Check size, schema and duplicates where the file is consumed, and keep serving the last good version when a new one fails the check.
- Decide how each module fails before it does. Bot scoring enriches a request; the request can be served without it. Write down per module whether it fails stale (last good config), fails open (skip it) or fails closed (refuse), and make the code do that instead of whatever an
unwrap()does. - Name the database in metadata queries.
system.columnsanswers with whatever the current user may see, so the result changes when grants change. Filter ondatabaseas well astable, or better, build the list from a schema you own. - A hard limit needs an alarm well below it. The limit was 200 and normal use was about 60, so nobody watched the gap. Alert on a big jump between two versions of a generated file and refuse to publish it, long before the limit is reached.
- Flapping is a clue. A fault that clears and returns on a steady rhythm usually has a periodic job behind it. List the jobs that run on that period before you go looking for an attacker.