Replacing a 480-core Spark job with 10 cores of Rust

The MyJio real-time log parser was a Spark 2 streaming job that called out to Perl for the actual parsing. It read one Kafka topic, pulled structured fields out of each log line, and wrote them downstream. It was sized at 120 executors with 4 cores each, around 480 GB of memory, spread over about a dozen physical nodes.

That sizing was not a mistake. It was what the job needed to keep up. Log parsing is almost entirely CPU work on small strings, and a JVM streaming engine brings a lot of fixed cost to that problem: executor heaps, serialisation between stages, garbage collection, and in this case a process boundary to Perl for every batch.

This post is about replacing it with a Rust binary, and what I learned doing it.

The numbers

On a 10 million record replay from the production topic:

Metric	Spark 2 + Perl	Rust
CPU cores	480	10
Memory	480 GB	4.2 GB peak
Footprint	~12 nodes	1 pod
Batch latency	10 s micro-batch	2.3 s average

It processed the 10 million records in about 70 seconds. Across the other application groups the same approach takes roughly 3,500 cores down to about 200, and 8.7 TB of executor memory down to under 200 GB.

None of that is because Rust is magic. It is because the workload never needed a distributed engine. One machine with a tight loop was always enough; the old design just could not express that.

Where 480 GB goes

480 GB across 120 executors is 4 GB each, and very little of it ever held a log line.

Every executor is its own JVM, and Spark carves up the heap before your code sees any of it. 300 MB is reserved outright. Of the rest, 60% by default (spark.memory.fraction) becomes the unified pool that execution and caching share; the remainder is for user objects and Spark’s own bookkeeping. On top of the heap, each container asks the cluster for overhead memory, spark.executor.memoryOverhead, which defaults to 10% of executor memory with a 384 MB floor, to cover thread stacks, direct buffers and native allocations. The garbage collector needs headroom on top of that to avoid long pauses, and here there were also the Perl processes doing the actual parsing.

Multiply all of that by 120 and most of the memory is paying for the runtime, not holding records. The Rust process has one address space, one copy of every compiled regex, no garbage collector, and a working set dominated by data in flight: channel buffers, batches waiting to flush, and the Kafka client’s fetch queues. That is how it peaks at 4.2 GB.

Parity first, speed second

A parser that is fast and wrong is worse than useless, because nobody notices until a downstream report is off by 3% for a month. So before measuring anything I captured 20,000 live production records and compared the Rust output against the legacy output field by field.

In that sample, 1,975 records went through the legacy regex paths, and every one of them had to come out identical. Getting to 100% on that set was the gate for everything that followed, and it is the reason the rollout itself was boring, which is the goal.

Regexes without backtracking

Perl’s regex engine backtracks. It is expressive, and on the wrong input a pattern can take far longer than it should.

Rust’s regex crate works differently. It compiles patterns to finite automata, in the same family as RE2 and Go’s engine, and guarantees matching time linear in the size of the input. In practice most searches run on a lazy DFA that is built on the fly as bytes arrive. The price is features: no look-around and no backreferences.

Two things follow from that for a migration like this:

• Audit the patterns first. Anything that leans on look-around or backreferences has to be rewritten as a plain pattern plus a few lines of code, or handed to fancy-regex, which backtracks only for the parts that need it and delegates everything else to regex.
• Compile once. Building a Regex is expensive; using one is cheap. Compile everything at start-up and share it across worker threads, since compiled regexes are safe to share. When a line has to be tested against many patterns, a RegexSet checks all of them in a single pass.

On the Kafka side, rdkafka wraps librdkafka, the same C client that sits under Confluent’s Python, Go and .NET clients, so fetch batching and consumer-group behaviour are the well-trodden kind.

The shape of the binary

The design is deliberately plain. A Kafka consumer feeds a pool of worker threads through bounded channels. Workers parse and push rows to a batcher, which flushes to the sink on size or time.

// A sketch of the shape, not the production code.
let (tx, rx) = crossbeam_channel::bounded::<OwnedMessage>(QUEUE_DEPTH);
let (out_tx, out_rx) = crossbeam_channel::bounded::<Row>(QUEUE_DEPTH);

for _ in 0..workers {
    let rx = rx.clone();
    let out = out_tx.clone();
    std::thread::spawn(move || {
        for msg in rx {
            if let Some(row) = parse(msg.payload().unwrap_or_default()) {
                out.send(row).ok();
            }
        }
    });
}

Bounded channels matter more than anything else here. When the sink slows down, the batcher blocks, the workers block, and the consumer stops polling. Memory stays flat instead of growing until the pod is killed.

The volume gap

During the parallel run, the Rust parser consistently produced fewer records than the Spark job over the same window. Not by a lot, but enough that the rollout stopped.

It was not a correctness bug. The parity tests still passed on every sample. The Rust side was simply falling behind and catching up in bursts, so any fixed comparison window showed a gap.

Two things were wrong:

1. Oversubscribed threads. I had configured 48 worker threads on a 12-core pod, reasoning that more parallelism could not hurt. It can. Four threads per core on CPU-bound work means constant context switching and cold caches, and throughput went down, not up.
2. One consumer, 110 partitions. A single consumer instance was reading every partition of the topic. Kafka will let you do that, but it caps your read throughput at whatever one client can pull.

Setting workers to the core count and running three pods, so the partitions were split across three consumers, closed the difference to between 0.14% and 0.17%. The job went to production shortly after with no consumer lag, and the approach became the template for the other groups.

From one job to a framework

After the second migration it was obvious the jobs were 80% identical: consume, decode, transform, batch, write. So the common parts became an SDK, and the per-job parts became a manifest. That became DataCraft, which has its own architecture page.

What I would tell someone doing this

• Measure the workload, not the framework. If one core can parse N records a second, you know the floor for the whole thing before writing any code.
• Buy parity with captured production data, not synthetic tests.
• Match threads to cores for CPU-bound work. Scale by adding consumers, not threads.
• Keep the first version boring. Everything clever can come after the old job is switched off.