Writing
Notes from production.
Long-form write-ups of problems worth remembering. Subscribe with RSS.
- Moving 400+ Spark jobs from DStreams to Structured StreamingA naive port made our hardest job twice as slow and quietly lost data. Here is what it took to reach parity and then pull ahead: bounded batches, fewer scheduling waves, a watermark guard, two timezone bugs and one very deep query plan.6 min
- Building the Spark UI that Structured Streaming should have hadThe stock Spark UI forgets a streaming query the moment it stops, and it cannot tell you how far behind Kafka you are. So I built a Streaming Stats tab into our framework: a SparkPlugin, a query listener, a ring buffer and server-rendered SVG.5 min
- Replacing a 480-core Spark job with 10 cores of RustHow the MyJio log parser went from 120 Spark executors to a single pod, what parity testing looked like, and the thread-count bug that almost made it look worse than it was.5 min
- Sub-20 ms lookups on trillion-row tables: ClickHouse sort keys in practiceSubscriber lookups went from full-partition scans in the hundreds of milliseconds to under 20 ms reading three granules. The fix was one line of DDL, and the reasons it works are worth knowing properly.4 min
- The 497-day bugHealthy switches and routers were showing up as down in SLA reports. The common factor was uptime: about 497 days of it, which is exactly where a 32-bit counter of hundredths of a second runs out.3 min
- How a cloud SLA is actually calculated99.95% availability leaves about 22 minutes of downtime a month. Deciding which minutes count is the hard part: HA pairs, reboots nobody saw, outages that span a weekend, maintenance windows and assets that did not exist yet.5 min