Atharva Pandey Gopherine

When I first started writing Rust, I wrote everything as for loops. Old habits from C. Then someone on my team rewrote one of my loops as an iterator chain and I got annoyed — it looked “slower” to me. More function calls, closures, chaining. Obviously that’s more overhead, right?

I benchmarked it. Same performance. Down to the nanosecond. I looked at the assembly. Identical. That was the day I stopped assuming and started measuring.

A junior on my team once asked why Rust compiles so slowly compared to Go. I gave the standard answer about monomorphization and LLVM, but realized I couldn’t actually explain the full pipeline. So I spent a weekend reading the rustc dev guide and poking at compiler internals. What I found was a surprisingly elegant six-stage pipeline, and understanding it changed how I think about Rust’s design tradeoffs.

The Big Picture

When you run cargo build, your source code goes through six major stages before becoming a binary:

The sync package is where Go’s concurrency tools live when channels aren’t the right answer. sync.Mutex, sync.RWMutex, sync.WaitGroup, sync.Once, sync.Pool, sync.Map — each one solves a specific problem, and using the wrong one is a reliable way to introduce bugs or performance regressions. I’ve misused all of them at various points.

The general guidance is channels for communication, mutexes for protecting shared state. But within the mutex family, the choice between Mutex, RWMutex, and sync.Map has performance implications that matter in hot paths. And sync.Pool is frequently misunderstood — it’s not a general-purpose object pool; it’s a GC-aware buffer recycler.

I profiled a Rust web service once and found it was allocating 47,000 times per request. Forty-seven thousand. Most were tiny — 16-byte strings, 3-element vectors, temporary buffers. Each individual allocation was fast (jemalloc is good), but 47,000 of them at ~30ns each is 1.4ms of pure allocator overhead. Per request. At 10K RPS that’s 14 seconds of CPU time per second, just asking the allocator for memory.

The fix took half a day and cut allocations to about 200 per request. Here’s everything I know about reducing allocations in Rust.

Here’s a bug that’s bitten almost every distributed system I’ve worked on: a service saves a record to the database and then publishes an event to Kafka. The record is saved. The process crashes before the publish. Now the database says the order exists, but the fulfillment service never heard about it. The order sits in limbo forever.

The naive fix — wrapping both operations in a transaction — doesn’t work because Kafka isn’t a participant in your database transaction. You can’t two-phase commit across Postgres and Kafka without a distributed transaction coordinator, and nobody wants to operate one of those in production.

A colleague once asked me to look at a Rust service that was “slow.” They’d already spent a week trying to optimize the JSON parsing layer because “parsing is always the bottleneck.” I ran a profiler. Sixty-three percent of CPU time was spent in Drop implementations, deallocating thousands of small strings that were created and immediately discarded. The JSON parsing was 4% of runtime.

Profiling would’ve found that in five minutes. That’s why this lesson exists.

I once shipped a Rust library with an unsafe block that worked perfectly on x86, passed all tests, and ran flawlessly in production for months. Then someone compiled it on ARM and it segfaulted immediately. The undefined behavior had been lurking the whole time — x86 just happened to tolerate the misaligned read that ARM wouldn’t. If I’d run Miri before releasing, I would have caught it in thirty seconds. Lesson learned the hard way.

The most stressful hours of my on-call life have been in Kubernetes clusters where something was wrong but nothing was obviously broken. Pods in Pending for reasons that weren’t clear. Services returning 503s but all pods showing as Running. Memory usage climbing slowly across a fleet for two days before things started dying. Over years of debugging production Kubernetes issues, I’ve accumulated a set of commands and mental models that cut through the noise. This article is that collection.

Last year I reviewed a PR where someone claimed their new serialization code was “2x faster.” Their benchmark? std::time::Instant::now() called once before and once after. Single run. No warmup. No statistical analysis. The “2x speedup” was thermal throttling on the first run.

Benchmarking is harder than it looks. Let’s do it properly.

Why Naive Benchmarks Lie

Before we get into the tools, let me show you all the ways a naive benchmark can mislead you:

There’s a smell I encounter in almost every codebase I’ve reviewed — including ones I wrote early in my Go career. Someone discovers that err != nil and, out of an abundance of caution, they log it right there. Then return it. Then the caller logs it again. Then the middleware logs it a third time. By the time a single failed database query reaches the response, it’s appeared in the logs four times with slightly different messages and you can’t tell whether four things failed or one thing failed four times.