Atharva Pandey Gopherine

I spent a Tuesday afternoon tracking down why our service continued doing expensive database work after clients had long since disconnected. An HTTP client with a 3-second timeout would cancel the request, but our handler would still run three downstream database queries that together took 8 seconds. The handler was checking for errors but never checking the context. By the time it finished, the client had retried twice, and we were now running three copies of the same 8-second work simultaneously.

There’s a moment in every Rust developer’s life when they try to send an Rc<RefCell<Vec<String>>> to another thread and get a wall of compiler errors. They Google the error, see something about Send and Sync, patch the type to Arc<Mutex<Vec<String>>>, and move on without understanding why.

I did exactly that for six months. Then I needed to write a custom type that crossed thread boundaries, and I had to actually learn what these traits mean. Turns out they’re beautifully simple once you see the design.

After I understood how hashicorp/go-plugin worked with net/rpc, the next question was obvious: how does Terraform manage hundreds of community-contributed providers, written by different teams, evolving on different schedules, sometimes in languages other than Go? The answer is gRPC. Terraform’s provider protocol is a protobuf schema, communicated over the same subprocess-plus-RPC architecture from Lesson 1 — but with gRPC instead of net/rpc. That substitution buys you schema evolution, multi-language support, streaming, and a strongly-typed IDL. This lesson shows you how to build a plugin system using that pattern.

I’ve integrated all three of these systems in production Go services, and the question I get most often is: “Which one should I use?” The honest answer is that it depends on what guarantee you actually need — and most engineers pick based on familiarity or hype rather than requirements. NATS, RabbitMQ, and Kafka solve meaningfully different problems, and choosing the wrong one creates operational pain that no amount of clever application code can fix.

Memory ordering is the thing that separates people who use concurrent code from people who write concurrent primitives. I avoided understanding it for years, using SeqCst everywhere like a safety blanket. It worked. But it left performance on the table and, more importantly, left me unable to read half the lock-free code I encountered.

So here’s the actual explanation — no hand-waving.

Why Ordering Matters

Modern CPUs and compilers reorder instructions for performance. Your code says “write A, then write B,” but the CPU might execute B first if there’s no dependency between them. On a single thread, this is invisible — the final result is the same.

CAP theorem is probably the most cited and most misunderstood concept in distributed systems interviews. Candidates memorize “you can only pick two of consistency, availability, and partition tolerance” and then either over-apply it (treating every design decision as a CAP trade-off) or under-apply it (never relating it to actual design choices). The theorem is real and important, but the way it’s usually taught in 30-second summaries strips out the nuance that makes it actually useful. This final lesson clears that up.

The time package looks straightforward right up until the moment a bug report arrives saying “the nightly job didn’t run last Sunday.” It never runs on the Sunday when daylight saving time ends. Because that Sunday has 25 hours, and the cron expression fires twice at the ambiguous time, and the second firing is skipped because the code thinks it already ran. I’ve debugged this exact bug, and the root cause is always the same: someone — usually me — assumed time is simpler than it is.

I once replaced a Mutex<u64> counter in a hot path with an AtomicU64 and saw throughput jump 40%. Not because mutexes are slow — they’re fast. But for a single integer being incremented by 32 threads, the overhead of acquiring and releasing a lock millions of times per second adds up to real time.

Atomics are the foundation of lock-free programming. They let you do thread-safe operations on primitive values without any lock at all.

I want to end this course with the idea I wish I’d understood at the start — not just intellectually, but in my gut. It’s this: duplication is a problem you can see. Bad abstraction is a problem you can’t see until it’s too late.

Duplicated code is visible. You grep for it, you find it, you fix it. It’s annoying, but it’s honest. Bad abstraction hides behind clean-looking code. It’s the function that requires fifteen minutes of context to understand, the type parameter that nobody knows how to satisfy, the constraint that’s technically correct but practically useless. It costs you in onboarding, in debugging, in every change that touches it for the rest of the codebase’s life.

I remember staring at Arc<Mutex<HashMap<String, Vec<u8>>>> in a codebase and thinking “this is the ugliest type I’ve ever seen.” Three months later, after debugging a similar system in Go that had zero type safety around its concurrent map access, I came crawling back to Rust’s ugly-but-correct approach.

The Arc<Mutex<T>> pattern is everywhere in Rust concurrent code. Understanding why it exists — not just how to type it — is the key.