JUNE 28, 2026 · 7 MIN READ
The One-Line Fix That Saved Us From a Redis Meltdown
TL;DR: one Redis call per request × a lot of requests = a bad time. Batch in memory, flush every 200ms. Embarrassingly simple.
Our token bucket was elegant, thoughtful, and completely unable to survive its own success. Here's what happens when good code meets three times the traffic it was designed for.
The setup was clean: one Redis call per request to check and decrement a counter. Neat, atomic, correct. At normal traffic levels, nobody noticed. Then traffic tripled overnight and suddenly that "one quick Redis call" was our most visited piece of infrastructure. Redis wasn't slow. Redis was fine. Redis would like you to know it did nothing wrong.
The fix was almost insulting in its simplicity: buffer counts in memory per instance, flush to Redis every 200ms with a Lua script that reconciles the shared counter atomically. You trade a tiny sliver of precision for a dramatic reduction in Redis traffic. Requests per second on the limiter went from "the bottleneck" to "statistically irrelevant". The lesson, as always: the bottleneck is never where you think it is until it's painfully obvious.
JUNE 15, 2026 · 10 MIN READ
The 3-Week Memory Leak Caused by One Missing Line
TL;DR: un-removed event listener. Three weeks of polite, catastrophic memory growth. One line fixed it. We don't talk about those three weeks.
Memory was climbing. Slowly. Calmly. Like it had nowhere to be and all the time in the world. Here's how one missing removeListener quietly held every request object hostage for three weeks.
node --inspect --max-old-space-size=4096 server.js
// then attach chrome://inspect and take
// two heap snapshots 5 min apart
The pod didn't crash dramatically. It crept. Memory grew slowly enough that alerts didn't fire until week three, when a traffic spike finally pushed it over the edge and the pod OOM-killed itself with the energy of someone who had been patient for far too long. Two heap snapshots, five minutes apart, diffed in Chrome DevTools. The result was immediate and embarrassing: an array of request objects, growing, never shrinking.
The culprit was an event listener registered per-request on a singleton logger, never removed, quietly accumulating a complete record of every request the server had ever seen — headers, body, the works. One removeListener call fixed everything. The fix took thirty seconds. We do not discuss how long the investigation took.
JUNE 2, 2026 · 6 MIN READ
5 Postgres Indexes That Made Me Feel Dumb
TL;DR: one index turned a 4s query into 40ms. The other four were equally obvious in hindsight. You're welcome and I'm sorry.
Five real query plans, five real moments of feeling simultaneously clever and deeply stupid. Here's what a composite index, a partial index, and a covering index look like when you finally add them after not adding them for way too long.
The dashboard query ran in four seconds. Four full seconds, every page load, for every user, for longer than anyone involved would like to admit. The fix was a composite index on (tenant_id, created_at) — the two columns in literally every WHERE and ORDER BY clause in the query. Postgres was scanning the entire table and sorting in memory. With the index, it walked the tree in order and went home early. Forty milliseconds.
A close second: a partial index covering only WHERE status = 'active' rows. About 90% of all queries only ever touched active records. The full index was large, slow to write, and working very hard to index rows nobody ever asked about. The partial index is smaller, faster, and only covers what matters. EXPLAIN ANALYZE is free. Run it earlier than I did.
MAY 20, 2026 · 5 MIN READ
Our CI Was Lying to Us. Here's the Fix.
TL;DR: tests passed. Production broke. Turns out tests only catch what you thought to test. Type-checking and staged rollouts catch the rest.
The pipeline was green. It had always been green. It would be green right up until something broke in production. Here's the setup that finally started telling the truth.
Tests are great at catching things you anticipated. They are completely useless at catching things you didn't — which is, historically, the category that actually breaks production. Adding a strict TypeScript check with zero allowed errors caught more real issues in the first week than the previous month of new tests. Most were type mismatches: signatures that had drifted from their callers, null checks that had been optimistic, return types that had quietly changed. None of them had tests.
The last piece was staged rollouts: every merge to main deploys to 5% of traffic first, watched for 15 minutes against error-rate and latency dashboards, then promoted to 100%. It's caught two regressions since — both passed every test, both broke immediately under real traffic. The tests were right. They just weren't testing the right things.