Rate Limiting: How a Bucket of Tokens Keeps Your API Alive

At 02:14 a mobile client shipped a bug: on any 500, retry immediately, forever. It was one phone. By 02:16 it was forty thousand phones, because the release was already at 60% rollout and they all hit the same error at once. Each device fired the same request several times a second with no backoff. The API tier — sized comfortably for real traffic — spent every thread it had answering a request that was going to fail again, the database connection pool saturated, health checks timed out, and the load balancer started evicting healthy instances. Nobody attacked anything. A client that was technically within its rights to call the API simply called it too much, and the whole service went dark.

The fix isn’t more servers. It’s a doorman: a small, fast decision made before the expensive work starts — has this caller had its fair share in this slice of time? If yes, turn it away cheaply with a 429 and a hint about when to come back. That doorman is a rate limiter.

Why it exists: four jobs, one mechanism

Rate limiting looks like one feature but earns its keep four different ways. Protecting resources — the CPU, connection pool, and downstream databases behind the API have a finite ceiling, and a limiter keeps demand under it so a spike degrades one caller instead of everyone. Fairness — on a multi-tenant API, one greedy tenant must not starve the other thousand; a per-caller limit is how you divide a shared pie. Cost control — when every call fans out to a metered LLM token, an SMS, or a third-party API you pay for, the limiter is a spend cap. Abuse defense — credential-stuffing, scraping, and volumetric DoS all look like “too many requests from one source,” which is exactly the shape a limiter rejects.

From here the members-only walkthrough builds the whole limiter: the five classic algorithms and the token-bucket diagram, the annotated token-bucket code and the atomic Redis + Lua scripts that make it correct across a fleet, where to enforce and the exact 429 / Retry-After response contract, the token-bucket-vs-sliding-window tradeoffs, and an interview corner with a distributed-limiter challenge and a quiz.