Deep Dive
system design interviewalgorithmscachingconcurrency

Design a Distributed Rate Limiter

Picking token bucket vs sliding window is the easy part. The real interview is what happens when the limiter runs on fifty machines: where the counter lives, why 'local counters' quietly lets 50× your limit through, and how to enforce one global cap without a round-trip on every request.

·17 min read
Hard
Asked atStripeCloudflareGitHubAmazonGoogle

The algorithm question — token bucket, leaky bucket, sliding window — is the warm-up, and it has a clean answer covered in depth in the Rate Limiting deep dive. This case study is the follow-up that actually separates candidates: you have one global limit and a fleet of machines. Where does the counter live?

The trap: “just count locally on each node”

It’s the answer everyone reaches for first, and it’s wrong in a specific, quantifiable way. If each of your N nodes independently allows up to the limit L, the fleet as a whole admits up to N × L — because no node can see what the others have let through.

4 × LLoad balancerNode 1local ≤100Node 2local ≤100Node 3local ≤100Node 4local ≤100Backendgets ~400/sec
One global limit of 100/sec, four nodes, each counting locally. A load balancer spreads traffic evenly, so each node sees ~100 and allows all of it — the backend receives ~400/sec, four times the intended cap. The limiter is 'working' on every node and still failing globally.

The three places the counter can live

Central store (Redis)
  • One shared counter — the limit is exact
  • Atomic INCR: no cross-fleet race
  • Cost: a network round-trip per request
  • The store becomes a hot path + dependency
Local / divided counters
  • Local, no sync: fast, but admits N × L
  • Divided (L/N per node): holds the cap…
  • …until traffic skews and hot nodes starve
  • No round-trip — coordination is the price

Every production design lands somewhere on this axis — exact-but-coordinated versus fast-but-approximate — and the interesting systems (Stripe, Cloudflare) deliberately choose approximate at the edge and exact only where money is on the line.

The members-only build makes this concrete: an interactive fleet where you watch “local counters” overshoot to N× the limit and “divided quota” starve a hot node in real time, the atomic Redis approach and why INCR beats read-modify-write, where to enforce (edge vs gateway vs service) and the fail-open-vs-fail-closed decision, the 429 + Retry-After contract, and the interview corner on making a global limit both correct and cheap.

Members only

Keep reading with Premium

You've reached the members-only part of this deep-dive — the full implementation, the interactive ring simulator, and the step-by-step walkthrough. Unlock it with a membership.

Go PremiumAlready a member? Sign in