Load Balancing: One Front Door for a Fleet That Keeps Changing

It’s 8:00 PM on the last Thursday of November. One box has served your storefront all year at a comfortable 20% CPU. Then the campaign email lands in ten million inboxes at once. Connections pile up faster than they drain; the accept queue overflows; the run queue climbs; p99 latency crosses a second, then five. The health dashboard goes from green to red not because the code broke but because one machine cannot be in two places at once. There is no bug to fix. There is only one server where there should have been ten.

The fix isn’t a faster box — you’ll melt that one too next year. The fix is to put many boxes behind one address, and something in front that decides which box each request lands on. That something is a load balancer.

The intuition: the maître d’, not the kitchen

Walk into a busy restaurant. You don’t wander the floor hunting for an open table — a host at the door seats you, watching which sections are slammed and which just cleared. Add two servers to the floor mid-shift and the host simply starts seating those sections too; you never learn their names. Fire a server and the host stops seating that section; the diners already there get reassigned.

That host is the load balancer. Diners are requests, sections are backends, and the whole trick is that the entrance never moves even as the floor behind it churns. Clients hold one stable address — a DNS name, a virtual IP — and the balancer absorbs every change in the fleet behind it.

Where it sits

The balancer keeps a live picture of the pool — who’s up, how loaded — and forwards each request to a member of the healthy set. Everything interesting is in two decisions: how deep does the balancer read the request, and how does it pick a backend.

L4 vs L7: how deep does it look?

The first decision is which layer the balancer operates at. It’s the difference between forwarding a sealed envelope and reading the letter inside.

An L4 (transport-layer) balancer works at the TCP/UDP level. It sees a connection — source IP, destination IP, ports — and pins that whole connection to one backend. It never parses HTTP. It’s a fast, dumb splice: bytes in, bytes out, one connection to one server for the connection’s lifetime.

An L7 (application-layer) balancer terminates the connection, reads the actual request — the HTTP method, path, headers, cookies, TLS SNI — and routes on content. /api/* to the API pool, /images/* to the static pool, a canary header to the canary fleet. It can retry a failed request on another backend, rewrite headers, and terminate TLS. That intelligence costs CPU and a little latency per request.

The rule of thumb: reach for L7 when routing decisions depend on what’s in the request (microservice routing, canary by header, path splits) and L4 when you just need to spray raw connections fast across a homogeneous pool, or the protocol isn’t HTTP at all. Real systems make this an explicit product choice — AWS ships two boxes: the Application Load Balancer (ALB) is L7, the Network Load Balancer (NLB) is L4, and you pick per workload.

Picking a backend: the algorithms

Once the balancer knows the healthy set, it needs a rule to choose one member. The rules climb a ladder from “knows nothing” to “knows load.”

• Round robin — hand out backends in rotation: 1, 2, 3, 1, 2, 3. Zero state, dead simple. Assumes every request costs the same and every backend is identical — often false.
• Weighted round robin — give beefier boxes a bigger share of the rotation. Fixes heterogeneous hardware, still blind to actual per-request cost.
• Least connections — send the next request to the backend with the fewest in-flight requests. Now the balancer reacts to real load: a backend stuck on slow requests stops receiving new ones. Weighted least-connections normalizes that count by capacity.
• Least response time (EWMA) — pick by lowest smoothed latency, blending recent response times with an exponentially-weighted moving average so one slow request doesn’t whipsaw the choice. This is what steers traffic away from a backend that’s technically up but degrading.
• IP hash / consistent hash — hash a key (client IP, a header) to a backend so the same key keeps landing on the same backend. This is how you get stickiness without cookies — and doing it so that adding/removing a backend only remaps a fraction of keys is exactly the job of consistent hashing.

Least-connections is the workhorse default for uneven request costs. Here’s the core — a min-pick over the healthy set, load normalized by weight:

The discipline is in three lines. pick() filters to healthy backends first — an ejected box is invisible, not just deprioritized. It compares active / weight, so capacity is a first-class input rather than an afterthought. And the count is incremented on acquire and decremented on release, which means the signal tracks in-flight work, not requests-ever-sent. Get the release wrong and every backend looks permanently busy.

Health checks and outlier ejection

A balancer that forwards to a dead backend is worse than no balancer — it turns one failure into a steady drip of errors. So the balancer must know who’s alive, and there are two ways to find out.

• Active checks — the balancer probes each backend on a schedule (GET /healthz every few seconds) and marks it down after N consecutive failures, up after M consecutive successes. Proactive, but adds probe traffic and reacts on a timer.
• Passive checks (outlier detection) — the balancer watches real traffic and ejects a backend that starts returning 5xx or timing out, then lets it back in after a cooldown. Reactive and free of probe traffic, but it needs live requests to notice anything.

Production systems use both: active checks catch a backend that went dark, passive ejection catches one that’s up but lying — accepting connections while every response is a 503. Envoy calls the passive side outlier detection and will eject a host from the pool for a growing back-off window after it trips a consecutive-error threshold.

Session affinity, and why it fights elasticity

Sometimes you want a client pinned to one backend — classic reason: session state (a login, a shopping cart) lives in that backend’s memory, so any other backend wouldn’t recognize the user. Sticky sessions do this, usually by having the L7 balancer set a cookie that names the chosen backend, or by hashing the client IP.

It works, and it quietly defeats the thing you bought the balancer for.

The balancer as a single point of failure

Put everything behind one box and that box is now the whole system’s fate. If it dies, ten healthy backends are unreachable. You do not solve availability by creating a new way to be unavailable — so the balancer itself must be highly available, and there are three standard moves, often layered.

• Active-passive VIP — two balancer instances share one virtual IP. A protocol like VRRP keeps a heartbeat between them; if the active dies, the passive grabs the VIP within seconds and traffic continues to the same address.
• Anycast — advertise the same IP from many locations over BGP. The network routes each client to the nearest healthy site; if a site drops, routes reconverge to the next. This is how global edge networks (Cloudflare) survive whole-datacenter loss.
• DNS — return multiple balancer IPs, or use short-TTL health-aware DNS to steer away from a dead region. Coarse and slow (caches, TTLs), but it’s the outermost layer and needs no shared network.

In the wild

Interview corner

Check yourself

References

Official documentation

• NGINX — HTTP Load Balancing (round robin, least_conn, ip_hash) and the ngx_http_upstream module.
• HAProxy — Configuration Manual, balance directive (roundrobin, leastconn, source) and health checks.
• Envoy Proxy — Load Balancing and Outlier Detection documentation (least-request, ring-hash, passive ejection).
• AWS — Application Load Balancer (L7) and Network Load Balancer (L4) developer guides.

RFCs / Papers

• Eisenbud et al. — Maglev: A Fast and Reliable Software Network Load Balancer, USENIX NSDI 2016 (consistent hashing + connection tracking).
• RFC 5798 — Virtual Router Redundancy Protocol (VRRP) Version 3 (the active-passive VIP failover mechanism).
• RFC 7231 / RFC 9110 — HTTP semantics (methods, headers, status codes an L7 balancer routes on).

Engineering blogs

• Cloudflare — A brief primer on anycast and how anycast survives datacenter-level failure.
• Netflix — engineering posts on client-side and zone-aware load balancing (Ribbon / Eureka lineage).

Books

• Martin Kleppmann — Designing Data-Intensive Applications, on scaling and fault tolerance behind a service boundary.