Database Replication: Copies That Disagree With Each Other

A user posts a comment. The write lands on the primary, the request returns 200, the page reloads — and their comment is gone. They post it again. Now there are two. They refresh once more and one of the duplicates vanishes, then reappears.

Nothing is broken. The write went to the leader; the reload was served by a read replica that hadn’t received the new row yet. You added that replica last week to take read load off the primary, and it worked — reads got faster and cheaper. What you also bought, without noticing, was a window of time during which two copies of your database disagree about what is true. Most of the hard parts of replication are about surviving that window.

Why keep more than one copy

Three forces, and they compound. Read scaling: a single node has a ceiling on queries per second; add followers and reads fan out across them. Availability: if the one machine holding your data dies, so does your service — a second copy is the difference between a failover and an outage. Latency: a reader in Singapore hitting a database in Virginia pays ~200ms of round trip they’ll never get back unless there’s a copy near them.

The catch is that the instant you have two copies, you have a distributed-systems problem. A write applied here is not yet applied there. Everything below is a consequence of that gap.

Single-leader: one writer, many readers

The most common topology by far. One node is the leader (primary); it takes all writes. Every other node is a follower (replica, standby) that receives the leader’s change stream and applies it in order. Reads can go to any node; writes must go to the leader.

How long the leader waits for followers is the knob that matters — and that’s where the members-only walkthrough picks up: the synchronous-vs-async tradeoff table, the read-your-own-writes and monotonic-reads anomalies with their code fixes, failover and split-brain fencing, multi-leader and leaderless quorums (R + W > N), how the change log actually ships, and the interview corner.