Raft, Driven: Watch a Cluster Elect, Replicate, and Heal

Most explanations of Raft stop at the election: a node times out, asks for votes, wins a majority, done. That’s the famous part — and it’s the easy part. The part that actually keeps your data correct is what happens to the log afterward: how a follower that was asleep, slow, or partitioned during three writes gets caught up without anyone reading stale data, and the single non-obvious rule that guarantees a committed entry survives every future election.

So before any prose, take the controls. This is a real 5-node Raft cluster — elect a leader, write to it, replicate, then break it: crash the leader mid-flight, or split the network 3-against-2 and watch which side keeps working.

Three things to try

• Elect → write → replicate. Notice a write is uncommitted (dim bar) until a majority stores it, then it commits (solid underline). Commit is a majority ack, not a unanimous one.
• Crash the leader after a write. The cluster stalls until you trigger a new election — and the new leader is always one that already has the committed entry. That’s not luck; it’s the vote rule.
• Split the network 3 | 2. The 3-side keeps committing. The 2-side can start elections forever and never win one. A minority that can’t act is Raft working correctly — the alternative is split-brain.

Why the log is the whole story

The election picks who writes. Everything that makes Raft safe — that a committed entry is never lost, that two nodes never disagree about entry number five — lives in how the log is replicated, checked, and committed. The election is a few hundred milliseconds of drama; the log is forever.

The members-only deep-dive is the mechanism, end to end: the Log Matching Property and the AppendEntries consistency check that enforces it; how a leader repairs a divergent follower by backing up and overwriting its tail; the deceptively subtle commit rule (why a leader may not commit an entry from a previous term by counting replicas — the Raft “Figure 8” hazard) with the exact guard that fixes it; the Leader Completeness and State Machine Safety properties that tie it together; cluster membership changes without a moment of two-leader risk; how etcd, CockroachDB, TiKV, and MongoDB run this in production (including how they serve linearizable reads without a disk write); and an interview challenge with a worked answer.