Consensus & Leader Election: How a Cluster Agrees on One Truth

Two nodes in the same cluster both believe they are the leader. Node A takes a write — deducts ₹500 from an account. Node B, on the far side of a network partition, takes a different write against the same account. Neither is lying; each holds a lease it thinks is valid. When the partition heals, you don’t have a conflict you can merge — you have two divergent histories of one bank account, and no principled way to say which one happened.

This is split-brain, and it is the failure that consensus exists to prevent. The moment two nodes can both act as the authority, correctness is gone — not with an error, but silently, in the data. The entire discipline of consensus is one question asked over and over: how do N unreliable machines agree on exactly one value, even while some of them crash and the network delays or drops messages?

Why so many problems are secretly this problem

Consensus feels abstract until you notice how many concrete questions reduce to it. Who is the leader? is consensus on a node id. What is the next entry in the replicated log? is consensus on a value at a slot. Is this distributed lock currently held, and by whom? is consensus on ownership. Has this config change been committed everywhere? is consensus on a version. Solve “agree on one value despite crashes” once, and you’ve solved leader election, replicated state machines, distributed locks, and config propagation with the same core.

For a decade the canonical answer was Paxos (Lamport, 1998) — provably correct and famously hard to hold in your head. In 2014 Diego Ongaro and John Ousterhout published Raft, whose explicit design goal was understandability: same guarantees, decomposed into pieces a human can reason about. Raft is what we’ll use, because you can actually keep it in your head.

From here the members-only walkthrough builds the whole machine: the three roles and terms-as-a-logical-clock, why randomized election timeouts break symmetry (with the candidate and vote-rule code), the set-overlap argument for why a majority is the only safe quorum, log replication and the safety rule, FLP and why Raft trades liveness for safety, fencing tokens that stop a returning zombie leader from corrupting an external store, the etcd/ZooKeeper/Consul systems in the wild, the Raft-vs-Paxos tradeoffs, a fault-tolerant distributed-lock design challenge, and the interview quiz.