etcd is a replicated log, and Raft is the algorithm that keeps the replicas identical. One member is leader for a term and is the only one allowed to propose entries; an entry is committed once a majority has it. Leader election decides who may write, and log replication decides what is durable — separate mechanisms with separate failure modes.

Both are governed by one number. The quorum sets how many acknowledgements a write costs, how many members you can lose, and what a partition does. Get the member count wrong and you pay more for the same resilience; lose the quorum and the cluster stops writing while still answering reads.

One leader per term

A member becomes a candidate when it has not heard from a leader within its election timeout, randomised so members do not all campaign at once. It increments the term, votes for itself, and asks the others; a member grants at most one vote per term and refuses a lower term. A majority makes it leader. If two members time out together the vote can split — in a four-member cluster the result can be two against two, short of the quorum of three — wasting the round for another election timeout.

During an election the cluster cannot process writes: etcd queues them until a leader exists. Writes sent to the old leader but not yet committed may be lost, because the new leader may overwrite uncommitted entries; committed writes are never lost. The term is the clock that makes this safe, and why an isolated member with a higher term is a hazard.

Quorum arithmetic, and why odd wins

Quorum is a bare majority, floor(N/2) + 1, so the failures a cluster tolerates are N - quorum. Three members tolerate one failure and need two acknowledgements; four tolerate one but need three. Five tolerate two with three acks; six also tolerate two but need four. An even member count buys no resilience and costs an extra acknowledgement per write.

Membership changes move the quorum immediately, which is where even counts get dangerous. The documented example is a one-member cluster: adding a member raises the quorum from one to two, and the previous leader, finding it has no active quorum, triggers an election. Growing three members to four raises the quorum from two to three, so a 2-and-2 partition leaves neither side able to elect anyone — a split the three-member cluster would have survived.

Learners, and the two mechanisms apart

A learner receives the leader’s log but does not vote, which is what you want for a node that just joined: adding one to a three-member cluster leaves the quorum at two, so the join cannot trigger an election. A learner rejects client reads and writes, leadership cannot be transferred to it, and it never promotes itself. Only once its log has caught up can an operator promote it; only then does it count toward the quorum.

Keeping election and replication distinct explains the confusing symptoms. Losing the leader stops appends until a new one is elected, even though every member is healthy. A healthy leader with unreachable followers cannot commit. And a follower can be arbitrarily far behind while the cluster commits happily, because only a majority needs the entry — lag is a replication problem, not an availability one, until the majority itself is gone.

When it breaks

The classic failure is losing the quorum. Writes stop: a cluster whose majority has failed cannot accept more writes, and recovers only once a majority is available again — otherwise disaster recovery from a backup is the only route. The trap is that reads may still work. A linearizable read is a consensus operation and fails like a write, but a serializable read is served from the member’s own state and can return data the cluster has already moved past, so “it still answers” is not evidence of health.

The second failure is the partitioned minority. It cannot elect a leader, so it has no quorum and no writes, and there is no split brain because membership changes need the majority. At the Raft level those isolated members keep timing out and raising their term while the majority stays put; when the partition heals, the higher term forces the leader to step down and a fresh election runs. A healed partition therefore produces a short write outage.

Check it quickly

Read the leader, the term and the per-member log lag:

etcdctl --endpoints=https://10.0.1.11:2379 endpoint status --write-out=table
etcdctl --endpoints=https://10.0.1.11:2379,https://10.0.1.12:2379,https://10.0.1.13:2379 endpoint health --write-out=table
etcdctl --endpoints=https://10.0.1.11:2379 member list --write-out=table
etcdctl --endpoints=https://10.0.1.11:2379 --consistency=s get /registry/ --prefix --keys-only | head

The status table has exactly one true in the LEADER column and a RAFT TERM every member should share; a much higher term marks the member that was partitioned. If endpoint health fails on enough endpoints to break the majority, writes will be refused. A member listed learner=true does not vote and does not count toward the quorum. The last command is the serializable read that still succeeds without a quorum, which is why it is a poor health check.

Sources

  • etcd failures, leader failure, majority failure, network partitions and what happens to writes, checked 2026-09-16.
  • Runtime reconfiguration, majority commit and the cost of changing cluster size, checked 2026-09-16.
  • etcd learner design, the non-voting learner state and promotion, checked 2026-09-16.
  • etcd API guarantees, linearizable versus serializable reads, checked 2026-09-16.