Distributed · XB Field Notes

etcd elects a leader so writes happen in order

Thu, 09 Mar 2028 00:00:00 +0000

etcd is a replicated log, and Raft is the algorithm that keeps the replicas identical. One member is leader for a term and is the only one allowed to propose entries; an entry is committed once a majority has it. Leader election decides who may write, and log replication decides what is durable — separate mechanisms with separate failure modes.

Both are governed by one number. The quorum sets how many acknowledgements a write costs, how many members you can lose, and what a partition does. Get the member count wrong and you pay more for the same resilience; lose the quorum and the cluster stops writing while still answering reads.

One leader per term

A member becomes a candidate when it has not heard from a leader within its election timeout, randomised so members do not all campaign at once. It increments the term, votes for itself, and asks the others; a member grants at most one vote per term and refuses a lower term. A majority makes it leader. If two members time out together the vote can split — in a four-member cluster the result can be two against two, short of the quorum of three — wasting the round for another election timeout.

During an election the cluster cannot process writes: etcd queues them until a leader exists. Writes sent to the old leader but not yet committed may be lost, because the new leader may overwrite uncommitted entries; committed writes are never lost. The term is the clock that makes this safe, and why an isolated member with a higher term is a hazard.

Quorum arithmetic, and why odd wins

Quorum is a bare majority, floor(N/2) + 1, so the failures a cluster tolerates are N - quorum. Three members tolerate one failure and need two acknowledgements; four tolerate one but need three. Five tolerate two with three acks; six also tolerate two but need four. An even member count buys no resilience and costs an extra acknowledgement per write.

Membership changes move the quorum immediately, which is where even counts get dangerous. The documented example is a one-member cluster: adding a member raises the quorum from one to two, and the previous leader, finding it has no active quorum, triggers an election. Growing three members to four raises the quorum from two to three, so a 2-and-2 partition leaves neither side able to elect anyone — a split the three-member cluster would have survived.

Learners, and the two mechanisms apart

A learner receives the leader’s log but does not vote, which is what you want for a node that just joined: adding one to a three-member cluster leaves the quorum at two, so the join cannot trigger an election. A learner rejects client reads and writes, leadership cannot be transferred to it, and it never promotes itself. Only once its log has caught up can an operator promote it; only then does it count toward the quorum.

Keeping election and replication distinct explains the confusing symptoms. Losing the leader stops appends until a new one is elected, even though every member is healthy. A healthy leader with unreachable followers cannot commit. And a follower can be arbitrarily far behind while the cluster commits happily, because only a majority needs the entry — lag is a replication problem, not an availability one, until the majority itself is gone.

When it breaks

The classic failure is losing the quorum. Writes stop: a cluster whose majority has failed cannot accept more writes, and recovers only once a majority is available again — otherwise disaster recovery from a backup is the only route. The trap is that reads may still work. A linearizable read is a consensus operation and fails like a write, but a serializable read is served from the member’s own state and can return data the cluster has already moved past, so “it still answers” is not evidence of health.

The second failure is the partitioned minority. It cannot elect a leader, so it has no quorum and no writes, and there is no split brain because membership changes need the majority. At the Raft level those isolated members keep timing out and raising their term while the majority stays put; when the partition heals, the higher term forces the leader to step down and a fresh election runs. A healed partition therefore produces a short write outage.

Check it quickly

Read the leader, the term and the per-member log lag:

etcdctl --endpoints=https://10.0.1.11:2379 endpoint status --write-out=table
etcdctl --endpoints=https://10.0.1.11:2379,https://10.0.1.12:2379,https://10.0.1.13:2379 endpoint health --write-out=table
etcdctl --endpoints=https://10.0.1.11:2379 member list --write-out=table
etcdctl --endpoints=https://10.0.1.11:2379 --consistency=s get /registry/ --prefix --keys-only | head

The status table has exactly one true in the LEADER column and a RAFT TERM every member should share; a much higher term marks the member that was partitioned. If endpoint health fails on enough endpoints to break the majority, writes will be refused. A member listed learner=true does not vote and does not count toward the quorum. The last command is the serializable read that still succeeds without a quorum, which is why it is a poor health check.

Sources

etcd failures, leader failure, majority failure, network partitions and what happens to writes, checked 2026-09-16.
Runtime reconfiguration, majority commit and the cost of changing cluster size, checked 2026-09-16.
etcd learner design, the non-voting learner state and promotion, checked 2026-09-16.
etcd API guarantees, linearizable versus serializable reads, checked 2026-09-16.

Leader election needs a majority

Tue, 21 Sep 2027 00:00:00 +0000

Majority quorum is what forbids a split brain. When the cluster is partitioned, the side with fewer than half the nodes cannot form a quorum or stay authoritative.

One leader, many votes

A replicated system that must not diverge elects a single leader for a term. A candidate asks peers for a vote, and to become leader it needs a quorum, more than half of the voting members in a fault-tolerant scheme, to say yes within a term. That majority is what lets the winner act with confidence that no split-off minority can out-vote it, so the group can make decisions with one authority at a time. Exactly half is not enough: quorums are defined as strictly more than half, which is what makes two halves never both be authoritative.

Quorum kills split brain

Partitioning is the failure that leader election exists to survive. If a cluster is split into two halves, at most one half can hold a majority; the minority cannot form a quorum, so it is forced to wait rather than accept writes it could not back. That is the whole point: an old leader on the minority side is safely replaced, and two partitions never both think they own the data and write divergently. The majority rule is what turns ‘who is in charge’ from a guess into an operation with a single, safe answer.

Terms and ticks keep it tidy

Leadership has to be re-proved, not assumed forever. Terms and heartbeat ticks bound how long a leader is trusted: a leader renews its lease by staying visible, and when the ticks lapse because it died or the network cut, followers start a new election and a new term. That constant renewal is what lets the group cope with machines joining and leaving, because no leader’s authority depends on staying alive forever. The result is a loop, elect, renew, detect loss, re-elect, which keeps the group honest without anyone needing to agree on a single absolute clock.

When it breaks

An even-sized cluster learns this rule the hard way. Four nodes split into two pairs need three votes to elect anyone, so neither pair can produce a leader and the cluster goes quiet until the partition heals. Six nodes split three against three deadlock the same way. That is why production clusters usually run three or five voters: any partition leaves at most one side holding a strict majority.

A side without a majority must stop accepting writes, and that stall is the price of safety. The old leader stranded on the minority side steps down once it can no longer renew its quorum, so clients pinned to that side see errors instead of progress. Treating those errors as a bug and forcing a local write reintroduces exactly the split brain the majority rule was bought to prevent.

Elections can also livelock with no partition at all. Candidates starting in the same instant split the vote so nobody reaches a majority, terms churn, leadership flaps, and write latency spikes while every ballot fails. Randomized election timeouts exist to break this symmetry, so identical fixed timeouts or saturated nodes that delay votes reproduce the storm on demand.

The last trap is term regression. A node returning from a long partition campaigns on a stale term, or a candidate with a lagging log asks for votes it has not earned. Voters must reject lower terms and demand an up-to-date log; skipping that freshness check elects a leader that then overwrites entries a previous majority had already committed.

Check it quickly

Save this as majority.py and run python3 majority.py. It uses only the Python standard library and models five voters with one ballot each per term, plus a partition that decides whose votes a candidate can actually collect:

def quorum(n):
 """Strict majority: more than half of all voters, not of one side."""
 return n // 2 + 1


class Cluster:
 """Minimal ballot box: N voters, one vote per node per term."""

 def __init__(self, size):
 self.size = size
 self.terms = [0] * size
 self.voted_for = [None] * size

 def request_votes(self, candidate, term, reachable):
 if term <= self.terms[candidate]:
 return [] # stale term: the candidate itself has seen newer
 votes = []
 for node in reachable:
 if term > self.terms[node]:
 self.terms[node] = term
 self.voted_for[node] = candidate
 votes.append(node)
 elif term == self.terms[node] and self.voted_for[node] in (None, candidate):
 votes.append(node)
 return votes

 def elect(self, candidate, term, reachable):
 votes = self.request_votes(candidate, term, reachable)
 return len(votes) >= quorum(self.size), votes


assert quorum(3) == 2 and quorum(4) == 3 and quorum(5) == 3

majority = Cluster(5)
won, votes = majority.elect(candidate=0, term=1, reachable=[0, 1, 2])
assert won and sorted(votes) == [0, 1, 2], votes
print(f"5 nodes, side of 3: elected with votes {sorted(votes)} (need {quorum(5)})")

minority = Cluster(5)
won, votes = minority.elect(candidate=3, term=1, reachable=[3, 4])
assert not won and sorted(votes) == [3, 4], votes
print(f"5 nodes, side of 2: no leader with votes {sorted(votes)} (need {quorum(5)})")

even = Cluster(4)
for candidate, reachable in ((0, [0, 1]), (2, [2, 3])):
 won, votes = even.elect(candidate, term=1, reachable=reachable)
 assert not won, (candidate, votes)
print(f"4 nodes split 2-2: neither side reaches {quorum(4)}, cluster stalls")

fresh = Cluster(5)
won, _ = fresh.elect(candidate=0, term=2, reachable=[0, 1, 2, 3, 4])
assert won
won, votes = fresh.elect(candidate=4, term=1, reachable=[0, 1, 2, 3, 4])
assert not won and votes == [], votes
print("stale term 1 after term 2: rejected with zero votes")

print("PASS: majority side elects, minority and even split stall, stale term loses")

The script asserts the election semantics directly. The three-node side gathers three of five votes and elects a leader for the term, while the two-node side stalls at two and elects nobody. The even split replays four nodes cut two against two: neither side reaches three, so no leader emerges and the cluster waits. A stale-term ballot is rejected with zero votes even though the arithmetic alone would pass. Expect this output:

5 nodes, side of 3: elected with votes [0, 1, 2] (need 3)
5 nodes, side of 2: no leader with votes [3, 4] (need 3)
4 nodes split 2-2: neither side reaches 3, cluster stalls
stale term 1 after term 2: rejected with zero votes
PASS: majority side elects, minority and even split stall, stale term loses

The assertions passed with Python 3.12 on macOS. The model covers vote counting and term freshness only, not log replication, durable state, or randomized timers; for those rules, read the Raft paper’s election section next.

Sources

In Search of an Understandable Consensus Algorithm (Raft), leader election and majority quorum in section 5, checked 2026-09-09.
etcd: etcd versus other key-value stores, leader election and distributed coordination over Raft, checked 2026-09-09.
ZooKeeper Internals, majority quorums for atomic broadcast and leader election, checked 2026-09-09.

Gossip protocols converge eventually

Mon, 20 Sep 2027 00:00:00 +0000

Peers keep exchanging and forwarding state with random partners. The system ends up consistent and tolerates messages being dropped along the way.

Everyone tells someone

Each round a node picks a random peer and exchanges state, so a single fact spreads by repetition rather than by a single authoritative hop. Because the partner changes each round, information fans out geometrically: each round roughly doubles the nodes that know, which is why a new membership entry can reach a large cluster in a handful of rounds. The exchange is usually symmetrical, both sides learning from each other in one message pair, which keeps the network cheap and symmetric. It trades the guarantee of a single message delivered in order for a high, self-healing probability that the state eventually arrives.

Converges without a center

There is no coordinator to become the single point of failure, because convergence is an emergent property of repeated random contact rather than a schedule run by someone. That makes gossip the right tool for peer sets whose size and membership drift: nodes can join or leave without a directory being updated first. The price is eventual, not immediate: everyone agrees on the same state only after enough rounds, and a round-trip estimate bounds how long convergence takes. As the cluster grows, adding jump/renaming nodes keeps the reach close to logarithmic rather than linear in the cluster size.

A lost message is absorbed

Dropping a message does not stop the protocol, because the same fact will be rediscovered by a later random pair and spread again from the node that kept it. A failure simply slows convergence to the next exchange round rather than ending the flow of state. It is resilient because state is copied forward, not decremented and lost at a single point. The practical consequence is that extra rounds are cheap insurance: keep the round interval small enough to bound convergence, but warm the state so a node joining with nothing can be brought up quickly.

When it breaks

A network partition does not stop gossip; it splits it. Each side keeps exchanging state internally and converges on its own membership view, so both halves believe they are healthy. When the link heals, the merge rule decides what survives: keeping the latest wall-clock timestamp can silently discard writes accepted on the losing side. Production systems therefore pair gossip with version vectors or tombstones so healing reconciles instead of amputating.

Failure detection fails in the opposite direction: nothing is partitioned, but a slow node looks dead. A long garbage-collection pause or a saturated link can delay heartbeats past the suspicion timeout, so the cluster marks a live node failed and re-replicates its ranges, then undoes that work when it answers. SWIM-style suspicion exists for this reason: suspect first, declare death late, and let the accused refute with a higher incarnation number.

The third failure is repair traffic itself. When many nodes rejoin at once, or the round interval is too aggressive, every node pulls full state from everyone else: an anti-entropy storm. The fix is digest-first exchange — compare version digests and send only missing entries — plus jitter and bounded fanout. A restored disk image must never rejoin with a reset counter: a stale copy with a lower version loses correctly, but a restarted counter can overwrite newer state, so versions must be monotonic across restarts.

Check it quickly

Save this as check_gossip.py and run python3 check_gossip.py. It models 20 nodes that each keep a versioned key-value map, seeds one update on node 0, then runs push-pull rounds where every node picks a random peer and both keep the highest version per key. A fixed seed keeps peer choices reproducible. The script prints how many nodes know the update after each round and asserts full convergence within 12 rounds.

import random

NODES = 20
ROUNDS = 12
SEED = 7

random.seed(SEED)
# Each node holds a versioned map: key -> (version, value).
nodes = [{} for _ in range(NODES)]
nodes[0]["config"] = (1, "blue")


def merge(a, b):
 merged = dict(a)
 for key, (version, value) in b.items():
 if key not in merged or version > merged[key][0]:
 merged[key] = (version, value)
 return merged


history = []
for round_no in range(1, ROUNDS + 1):
 for i in range(NODES):
 peer = random.randrange(NODES)
 if peer == i:
 continue
 # Push-pull: both sides adopt the union, highest version wins.
 united = merge(nodes[i], nodes[peer])
 nodes[i] = dict(united)
 nodes[peer] = dict(united)
 informed = sum(1 for state in nodes if state.get("config") == (1, "blue"))
 history.append(informed)
 print(f"round {round_no}: {informed}/{NODES} nodes know the update")

assert history[-1] == NODES, f"gossip did not converge: {history}"
assert all(state.get("config") == (1, "blue") for state in nodes)
print(f"PASS: all {NODES} nodes converged within {ROUNDS} rounds: {history}")

Expect geometric spread: about a quarter knows after round one, nearly all by round three, and all 20 by round four. Pinning every node to the same peer stalls convergence, mirroring a partitioned cluster. The assertions passed with CPython 3.12 on macOS; the model omits suspicion timeouts, tombstone expiry, and bandwidth limits, so treat it as a sketch, not a membership implementation.

Sources

Dynamo: Amazon’s highly available key-value store, gossip-based membership and anti-entropy with version reconciliation, checked 2026-09-12.
SWIM: Scalable Weakly-consistent Infection-style process group Membership protocol, suspicion mechanism and incarnation numbers for failure detection, checked 2026-09-12.
Cassandra internode communications (gossip), peer-to-peer state exchange with versioned gossip messages, checked 2026-09-12.

Consistent hashing shrinks the rebalance

Sun, 19 Sep 2027 00:00:00 +0000

Consistent hashing limits placement changes when membership changes; it does not itself move data, make replicas agree, or distribute a hot key’s requests. In the classic clockwise ring, a key belongs to the first node point at or after its hash position, wrapping to the smallest point when necessary.

Get the ownership interval right

A point owns the interval after its predecessor through itself: (predecessor, point]. With points A at 20, B at 50, and C at 80, B owns positions above 20 through 50. A key beyond 80 wraps to A. This is not the arc from B to its successor.

Adding D at 35 takes positions above 20 through 35 from B, the new point’s clockwise successor. Removing B transfers B’s former interval to C. Unaffected points retain their positions and ownership. That preservation is the useful property; saying “only a small arc moves” is insufficient when an existing arc might already cover a large fraction of the ring.

By contrast, directly choosing node index hash(key) modulo node_count changes the divisor when a node is added. Many keys then choose another index. A balanced ring with four equal-capacity nodes expanded to five would ideally move about one fifth of keys, but exact movement depends on the actual points and key distribution, not a universal percentage.

Virtual nodes spread intervals, not individual keys

Assigning many ring points to each physical node usually reduces imbalance from a few randomly placed points. Each physical node owns several smaller intervals, and adding another can take pieces from multiple neighbors. Weights can allocate different point counts for unequal capacity. More points also cost routing-table memory and maintenance work.

One key still resolves to one physical owner in this primary-placement model. If that key receives most requests, increasing virtual-node count does not split those requests across machines. Even equal key counts do not imply equal bytes, request rates, or processing cost. Replication, read distribution, and hot-key handling require additional policies.

Stability alone also does not imply balance. All participants must agree on membership, persistent node identities, key encoding, hash rules, and point counts. A stable digest helps them reproduce positions, but cannot compensate for different membership views or a skewed workload. Rebuilding every node’s points on each membership change would defeat the placement-preservation property.

Placement is not migration or consistency

A new routing answer does not transfer cached or persistent values to the new owner. A cache may experience misses and an origin-load spike; a persistent store needs a migration and ownership protocol. Clients using old and new rings can disagree during rollout. Recovery and synchronization behavior belongs to the surrounding system.

Real products also choose different algorithms and metadata. Redis Cluster uses fixed hash slots whose ownership is reassigned; it is not this example’s direct node-point ring. Cassandra adds replica placement and token-management rules, while Envoy uses hashing to select upstream hosts. Do not infer one product’s failure guarantees from another product’s placement diagram.

When it breaks

A single point per node can be badly unbalanced, and renaming a node can unexpectedly relocate its points. Changing byte encoding or hash configuration can likewise remap keys without any physical machine joining. Record the mapping inputs as part of deployment configuration and compare ownership before rollout.

Measure moved keys separately from moved bytes and real traffic. Virtual nodes improve statistical placement, not a guaranteed maximum load. During membership churn, budget for migration work or cold-cache demand, and verify how clients learn the new view. A placement experiment is not an availability or data-integrity test.

Check it quickly

Save the following as ring_check.py and run python3 ring_check.py with Python 3.10 or newer. This ran with Python 3.14.4. It uses only the standard library, starts no servers, and performs no network or filesystem writes. SHA-256 and explicit UTF-8 inputs make the generated fixture reproducible.

from bisect import bisect_left
from collections import Counter
from hashlib import sha256


def token(value):
 return int.from_bytes(sha256(value.encode("utf-8")).digest(), "big")


def ring(nodes, copies):
 entries = sorted(
 (token(f"node:{node}:{copy}"), node)
 for node in nodes for copy in range(copies)
 )
 positions, owners = zip(*entries)
 assert len(set(positions)) == len(positions), "fixture token collision"
 return positions, owners


def lookup(layout, position):
 positions, owners = layout
 return owners[bisect_left(positions, position) % len(positions)]


simple = ((20, 50, 80), ("A", "B", "C"))
joined = ((20, 35, 50, 80), ("A", "D", "B", "C"))
removed = ((20, 80), ("A", "C"))
assert [lookup(simple, value) for value in (0, 20, 21, 50, 51, 80, 81)] == [
 "A", "A", "B", "B", "C", "C", "A",
]
for value in range(100):
 assert lookup(joined, value) == ("D" if 20 < value <= 35 else lookup(simple, value))
 assert lookup(removed, value) == ("C" if 20 < value <= 50 else lookup(simple, value))
assert lookup(joined, 35) == "D"
assert lookup(removed, 50) == "C"
print("a point owns (predecessor, point]; lookups wrap clockwise")

nodes = ("A", "B", "C", "D")
positions = [token(f"key:{index}") for index in range(20000)]
base = ring(nodes, 256)
expanded = ring(nodes + ("E",), 256)
contracted = ring(tuple(node for node in nodes if node != "B"), 256)
assert base == ring(tuple(reversed(nodes)), 256)
before = [lookup(base, position) for position in positions]
after_add = [lookup(expanded, position) for position in positions]
after_remove = [lookup(contracted, position) for position in positions]
add_count = sum(old != new for old, new in zip(before, after_add))
remove_count = sum(old != new for old, new in zip(before, after_remove))
assert 0 < add_count < len(positions)
assert add_count == after_add.count("E")
assert all(old == new or new == "E" for old, new in zip(before, after_add))
assert remove_count == before.count("B")
assert all(old == new or old == "B" for old, new in zip(before, after_remove))
modulo_count = sum(
 nodes[position % 4] != (nodes + ("E",))[position % 5]
 for position in positions
)
print(f"add E: {add_count}/20000 keys move ({add_count / 200:.2f}%)")
print(f"remove B: {remove_count}/20000 keys move ({remove_count / 200:.2f}%)")
print(f"modulo 4 -> 5: {modulo_count}/20000 keys move ({modulo_count / 200:.2f}%)")

single = ring(nodes, 1)
for label, layout in (("one point", single), ("256 points", base)):
 counts = Counter(lookup(layout, position) for position in positions)
 print(label, {node: counts[node] for node in nodes})

hot_owner = lookup(base, token("key:hot"))
synthetic_requests = Counter(before)
synthetic_requests[hot_owner] += 100000
assert synthetic_requests[hot_owner] == before.count(hot_owner) + 100000
print("100000 extra requests for one hot key all target", hot_owner)
print("synthetic request counts", dict(sorted(synthetic_requests.items())))

The integer example checks both interval endpoints and wraparound across every position from zero through 99. For 20,000 deterministic keys with 256 points per node, adding E moves 4,023 keys, removing B moves 4,895, and modulo resizing moves 16,013. Assertions also require every addition-related move to end at E and every removal-related move to originate at B.

The printed counts show this fixture’s improved balance with 256 points, not a universal bound. Assigning 100,000 extra synthetic requests to one hot key still directs them all to D. These are calculated placement and demand counts, not measured production requests, transferred bytes, or recovery timings.

Sources

Apache Cassandra’s Dynamo architecture, clockwise ownership, token ranges, and virtual nodes, checked 2026-09-08.
Envoy load-balancing algorithms, ring hashing, stable host identity, weights, and distribution, checked 2026-09-08.
Python hashlib, portable SHA-256 digest construction, checked 2026-09-08.
Redis Cluster specification, fixed slot mapping as a distinct placement design, checked 2026-09-08.

Vector clocks order concurrent writes

Sat, 18 Sep 2027 00:00:00 +0000

Each node keeps a counter per participant. Comparing vectors reveals a causal edge or a true concurrent write, which decides whether to merge or to ask.

One timestamp is not enough

A single timestamp cannot order concurrent events, because a clock is not a cause, so two nodes that each wrote something at the same label may still disagree about which came first. Vector clocks solve that by giving every participant its own counter: a process bumps its own slot whenever it does something, and merges the sender’s vector into its own whenever it receives a message. The result is a vector that grows as events happen, and comparing two vectors tells you whether one event provably preceded the other, or whether neither did.

Compare vectors to see causality

To compare vectors, look element by element: if every value in one vector is less than or equal to the matching value in the other, the first event happened before the second; otherwise the two events are concurrent. Because a causal order accumulates everywhere the two paths crossed, a vector captures the transitive and direct edges that a single scalar time forfeits. The moment a message passes from A to B, B’s vector reflects both B’s own progress and everything A had seen, which is how the model traces causality instead of guessing it.

Concurrent means someone decides

Concurrency in the vector sense does not produce a winner; it only says two events are unrelated or tied, so someone else must decide the order when the application needs one. That decision is usually made by comparing a key or by ordering on the vector values themselves, and it must be deterministic so every replica agrees. The design is honest: vector clocks do not fabricate a total order among unrelated writes, they only expose that no such order exists; the final ordering decision belongs to the rule above them.

When it breaks

Vector clocks fail first by growing. One entry exists per participant, so a key written by hundreds of short-lived clients carries hundreds of counters on every read, and every write ships them back. Dynamo reported exactly this pressure: reconciliation metadata grows with the writer set, and Riak operators see the same symptom as oversized causal context that slows reads and multiplies siblings. Evicting the oldest entries feels like relief, but it deletes causality.

Once node C is pruned from [A:3, B:2, C:5], later comparisons read the missing slot as zero, so an event C had already observed looks brand new, and a write that should have stayed subsumed resurrects as a false sibling.

The second failure misreads concurrent vectors as ordered. Comparing only the sum, the maximum, or a wall-clock timestamp beside the vector elects a winner between [A:2, B:1] and [A:1, B:2], yet neither dominates the other element by element, so the honest answer is concurrent, not before or after. The discarded sibling is a lost update the clock had flagged correctly. The mirror mistake treats one equal slot as equal history: [A:2] seen twice is the same event replayed, but [A:2] against [A:2, B:1] is a strict before relation, and overwriting the longer vector with the shorter one rewinds history.

The guard is strict: return causal context on every write, merge with element-wise maximum, increment only your own slot, and prune only with a dominance-preserving scheme such as dotted version vectors. A write without its read vector forks a sibling every time.

Check it quickly

Save this as check_vector_clocks.py and run python3 check_vector_clocks.py. It stores vectors as plain dicts with missing entries read as zero, then asserts all three outcomes: happens-before with equality in both directions, plus a concurrent pair whose merge dominates both parents.

def compare(a, b):
 """Return 'equal', 'before', 'after', or 'concurrent'."""
 nodes = set(a) | set(b)
 le = all(a.get(n, 0) <= b.get(n, 0) for n in nodes)
 ge = all(a.get(n, 0) >= b.get(n, 0) for n in nodes)
 if le and ge:
 return "equal"
 if le:
 return "before" # a happens-before b
 if ge:
 return "after" # b happens-before a
 return "concurrent"


def merge(a, b):
 return {n: max(a.get(n, 0), b.get(n, 0)) for n in set(a) | set(b)}


def tick(clock, node):
 nxt = dict(clock)
 nxt[node] = nxt.get(node, 0) + 1
 return nxt


# 1. Happens-before: b extends a with one new event from B.
a = {"A": 1}
b = {"A": 1, "B": 1}
assert compare(a, b) == "before"
assert compare(b, a) == "after"

# 2. Equal: the same causal history seen twice.
assert compare({"A": 2}, {"A": 2}) == "equal"

# 3. Concurrent: each side saw something the other missed.
c = {"A": 2, "B": 1}
d = {"A": 1, "B": 2}
assert compare(c, d) == "concurrent"

# Merge of concurrent histories dominates both parents.
m = merge(c, d)
assert m == {"A": 2, "B": 2}
assert compare(c, m) == "before"
assert compare(d, m) == "before"

# A truncated vector lies: dropping B makes m look merely later.
truncated = {"A": 2}
assert compare(truncated, m) == "before" # missing B reads as 0, hiding what was pruned
print("PASS: before/after/equal/concurrent and merge all behave as expected")

Expect before and after on the extended vector, equal on identical histories, and concurrent on the split pair. The merge reads [A:2, B:2] and follows each parent, while truncated [A:2] still reports before, showing how pruning hides causality. The assertions passed with CPython 3.12 on macOS; the sketch omits pruning, dotted versions, and persistence, so treat it as a comparison demo, not a store.

Sources

Dynamo: Amazon’s highly available key-value store, reconciliation with version vectors and sibling retention, checked 2026-09-12.
Riak KV 2.2.3: Causal context (vector clocks and dotted version vectors), vector-clock growth, pruning hazards, and dotted version vectors, checked 2026-09-12.

Idempotency keys scope one logical action

Mon, 12 Oct 2026 00:00:00 +0000

An idempotency key identifies one logical action only within a service’s defined scope and retention window. Reusing the same text is not a universal exactly-once guarantee. The service must associate that identity with the right principal and operation, coordinate competing attempts, and retain enough authoritative information to recognize the action later.

Scope is part of the identity

A key looked up globally can accidentally merge unrelated tenants or expose another caller’s stored result. Authenticate and authorize before consulting or replaying the record. Build the lookup identity from verified scope, a stable operation identifier, and the client’s action key; preserve component boundaries instead of ambiguous string concatenation.

Which dimensions belong in scope is an API decision. EC2 explicitly distinguishes regional and zonal idempotency. The same client token can permit separate actions in different regions or availability zones under those documented rules. Do not assume that moving a retry to another region preserves a global deduplication promise.

Request meaning matters too. The same scoped key with different parameters must follow a declared conflict policy. The example fingerprints one positive integer quantity; it does not compare arbitrary HTTP bodies, perform authentication, or infer business equivalence from a hash.

Couple the effect to its receipt

A lookup followed by an independent write can race: two workers both see no record and both execute. A unique key on the receipt table is insufficient if both business effects have already occurred before one receipt insert fails. The effect and remembered result need an appropriate atomic boundary.

When both live in one transactional database, they can be committed together. This fixture uses SQLite BEGIN IMMEDIATE before lookup and writes its business row and receipt in that transaction. SQLite permits one simultaneous writer, so competing connections serialize; the composite primary key also expresses the lookup identity.

The transaction’s protection ends at that database. An external payment, email, or message publication does not roll back because a local transaction fails. Such effects need their own end-to-end delivery and deduplication contract. Calling the whole workflow exactly once does not supply one.

Retention is a correctness boundary

If a remembered key expires while an old request can still return, that request may execute again. Stripe documents that a key reused after its record is pruned creates a new request; its minimum twenty-four-hour retention statement is provider-specific, not a universal timer.

Keep recovery windows and retention aligned, and distinguish a temporary deduplication receipt from a lasting business uniqueness rule. Expiring a receipt does not undo the original order. If the business forbids that order from ever being recreated, it needs an authoritative invariant that outlives a short retry cache.

Restoring an older receipt database, clearing state, or switching to an independent store can also forget completed work. Preserve the relationship between business data and operation records when planning recovery, not merely their individual file availability.

When it breaks

A lost response after commit leaves the client uncertain but does not remove the recorded result. A failed transaction before commit should leave neither a business effect nor a successful receipt. Those are different cases and require different evidence.

Do not treat database contention, unavailable storage, or unknown outcomes as an empty lookup. Nor should an in-progress attempt automatically permit another effect. Define whether duplicates wait, receive an in-progress response, or use another documented recovery path; successful replay is only one possible state.

Check it quickly

Save the block as scope_check.py and run python3 scope_check.py with Python 3.10 or newer. It ran with Python 3.14.4 and SQLite 3.53.0. The standard-library fixture creates a database in a private temporary directory and removes it afterward. Each attempt opens its own connection; tenant and operation inputs represent already verified fixture identities.

from concurrent.futures import ThreadPoolExecutor
from contextlib import closing
import json
from pathlib import Path
import sqlite3
from tempfile import TemporaryDirectory
from threading import Barrier


with TemporaryDirectory(prefix="idempotency-scope-") as directory:
 database = Path(directory) / "orders.sqlite"
 with closing(sqlite3.connect(database)) as db:
 db.executescript("""
 CREATE TABLE effects (
 id INTEGER PRIMARY KEY, tenant TEXT NOT NULL,
 operation TEXT NOT NULL, quantity INTEGER NOT NULL CHECK (quantity > 0)
 );
 CREATE TABLE receipts (
 tenant TEXT NOT NULL, operation TEXT NOT NULL, request_key TEXT NOT NULL,
 fingerprint TEXT NOT NULL, result TEXT NOT NULL, expires_at INTEGER NOT NULL,
 PRIMARY KEY (tenant, operation, request_key)
 );
 """)

 def execute(tenant, operation, key, quantity, now, fail_before_receipt=False):
 # Tenant/operation are already authenticated/routed fixture identifiers.
 assert type(quantity) is int and quantity > 0
 fingerprint = json.dumps({"quantity": quantity}, separators=(",", ":"))
 scope = (tenant, operation, key)
 with closing(sqlite3.connect(database, timeout=5)) as db:
 with db:
 db.execute("BEGIN IMMEDIATE")
 previous = db.execute(
 "SELECT fingerprint, result, expires_at FROM receipts "
 "WHERE tenant=? AND operation=? AND request_key=?", scope,
 ).fetchone()
 if previous is not None and previous[2] > now:
 if previous[0] != fingerprint:
 return "conflict", None
 return "replay", json.loads(previous[1])
 if previous is not None:
 db.execute("DELETE FROM receipts WHERE tenant=? AND operation=? AND request_key=?", scope)
 cursor = db.execute(
 "INSERT INTO effects (tenant, operation, quantity) VALUES (?, ?, ?)",
 (tenant, operation, quantity),
 )
 result = {"effect_id": cursor.lastrowid, "quantity": quantity}
 if fail_before_receipt:
 raise RuntimeError("injected failure after effect insertion, before receipt insertion")
 db.execute("INSERT INTO receipts VALUES (?, ?, ?, ?, ?, ?)",
 (*scope, fingerprint, json.dumps(result), now + 60))
 return "created", result

 def counts():
 with closing(sqlite3.connect(database)) as db:
 return db.execute("SELECT (SELECT count(*) FROM effects), (SELECT count(*) FROM receipts)").fetchone()

 gate = Barrier(8)

 def attempt(_):
 gate.wait(timeout=5)
 return execute("north", "web-order", "intent-a", 2, 0)

 with ThreadPoolExecutor(max_workers=8) as pool:
 outcomes = list(pool.map(attempt, range(8)))
 assert [kind for kind, _ in outcomes].count("created") == 1
 assert [kind for kind, _ in outcomes].count("replay") == 7
 original = outcomes[0][1]
 assert all(result == original for _, result in outcomes) and counts() == (1, 1)
 print("eight independent connections produce one effect, one receipt, and seven replays")

 assert execute("north", "web-order", "intent-a", 3, 1) == ("conflict", None)
 assert counts() == (1, 1)
 assert execute("south", "web-order", "intent-a", 2, 1)[0] == "created"
 assert execute("north", "batch-order", "intent-a", 2, 1)[0] == "created"
 assert execute("north", "web-order", "intent-b", 2, 1)[0] == "created"
 assert counts() == (4, 4)
 print("changed parameters conflict; a different tenant, operation, or intent key has separate scope")

 try:
 execute("north", "web-order", "rolled-back", 1, 2, fail_before_receipt=True)
 except RuntimeError as error:
 assert str(error).startswith("injected failure")
 else:
 raise AssertionError("expected the transaction to fail")
 assert counts() == (4, 4)
 assert execute("north", "web-order", "rolled-back", 1, 2)[0] == "created"
 assert counts() == (5, 5)
 print("a raised exception rolls back the effect and receipt together; retry then creates one effect")

 _, committed = execute("north", "web-order", "committed-no-reply", 1, 3)
 # Discard the reply, then query through a fresh connection; no network loss is simulated here.
 assert execute("north", "web-order", "committed-no-reply", 1, 3) == ("replay", committed)
 assert counts() == (6, 6)
 print("a fresh connection replays a committed result without adding another effect")

 assert execute("north", "web-order", "intent-a", 2, 59) == ("replay", original)
 kind, after_expiry = execute("north", "web-order", "intent-a", 2, 60)
 assert kind == "created" and after_expiry["effect_id"] != original["effect_id"]
 assert counts() == (7, 6)
 print("at the explicit 60-second retention boundary, the same key creates a new effect")
 print("SQLite", sqlite3.sqlite_version, "verified counts: 7 effects, 6 retained receipts")

Eight competing connections produce one effect, one receipt, and seven replays with identical results. Changed quantity conflicts. Changing tenant, operation, or action key creates separate work under the declared scope, reaching four effects and four receipts.

An exception after inserting an effect but before its receipt rolls back both. Retrying then creates one effect. A committed result is available through a fresh connection without adding work; this discards an application reply but does not simulate network loss.

Injected times verify replay at second fifty-nine and a new effect at the exact sixty-second expiration boundary. Final counts are seven effects and six retained receipts: the expired record disappears, not its original business row.

The sixty-second window is an explicit test policy, not a recommendation. This experiment does not test process crashes, power loss, distributed databases, external side effects, authentication, or production contention. BEGIN IMMEDIATE can still fail with database-busy errors outside the small successful workload.

Sources

EC2 API idempotency, regional and zonal scope plus parameter mismatch behavior, checked 2026-09-08.
Stripe: Idempotent requests, retention, pruning, parameter checks, and concurrent-request behavior, checked 2026-09-08.
SQLite transactions, one simultaneous writer, BEGIN IMMEDIATE, and SQLITE_BUSY, checked 2026-09-08.
SQLite atomic commit, the atomic boundary around database changes and its storage assumptions, checked 2026-09-08.
Python sqlite3, connection context-manager commit/rollback and explicit connection closing, checked 2026-09-08.

Distributed · XB Field Notes

etcd elects a leader so writes happen in order

One leader per term

Quorum arithmetic, and why odd wins

Learners, and the two mechanisms apart

When it breaks

Check it quickly

Sources

Leader election needs a majority

One leader, many votes

Quorum kills split brain

Terms and ticks keep it tidy

When it breaks

Check it quickly

Further reading

Sources

Gossip protocols converge eventually

Everyone tells someone

Converges without a center

A lost message is absorbed

When it breaks

Check it quickly

Further reading

Sources

Consistent hashing shrinks the rebalance

Get the ownership interval right

Virtual nodes spread intervals, not individual keys

Placement is not migration or consistency

When it breaks

Check it quickly

Further reading

Sources

Vector clocks order concurrent writes

One timestamp is not enough

Compare vectors to see causality

Concurrent means someone decides

When it breaks

Check it quickly

Further reading

Sources

Idempotency keys scope one logical action

Scope is part of the identity

Couple the effect to its receipt

Retention is a correctness boundary

When it breaks

Check it quickly

Further reading

Sources