A distributed lock looks simple, but when a client stalls and wakes late its claim to that lock becomes stale; the fencing token turns that stale claim into a verifiable number.

A lock alone is not enough

A distributed lock protects a resource from two writers, but it does nothing about a writer that already lost the lock. Network latency can freeze one node mid-write, the lease expires, another leader takes over, and the old leader resumes from the stale frame it still remembers as its permission. This is the classic condition a lock cannot see: the writer no longer holds the token but still believes it does. The result is corruption that looks like two legitimate owners wrote different data, because from the resource’s view both held valid credentials at different instants. Depending only on a hopeful lease expiry leaves a window where the corruption is baked in before anyone notices.

Issue a number before you act

A fencing token closes the hole with an increasing number. The current leader obtains the token, a monotonically increasing sequence issued by the coordination service, before it writes the shared resource, and every write carries that number. Any request with an older number is refused, because the resource can see that a newer owner is already writing. The token must come from a trusted source, not from the writer’s own clock, because a wall-clock timestamp is forgeable by exactly the process these fences are meant to constrain. The only thing the resource can safely compare is a number it trusts the issuer produced.

Reject anything older

The rejection rule is the point, and it has to be unconditional. A resource that checks token > last_seen rejects the stale writer not because it is malicious but because its number is older, which is the only proof of freshness the resource can rely on. The design gets simpler when issued tokens share one monotonically increasing sequence across all owners, so a single comparison rejects every superseded leader at once. The same principle extends past files: lease fences, epoch numbers and fencing tokens in distributed systems all exist to trade a fragile trust in clocks for a comparison of numbers.

When it breaks

A fence fails when its number can move backward. If the issuer keeps the counter in memory, a crash restarts the sequence at zero: the paused leader still holds 48 while the replacement receives 2, and the resource mistakes the stale write for the newest one. Persist the counter or derive it from a log index, and refuse to issue after an unclean restart until ownership is reconciled.

Timestamp tokens break the same promise. An NTP step backward, a container migration across skewed hosts, or two leaders stamping the same millisecond can produce equal or decreasing values. The resource then has no ordering to enforce, so equal tokens must be rejected as replays and wall-clock time must never substitute for an issuer sequence.

Split brain removes the single comparison point. Each partition elects a leader and mints its own token 7, each replica accepts its local write, and the merged histories both claim to be fenced. Without one issuer and one checking resource, old and new are just labels.

The quietest break is a lost token. A client upgrade drops the field, a proxy strips the header, or a compatibility shim treats a missing value as zero and skips the check, letting a stale write through. The mirror is a resource that forgets last_seen on restart and re-accepts 33 after serving 34. Send the token on every mutation, fail closed when it is absent, and persist the high-water mark with the data.

Check it quickly

Save this as fencing.py and run python3 fencing.py. It uses only the Python standard library and simulates one issuer plus one checking resource, so no server or clock configuration is needed:

class StaleFenceError(RuntimeError):
    pass


class FencingResource:
    """A resource that only accepts strictly increasing fence tokens."""

    def __init__(self):
        self.last_seen = 0
        self.data = None

    def write(self, token, value):
        if token is None:
            raise StaleFenceError("missing fence token")
        if token <= self.last_seen:
            raise StaleFenceError(f"stale token {token} <= {self.last_seen}")
        self.last_seen = token
        self.data = value
        return True


class Issuer:
    """A monotonic fence issuer; in production this counter must be durable."""

    def __init__(self):
        self.counter = 0

    def acquire(self):
        self.counter += 1
        return self.counter


issuer = Issuer()
resource = FencingResource()

old_leader = issuer.acquire()  # 1: lease holder before the pause
new_leader = issuer.acquire()  # 2: replacement after the lease expires
assert (old_leader, new_leader) == (1, 2)

assert resource.write(new_leader, "from-new-leader") is True
assert resource.data == "from-new-leader"

try:
    resource.write(old_leader, "from-stale-leader")
except StaleFenceError as error:
    print(f"rejected stale write: {error}")
else:
    raise AssertionError("stale token was accepted")

try:
    resource.write(new_leader, "replay")
except StaleFenceError as error:
    print(f"rejected replay: {error}")
else:
    raise AssertionError("equal token was accepted")

try:
    resource.write(None, "no-token")
except StaleFenceError as error:
    print(f"rejected missing token: {error}")
else:
    raise AssertionError("missing token was accepted")

next_token = issuer.acquire()  # 3
assert resource.write(next_token, "from-new-leader-again") is True
assert (resource.data, resource.last_seen) == ("from-new-leader-again", 3)
print("PASS: stale, replayed, and missing tokens rejected; newer token accepted")

The script replays the classic pause: the old leader keeps token 1 while the replacement advances to token 2 and writes first. The stale write, an exact replay of token 2, and a missing token are each rejected, while token 3 succeeds. Expect this output:

rejected stale write: stale token 1 <= 2
rejected replay: stale token 2 <= 2
rejected missing token: missing fence token
PASS: stale, replayed, and missing tokens rejected; newer token accepted

Further reading

Sources