A standby node only helps if you know how to switch to it and that the switch works. Failover is a decision, a procedure, and a rehearsal. Delegating it to a switch you have never thrown is how a graceful handoff becomes an outage.

Design the decision in advance

Decide, while everything is healthy, who owns the switch, what data loss is acceptable, and whether failover is automatic or requires a person to pull the trigger. Write the runbook while calm: the exact command or control-plane step, the check that confirms the replica has caught up, and the shout-outs to raise. Auto-failover wins on speed but needs the split-brain case thought through, because two writers each believing they are the leader corrupt more state than one degraded primary ever will. The goal is that the human in the middle does not have to invent policy during the outage; they only execute what was decided when the stakes were low.

The switch must be tested

Flip the switch on purpose, before the failure does it for you. A quarterly failover that actually cuts traffic off the primary and brings it up correctly is worth a hundred diagrams, because it validates DNS, certificates, config and the catch-up path all at once. Each test should start clean: commit the basis, confirm backups, and have a planned rollback before the drill begins. A failure during the drill is a gift, not a setback, because it surfaces the missing permission or the stale script at a moment you choose. Time the run and record the result, so drift is measured across quarters instead of assumed away.

The cost of not planning

The unplanned failover is where the real cost lands. Without a rehearsed turn, the first live switch burns minutes fumbling through half-recalled steps, and a second team-loop forms to argue about who gets to act. Meanwhile an unprotected primary that catches fire has every chance of elevating a short blip into a lost-writes incident. The discipline also rots silently: settings change, and the one unused procedure in the corner is exactly the one that breaks first when it finally matters. Because the switch only exists when it is standing, the most honest report of your readiness is the date of the last successful drill.

When it breaks

Five failure modes punish an unrehearsed switch. Split-brain dual-write is the worst: the old primary keeps accepting writes while the standby promotes itself, so two histories diverge and neither copy is complete; fencing the old primary before promotion is the only reliable fix. Stale-replica promotion is quieter: the standby lags by minutes, the switch reports success, and the missing writes surface the next morning as customer complaints. Automatic failover can flap when a borderline health check toggles, promoting back and forth until both copies look suspect and clients see alternating versions. DNS failover sticks for a different reason: resolvers and clients honor the old TTL, so part of the traffic keeps hitting the dead address long after the record changed. Finally, failback collides: the repaired old node returns with confident timestamps and an unguarded return overwrites writes accepted during the outage. Every case points to the same guards: fence first, check lag, and hold exactly one writer.

Check it quickly

Save the following as failover_drill.py in a scratch directory and run python3 failover_drill.py. It builds a two-node cluster in memory, applies health and lag guards, and changes no production state. The four refusals are negative controls, not recommended operations.

"""Failover drill: single-primary switch with health and lag guards (stdlib only)."""
from dataclasses import dataclass

MAX_LAG = 128

class FailoverRefused(Exception):
    pass

@dataclass
class Node:
    name: str
    healthy: bool = True
    lag: int = 0
    is_primary: bool = False

class Cluster:
    def __init__(self):
        self.nodes = {"a": Node("a", is_primary=True), "b": Node("b")}
        self.fenced: set[str] = set()

    @property
    def primary(self) -> str:
        primaries = [n for n in self.nodes.values() if n.is_primary]
        assert len(primaries) == 1, f"must keep exactly one primary, saw {[p.name for p in primaries]}"
        return primaries[0].name

    def health_check(self, name: str) -> bool:
        node = self.nodes[name]
        return node.healthy and node.lag <= MAX_LAG

    def fence(self, name: str) -> None:
        self.fenced.add(name)

    def failover(self, target: str, *, fence_old: bool) -> str:
        old = self.primary
        if target == old:
            raise FailoverRefused("target is already primary")
        if target not in self.nodes:
            raise FailoverRefused(f"unknown target {target}")
        candidate = self.nodes[target]
        if not fence_old or old not in self.fenced:
            raise FailoverRefused(f"refuse: old primary {old} is not fenced; fence first to avoid split brain")
        if not candidate.healthy:
            raise FailoverRefused(f"refuse: standby {target} is unhealthy")
        if candidate.lag > MAX_LAG:
            raise FailoverRefused(f"refuse: standby {target} lags by {candidate.lag} (> {MAX_LAG})")
        self.nodes[old].is_primary = False
        candidate.is_primary = True
        assert self.primary == target
        return target

def expect_refused(fn, label: str) -> None:
    try:
        fn()
    except FailoverRefused as exc:
        print(f"refused as expected ({label}): {exc}")
        return
    raise AssertionError(f"should have refused: {label}")

def main() -> None:
    # 1. Clean drill: healthy standby, caught-up replica, old primary fenced.
    cluster = Cluster()
    assert cluster.primary == "a"
    assert cluster.health_check("b")
    cluster.fence("a")
    assert cluster.failover("b", fence_old=True) == "b"
    print("clean switch: a -> b")

    # 2. Stale replica must not promote: switching would drop writes.
    cluster = Cluster()
    cluster.nodes["b"].lag = 900
    assert not cluster.health_check("b")
    cluster.fence("a")
    expect_refused(lambda: cluster.failover("b", fence_old=True), "stale replica")
    assert cluster.primary == "a"

    # 3. Unhealthy standby must not promote.
    cluster.nodes["b"].lag = 0
    cluster.nodes["b"].healthy = False
    expect_refused(lambda: cluster.failover("b", fence_old=True), "unhealthy standby")
    assert cluster.primary == "a"

    # 4. Unfenced old primary must block promotion (split-brain guard).
    cluster.nodes["b"].healthy = True
    unfenced = Cluster()  # nobody fenced yet
    expect_refused(lambda: unfenced.failover("b", fence_old=False), "split brain")
    expect_refused(lambda: unfenced.failover("b", fence_old=True), "split brain without fence record")
    assert unfenced.primary == "a"

    # 5. Failback needs the same guards: fence the failed node before returning.
    cluster = Cluster()
    cluster.fence("a")
    cluster.failover("b", fence_old=True)
    cluster.nodes["a"].healthy = True
    cluster.nodes["a"].lag = 0
    cluster.fence("b")
    assert cluster.failover("a", fence_old=True) == "a"
    print("failback: b -> a")

    print("PASS: single primary held; 4 refusals + clean switch + failback behaved as expected")

if __name__ == "__main__":
    main()

The five steps exercise one clean switch, three refusal paths, and one guarded failback. The clean switch promotes the caught-up standby only after the old primary is fenced. The stale-replica and unhealthy-standby cases keep the primary in place instead of dropping writes. The split-brain attempts fail because promotion without fencing is refused. Failback repeats the same rule in reverse. Expected output ends with PASS: single primary held; 4 refusals + clean switch + failback behaved as expected, plus the clean switch and failback lines and four refused as expected lines.

The assertions passed with Python 3.14.4 on macOS. The fixture models decision logic, not replication transport, DNS propagation, or storage fencing. Before an incident, repeat the drill against a staging pair with real health checks, real lag metrics, and the actual fence command from the runbook.

Further reading

Sources