Failover · XB Field Notes

Failover is a practiced switch

Thu, 25 Feb 2027 00:00:00 +0000

A standby node only helps if you know how to switch to it and that the switch works. Failover is a decision, a procedure, and a rehearsal. Delegating it to a switch you have never thrown is how a graceful handoff becomes an outage.

Design the decision in advance

Decide, while everything is healthy, who owns the switch, what data loss is acceptable, and whether failover is automatic or requires a person to pull the trigger. Write the runbook while calm: the exact command or control-plane step, the check that confirms the replica has caught up, and the shout-outs to raise. Auto-failover wins on speed but needs the split-brain case thought through, because two writers each believing they are the leader corrupt more state than one degraded primary ever will. The goal is that the human in the middle does not have to invent policy during the outage; they only execute what was decided when the stakes were low.

The switch must be tested

Flip the switch on purpose, before the failure does it for you. A quarterly failover that actually cuts traffic off the primary and brings it up correctly is worth a hundred diagrams, because it validates DNS, certificates, config and the catch-up path all at once. Each test should start clean: commit the basis, confirm backups, and have a planned rollback before the drill begins. A failure during the drill is a gift, not a setback, because it surfaces the missing permission or the stale script at a moment you choose. Time the run and record the result, so drift is measured across quarters instead of assumed away.

The cost of not planning

The unplanned failover is where the real cost lands. Without a rehearsed turn, the first live switch burns minutes fumbling through half-recalled steps, and a second team-loop forms to argue about who gets to act. Meanwhile an unprotected primary that catches fire has every chance of elevating a short blip into a lost-writes incident. The discipline also rots silently: settings change, and the one unused procedure in the corner is exactly the one that breaks first when it finally matters. Because the switch only exists when it is standing, the most honest report of your readiness is the date of the last successful drill.

When it breaks

Five failure modes punish an unrehearsed switch. Split-brain dual-write is the worst: the old primary keeps accepting writes while the standby promotes itself, so two histories diverge and neither copy is complete; fencing the old primary before promotion is the only reliable fix. Stale-replica promotion is quieter: the standby lags by minutes, the switch reports success, and the missing writes surface the next morning as customer complaints. Automatic failover can flap when a borderline health check toggles, promoting back and forth until both copies look suspect and clients see alternating versions. DNS failover sticks for a different reason: resolvers and clients honor the old TTL, so part of the traffic keeps hitting the dead address long after the record changed. Finally, failback collides: the repaired old node returns with confident timestamps and an unguarded return overwrites writes accepted during the outage. Every case points to the same guards: fence first, check lag, and hold exactly one writer.

Check it quickly

Save the following as failover_drill.py in a scratch directory and run python3 failover_drill.py. It builds a two-node cluster in memory, applies health and lag guards, and changes no production state. The four refusals are negative controls, not recommended operations.

"""Failover drill: single-primary switch with health and lag guards (stdlib only)."""
from dataclasses import dataclass

MAX_LAG = 128

class FailoverRefused(Exception):
 pass

@dataclass
class Node:
 name: str
 healthy: bool = True
 lag: int = 0
 is_primary: bool = False

class Cluster:
 def __init__(self):
 self.nodes = {"a": Node("a", is_primary=True), "b": Node("b")}
 self.fenced: set[str] = set()

 @property
 def primary(self) -> str:
 primaries = [n for n in self.nodes.values() if n.is_primary]
 assert len(primaries) == 1, f"must keep exactly one primary, saw {[p.name for p in primaries]}"
 return primaries[0].name

 def health_check(self, name: str) -> bool:
 node = self.nodes[name]
 return node.healthy and node.lag <= MAX_LAG

 def fence(self, name: str) -> None:
 self.fenced.add(name)

 def failover(self, target: str, *, fence_old: bool) -> str:
 old = self.primary
 if target == old:
 raise FailoverRefused("target is already primary")
 if target not in self.nodes:
 raise FailoverRefused(f"unknown target {target}")
 candidate = self.nodes[target]
 if not fence_old or old not in self.fenced:
 raise FailoverRefused(f"refuse: old primary {old} is not fenced; fence first to avoid split brain")
 if not candidate.healthy:
 raise FailoverRefused(f"refuse: standby {target} is unhealthy")
 if candidate.lag > MAX_LAG:
 raise FailoverRefused(f"refuse: standby {target} lags by {candidate.lag} (> {MAX_LAG})")
 self.nodes[old].is_primary = False
 candidate.is_primary = True
 assert self.primary == target
 return target

def expect_refused(fn, label: str) -> None:
 try:
 fn()
 except FailoverRefused as exc:
 print(f"refused as expected ({label}): {exc}")
 return
 raise AssertionError(f"should have refused: {label}")

def main() -> None:
 # 1. Clean drill: healthy standby, caught-up replica, old primary fenced.
 cluster = Cluster()
 assert cluster.primary == "a"
 assert cluster.health_check("b")
 cluster.fence("a")
 assert cluster.failover("b", fence_old=True) == "b"
 print("clean switch: a -> b")

 # 2. Stale replica must not promote: switching would drop writes.
 cluster = Cluster()
 cluster.nodes["b"].lag = 900
 assert not cluster.health_check("b")
 cluster.fence("a")
 expect_refused(lambda: cluster.failover("b", fence_old=True), "stale replica")
 assert cluster.primary == "a"

 # 3. Unhealthy standby must not promote.
 cluster.nodes["b"].lag = 0
 cluster.nodes["b"].healthy = False
 expect_refused(lambda: cluster.failover("b", fence_old=True), "unhealthy standby")
 assert cluster.primary == "a"

 # 4. Unfenced old primary must block promotion (split-brain guard).
 cluster.nodes["b"].healthy = True
 unfenced = Cluster() # nobody fenced yet
 expect_refused(lambda: unfenced.failover("b", fence_old=False), "split brain")
 expect_refused(lambda: unfenced.failover("b", fence_old=True), "split brain without fence record")
 assert unfenced.primary == "a"

 # 5. Failback needs the same guards: fence the failed node before returning.
 cluster = Cluster()
 cluster.fence("a")
 cluster.failover("b", fence_old=True)
 cluster.nodes["a"].healthy = True
 cluster.nodes["a"].lag = 0
 cluster.fence("b")
 assert cluster.failover("a", fence_old=True) == "a"
 print("failback: b -> a")

 print("PASS: single primary held; 4 refusals + clean switch + failback behaved as expected")

if __name__ == "__main__":
 main()

The five steps exercise one clean switch, three refusal paths, and one guarded failback. The clean switch promotes the caught-up standby only after the old primary is fenced. The stale-replica and unhealthy-standby cases keep the primary in place instead of dropping writes. The split-brain attempts fail because promotion without fencing is refused. Failback repeats the same rule in reverse. Expected output ends with PASS: single primary held; 4 refusals + clean switch + failback behaved as expected, plus the clean switch and failback lines and four refused as expected lines.

The assertions passed with Python 3.14.4 on macOS. The fixture models decision logic, not replication transport, DNS propagation, or storage fencing. Before an incident, repeat the drill against a staging pair with real health checks, real lag metrics, and the actual fence command from the runbook.

Sources

AWS: failover and Route 53, configuring a real failover, checked 2027-02-23.
Google SRE: disaster recovery layout, planning and testing recovery, checked 2027-02-23.
MariaDB/MySQL: replication failover, a database failover example, checked 2027-02-23.

Redis Sentinel promotes a new primary when the old one stops

Sat, 09 Jan 2027 00:00:00 +0000

Redis Sentinel supervises a non-Cluster primary/replica deployment and publishes the current primary’s address. It does not store the application’s dataset or proxy its commands. Automatic promotion can restore write availability after a primary fails, but discovery, replica freshness, client reconnection, and protection from stale writers remain separate parts of the system.

Failure agreement and election are different gates

A Sentinel first marks an instance subjectively down when acceptable replies are absent long enough. A primary becomes objectively down when the configured quorum agrees. That failure-detection threshold is not the same as the majority needed to authorize a failover leader. Election also needs at least the configured quorum; setting a higher quorum can make the requirement stricter.

For five known Sentinels with quorum two, two observers can agree about failure, but a majority of three is needed to authorize promotion. Conversely, quorum one does not let a single surviving observer authorize failover when it still knows about two others. SENTINEL CKQUORUM checks both reachability requirements; a healthy-looking Redis process does not establish either one.

Use at least three Sentinels in independent failure domains for a robust deployment. Three processes on one laptop help test protocol behavior but all fail with that machine. Each Sentinel needs a writable configuration file to persist discovered topology and epochs. Treat those files as operational state and protect any credentials they contain.

Promotion does not fence every old writer

Redis replication is asynchronous. A replica selected for promotion may lack a write that the old primary acknowledged. During a network partition, clients on the old primary’s side may even continue writing there while another side promotes a replacement. Sentinel’s election coordination does not physically revoke every old connection or turn Redis into a strongly consistent store.

Settings such as min-replicas-to-write and min-replicas-max-lag can restrict writes when replication health degrades, at an availability cost; they are not a universal fencing protocol. Independent backups and an explicit application recovery policy remain necessary. Test partitions and replica lag separately from merely killing a process.

Clients must rediscover and reconnect

A Sentinel-aware client asks observers for the primary address and then connects directly to Redis. It must handle temporary discovery failures, broken connections, and changed roles. A client pinned to the old address does not become healthy because an observer has a new answer. Likewise, a write timeout does not prove that its operation was never applied.

The example explicitly disconnects the application’s old pool after discovery reports the promoted replica, then checks a new connection. This isolates the reconnection step instead of promising invisible, interruption-free failover. Do not blindly repeat nonidempotent writes during that interval; promotion is not a business deduplication mechanism.

When it breaks

Insufficient reachable observers, no eligible replica, unwritable Sentinel state, or unusable announced addresses can prevent recovery. Authentication and network rules must permit observer-to-observer, observer-to-Redis, replication, and application connections as appropriate. Seeing one port respond to PING is not proof that all these paths work.

Monitor CKQUORUM, replica health, failover events, discovery results, and actual client operations. Detection and election take time; aggressive timeouts can mistake pauses for failures. Rehearse recovery with the deployed clients, topology, and security configuration before claiming an outage budget.

Check it quickly

Use a POSIX machine with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Redis 8.10.1, redis-py 8.1.0, and Python 3.14.4. The setup builds Redis core with libc and installs the client in a temporary virtual environment.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python". It creates one primary, one replica, and three Sentinels on reserved loopback ports, with private Unix-socket control connections and temporary writable configurations. Database persistence is disabled. Only its own subprocesses are killed; cleanup closes clients, stops remaining processes, and removes the fixture. Short detection settings are for this exercise, not production recommendations.

from contextlib import contextmanager, ExitStack
import os
from pathlib import Path
import signal
import socket
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry
from redis.sentinel import MasterNotFoundError, Sentinel


def until(check):
 deadline = time.monotonic() + 20
 while not check():
 assert time.monotonic() < deadline, "Sentinel state did not converge"
 time.sleep(0.05)


@contextmanager
def instance(directory, port, monitor=None):
 directory.mkdir()
 endpoint = str(directory / "redis.sock")
 config = directory / "redis.conf"
 lines = ["bind 127.0.0.1", f"port {port}", "protected-mode yes",
 f"unixsocket {endpoint}", "unixsocketperm 700", f"dir {directory}"]
 if monitor is None:
 lines += ['save ""', "appendonly no"]
 else:
 lines += [f"sentinel monitor lab 127.0.0.1 {monitor} 2",
 "sentinel down-after-milliseconds lab 500",
 "sentinel failover-timeout lab 5000",
 "sentinel parallel-syncs lab 1",
 "sentinel announce-ip 127.0.0.1"]
 config.write_text("\n".join(lines) + "\n")
 command = [os.environ["REDIS_SERVER"], str(config)]
 if monitor is not None:
 command.append("--sentinel")
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 process = subprocess.Popen(command, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 assert time.monotonic() < deadline, "Redis did not start"
 time.sleep(0.02)
 assert client.info("server")["process_id"] == process.pid
 yield client, process
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode in (0, -signal.SIGKILL), errors.decode()


def crash(process):
 assert process.poll() is None
 process.kill()
 assert process.wait(timeout=5) == -signal.SIGKILL


with TemporaryDirectory(prefix="redis-sentinel-", dir="/tmp") as directory, ExitStack() as stack:
 leases = [stack.enter_context(socket.socket()) for _ in range(5)]
 for lease in leases:
 lease.bind(("127.0.0.1", 0))
 ports = [lease.getsockname()[1] for lease in leases]
 nodes = []
 for index in range(5):
 leases[index].close()
 nodes.append(stack.enter_context(instance(
 Path(directory) / str(index), ports[index], ports[0] if index >= 2 else None,
 )))
 primary, replica = nodes[0][0], nodes[1][0]
 observers = [item[0] for item in nodes[2:]]
 assert replica.replicaof("127.0.0.1", ports[0])
 until(lambda: replica.info("replication")["master_link_status"] == "up")
 until(lambda: all(observer.sentinel_master("lab")["num-other-sentinels"] == 2
 and len(observer.execute_command("SENTINEL", "REPLICAS", "lab")) == 1
 for observer in observers))
 for observer in observers:
 assert observer.execute_command("SENTINEL", "CKQUORUM", "lab").startswith(b"OK 3 usable")
 with primary.client() as writer:
 assert writer.set("marker", "replicated-before-failure")
 assert writer.wait(1, 5000) == 1
 assert replica.get("marker") == b"replicated-before-failure"
 print("three Sentinels discover each other and a replica; WAIT confirms the fixture write")

 manager = Sentinel([("127.0.0.1", port) for port in ports[2:]],
 socket_timeout=1, retry=Retry(NoBackoff(), 0),
 sentinel_kwargs={"socket_timeout": 1, "retry": Retry(NoBackoff(), 0)})
 application = manager.master_for("lab")
 try:
 assert manager.discover_master("lab") == ("127.0.0.1", ports[0])
 assert application.get("marker") == b"replicated-before-failure"
 crash(nodes[0][1])
 try:
 primary.get("marker")
 except redis.ConnectionError:
 pass
 else:
 raise AssertionError("the original primary connection must fail")

 def switched():
 try:
 return (manager.discover_master("lab") == ("127.0.0.1", ports[1])
 and replica.info("replication")["role"] == "master")
 except MasterNotFoundError:
 return False

 until(switched)
 application.connection_pool.disconnect()
 assert application.get("marker") == b"replicated-before-failure"
 assert application.set("after-failover", "new-primary")
 assert replica.get("after-failover") == b"new-primary"
 print("automatic failover changes discovery; a reconnected client uses the promoted replica")
 finally:
 application.close()
 for sentinel_client in manager.sentinels:
 sentinel_client.close()

 for item in nodes[3:]:
 crash(item[1])
 until(lambda: len(observers[0].sentinel_sentinels("lab")) == 2
 and all(peer["is_sdown"] for peer in observers[0].sentinel_sentinels("lab")))
 assert observers[0].execute_command("SENTINEL", "SET", "lab", "quorum", 1) == b"OK"
 assert observers[0].sentinel_master("lab")["quorum"] == 1
 try:
 observers[0].execute_command("SENTINEL", "CKQUORUM", "lab")
 except redis.ResponseError as error:
 assert "majority" in str(error) and "specified quorum" not in str(error)
 print("one of three known Sentinels meets quorum=1 but cannot authorize failover")
 else:
 raise AssertionError("one surviving observer is not a majority of three")

The fixture waits for peer and replica discovery, then writes a marker and checks WAIT on that same connection. After the primary is killed, its old connection fails, discovery identifies the promoted replica, and a reconnected application reads the marker and writes new data.

Finally, two observers are killed and the survivor’s quorum is deliberately lowered to one. CKQUORUM still rejects authorization because one of three known observers is not a majority. This checks the diagnostic gate, not a second promotion attempt. WAIT confirms the fixture’s replication acknowledgment, not disk durability or strong consistency. Network partitions, stalled replicas, independent-host failures, and production recovery latency remain untested.

Sources

Redis Sentinel, failure detection, authorization, discovery, and partition risks, checked 2026-09-08.
Redis replication, asynchronous replication and write restrictions, checked 2026-09-08.
Redis WAIT, connection-scoped acknowledgment and consistency limits, checked 2026-09-08.
Redis 8.10.1 Sentinel source, separate quorum and majority checks, checked 2026-09-08.
redis-py 8.1.0 Sentinel client, discovery and primary connection pools, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only build and allocator selection, checked 2026-09-08.

Failover · XB Field Notes

Failover is a practiced switch

Design the decision in advance

The switch must be tested

The cost of not planning

When it breaks

Check it quickly

Further reading

Sources

Redis Sentinel promotes a new primary when the old one stops

Failure agreement and election are different gates

Promotion does not fence every old writer

Clients must rediscover and reconnect

When it breaks

Check it quickly

Sources