Redis Sentinel promotes a new primary when the old one stops

Sat, 09 Jan 2027 00:00:00 +0000

Redis Sentinel supervises a non-Cluster primary/replica deployment and publishes the current primary’s address. It does not store the application’s dataset or proxy its commands. Automatic promotion can restore write availability after a primary fails, but discovery, replica freshness, client reconnection, and protection from stale writers remain separate parts of the system.

Failure agreement and election are different gates

A Sentinel first marks an instance subjectively down when acceptable replies are absent long enough. A primary becomes objectively down when the configured quorum agrees. That failure-detection threshold is not the same as the majority needed to authorize a failover leader. Election also needs at least the configured quorum; setting a higher quorum can make the requirement stricter.

For five known Sentinels with quorum two, two observers can agree about failure, but a majority of three is needed to authorize promotion. Conversely, quorum one does not let a single surviving observer authorize failover when it still knows about two others. SENTINEL CKQUORUM checks both reachability requirements; a healthy-looking Redis process does not establish either one.

Use at least three Sentinels in independent failure domains for a robust deployment. Three processes on one laptop help test protocol behavior but all fail with that machine. Each Sentinel needs a writable configuration file to persist discovered topology and epochs. Treat those files as operational state and protect any credentials they contain.

Promotion does not fence every old writer

Redis replication is asynchronous. A replica selected for promotion may lack a write that the old primary acknowledged. During a network partition, clients on the old primary’s side may even continue writing there while another side promotes a replacement. Sentinel’s election coordination does not physically revoke every old connection or turn Redis into a strongly consistent store.

Settings such as min-replicas-to-write and min-replicas-max-lag can restrict writes when replication health degrades, at an availability cost; they are not a universal fencing protocol. Independent backups and an explicit application recovery policy remain necessary. Test partitions and replica lag separately from merely killing a process.

Clients must rediscover and reconnect

A Sentinel-aware client asks observers for the primary address and then connects directly to Redis. It must handle temporary discovery failures, broken connections, and changed roles. A client pinned to the old address does not become healthy because an observer has a new answer. Likewise, a write timeout does not prove that its operation was never applied.

The example explicitly disconnects the application’s old pool after discovery reports the promoted replica, then checks a new connection. This isolates the reconnection step instead of promising invisible, interruption-free failover. Do not blindly repeat nonidempotent writes during that interval; promotion is not a business deduplication mechanism.

When it breaks

Insufficient reachable observers, no eligible replica, unwritable Sentinel state, or unusable announced addresses can prevent recovery. Authentication and network rules must permit observer-to-observer, observer-to-Redis, replication, and application connections as appropriate. Seeing one port respond to PING is not proof that all these paths work.

Monitor CKQUORUM, replica health, failover events, discovery results, and actual client operations. Detection and election take time; aggressive timeouts can mistake pauses for failures. Rehearse recovery with the deployed clients, topology, and security configuration before claiming an outage budget.

Check it quickly

Use a POSIX machine with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Redis 8.10.1, redis-py 8.1.0, and Python 3.14.4. The setup builds Redis core with libc and installs the client in a temporary virtual environment.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python". It creates one primary, one replica, and three Sentinels on reserved loopback ports, with private Unix-socket control connections and temporary writable configurations. Database persistence is disabled. Only its own subprocesses are killed; cleanup closes clients, stops remaining processes, and removes the fixture. Short detection settings are for this exercise, not production recommendations.

from contextlib import contextmanager, ExitStack
import os
from pathlib import Path
import signal
import socket
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry
from redis.sentinel import MasterNotFoundError, Sentinel


def until(check):
 deadline = time.monotonic() + 20
 while not check():
 assert time.monotonic() < deadline, "Sentinel state did not converge"
 time.sleep(0.05)


@contextmanager
def instance(directory, port, monitor=None):
 directory.mkdir()
 endpoint = str(directory / "redis.sock")
 config = directory / "redis.conf"
 lines = ["bind 127.0.0.1", f"port {port}", "protected-mode yes",
 f"unixsocket {endpoint}", "unixsocketperm 700", f"dir {directory}"]
 if monitor is None:
 lines += ['save ""', "appendonly no"]
 else:
 lines += [f"sentinel monitor lab 127.0.0.1 {monitor} 2",
 "sentinel down-after-milliseconds lab 500",
 "sentinel failover-timeout lab 5000",
 "sentinel parallel-syncs lab 1",
 "sentinel announce-ip 127.0.0.1"]
 config.write_text("\n".join(lines) + "\n")
 command = [os.environ["REDIS_SERVER"], str(config)]
 if monitor is not None:
 command.append("--sentinel")
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 process = subprocess.Popen(command, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 assert time.monotonic() < deadline, "Redis did not start"
 time.sleep(0.02)
 assert client.info("server")["process_id"] == process.pid
 yield client, process
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode in (0, -signal.SIGKILL), errors.decode()


def crash(process):
 assert process.poll() is None
 process.kill()
 assert process.wait(timeout=5) == -signal.SIGKILL


with TemporaryDirectory(prefix="redis-sentinel-", dir="/tmp") as directory, ExitStack() as stack:
 leases = [stack.enter_context(socket.socket()) for _ in range(5)]
 for lease in leases:
 lease.bind(("127.0.0.1", 0))
 ports = [lease.getsockname()[1] for lease in leases]
 nodes = []
 for index in range(5):
 leases[index].close()
 nodes.append(stack.enter_context(instance(
 Path(directory) / str(index), ports[index], ports[0] if index >= 2 else None,
 )))
 primary, replica = nodes[0][0], nodes[1][0]
 observers = [item[0] for item in nodes[2:]]
 assert replica.replicaof("127.0.0.1", ports[0])
 until(lambda: replica.info("replication")["master_link_status"] == "up")
 until(lambda: all(observer.sentinel_master("lab")["num-other-sentinels"] == 2
 and len(observer.execute_command("SENTINEL", "REPLICAS", "lab")) == 1
 for observer in observers))
 for observer in observers:
 assert observer.execute_command("SENTINEL", "CKQUORUM", "lab").startswith(b"OK 3 usable")
 with primary.client() as writer:
 assert writer.set("marker", "replicated-before-failure")
 assert writer.wait(1, 5000) == 1
 assert replica.get("marker") == b"replicated-before-failure"
 print("three Sentinels discover each other and a replica; WAIT confirms the fixture write")

 manager = Sentinel([("127.0.0.1", port) for port in ports[2:]],
 socket_timeout=1, retry=Retry(NoBackoff(), 0),
 sentinel_kwargs={"socket_timeout": 1, "retry": Retry(NoBackoff(), 0)})
 application = manager.master_for("lab")
 try:
 assert manager.discover_master("lab") == ("127.0.0.1", ports[0])
 assert application.get("marker") == b"replicated-before-failure"
 crash(nodes[0][1])
 try:
 primary.get("marker")
 except redis.ConnectionError:
 pass
 else:
 raise AssertionError("the original primary connection must fail")

 def switched():
 try:
 return (manager.discover_master("lab") == ("127.0.0.1", ports[1])
 and replica.info("replication")["role"] == "master")
 except MasterNotFoundError:
 return False

 until(switched)
 application.connection_pool.disconnect()
 assert application.get("marker") == b"replicated-before-failure"
 assert application.set("after-failover", "new-primary")
 assert replica.get("after-failover") == b"new-primary"
 print("automatic failover changes discovery; a reconnected client uses the promoted replica")
 finally:
 application.close()
 for sentinel_client in manager.sentinels:
 sentinel_client.close()

 for item in nodes[3:]:
 crash(item[1])
 until(lambda: len(observers[0].sentinel_sentinels("lab")) == 2
 and all(peer["is_sdown"] for peer in observers[0].sentinel_sentinels("lab")))
 assert observers[0].execute_command("SENTINEL", "SET", "lab", "quorum", 1) == b"OK"
 assert observers[0].sentinel_master("lab")["quorum"] == 1
 try:
 observers[0].execute_command("SENTINEL", "CKQUORUM", "lab")
 except redis.ResponseError as error:
 assert "majority" in str(error) and "specified quorum" not in str(error)
 print("one of three known Sentinels meets quorum=1 but cannot authorize failover")
 else:
 raise AssertionError("one surviving observer is not a majority of three")

The fixture waits for peer and replica discovery, then writes a marker and checks WAIT on that same connection. After the primary is killed, its old connection fails, discovery identifies the promoted replica, and a reconnected application reads the marker and writes new data.

Finally, two observers are killed and the survivor’s quorum is deliberately lowered to one. CKQUORUM still rejects authorization because one of three known observers is not a majority. This checks the diagnostic gate, not a second promotion attempt. WAIT confirms the fixture’s replication acknowledgment, not disk durability or strong consistency. Network partitions, stalled replicas, independent-host failures, and production recovery latency remain untested.

Sources

Redis Sentinel, failure detection, authorization, discovery, and partition risks, checked 2026-09-08.
Redis replication, asynchronous replication and write restrictions, checked 2026-09-08.
Redis WAIT, connection-scoped acknowledgment and consistency limits, checked 2026-09-08.
Redis 8.10.1 Sentinel source, separate quorum and majority checks, checked 2026-09-08.
redis-py 8.1.0 Sentinel client, discovery and primary connection pools, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only build and allocator selection, checked 2026-09-08.