Redis · XB Field Notes

A Redis hash stores many fields in one key

Thu, 24 Feb 2028 00:00:00 +0000

A Redis hash is a field-addressable record under one key, not a schema-checked object or an automatically nested document. It is useful when callers need a few attributes without replacing the whole record. That advantage depends on access patterns, field sizes, and the meaning of replies, not merely on choosing the hash type.

Read the fields your caller actually needs

HGET retrieves one field. HMGET retrieves selected fields in the requested order, while HGETALL returns the entire mapping. With redis-py’s binary responses, an absent field becomes None and an existing empty value becomes empty bytes. A truthiness check incorrectly treats those two states as equivalent. An absent key also produces missing values through HMGET.

Field names and values are bytes; a name containing a dot is still a literal name. Redis does not interpret name.first as traversal through a nested object. Likewise, a string containing JSON remains a Redis string: HGET does not parse it. Decide serialization and validation in the application and preserve the distinction between an omitted attribute and an intentionally empty one.

Updates have precise return values

HSET changes the specified fields and leaves others untouched. A multi-field call is one command, but its integer reply counts newly added fields, not successful replacements. Updating an existing name can correctly return zero. Treating that zero as failure can cause needless retries or misleading error reports.

HINCRBY changes an integer field in one command and starts from zero when the field is absent. It rejects noninteger values and overflow beyond signed 64-bit range. This avoids a read-modify-write race for that increment; it does not make surrounding application checks atomic. Separate HGET, local arithmetic, and HSET can still overwrite another client’s update.

HDEL counts fields actually removed, and deleting the final field removes the key. Do not use an empty hash as a persistent placeholder for an entity. Whole-key expiration also removes its fields. Current Redis additionally supports field deadlines; the exercise below assigns no expirations and does not test lifetime behavior.

Compact representation is an implementation choice

Small hashes can use listpack encoding instead of a hash table. Redis 8.10.1’s standard thresholds are 512 entries and 64 bytes per field name or value; deployed settings can differ. Crossing an applicable limit changes the representation, not the public hash operations or the stored values.

Compact storage trades internal scanning work for reduced overhead and locality. Raising thresholds substantially requires measurement: a command’s advertised complexity is not a promise that arbitrarily large fields or changed configuration have no cost. Inspect OBJECT ENCODING when diagnosing representation, but do not make application correctness depend on that internal label or an assumed memory-saving ratio.

When it breaks

HGETALL grows with the whole hash and transfers both names and values. Fetching thousands of fields when the caller needs two wastes work, even if they share one key. HMGET’s work grows with the requested field count; it is not free either. Never use returned map iteration order as an application ordering contract.

WRONGTYPE indicates that the key holds another Redis type, not that a field is missing. Noninteger counters require an explicit data decision, not a retry loop. One huge hash also remains one key rather than automatically spreading its fields across cluster nodes. Choose record boundaries around actual access and ownership rather than packing unrelated data together.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Redis 8.10.1, redis-py 8.1.0, and Python 3.14.4. The setup builds only Redis core with libc and installs the client in a temporary virtual environment.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It starts a private Unix-socket server with TCP and persistence disabled. Only this server receives deliberately small thresholds: four entries and eight bytes. The test closes its client, terminates its own process, and removes the temporary runtime directory; it never connects to a default local Redis.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-hash-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--hash-max-listpack-entries", "4", "--hash-max-listpack-value", "8",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


with instance() as client:
 assert client.hset("profile", mapping={"name": "Ada", "visits": "0", "empty": b""}) == 3
 assert client.hset("profile", "name", "Ada L") == 0
 assert client.hmget("profile", ["visits", "name", "missing", "empty"]) == [
 b"0", b"Ada L", None, b"",
 ]
 assert client.hmget("absent", ["a", "b"]) == [None, None]
 assert client.hgetall("profile") == {b"name": b"Ada L", b"visits": b"0", b"empty": b""}
 assert client.hincrby("profile", "visits", 1) == 1
 assert client.hget("profile", "name") == b"Ada L"
 print("HSET counts new fields; HMGET preserves order and distinguishes missing from empty")

 client.set("json", '{"name":"Ada"}')
 try:
 client.hget("json", "name")
 except redis.ResponseError as error:
 assert "WRONGTYPE" in str(error)
 else:
 raise AssertionError("a JSON string is not a hash")
 client.hset("literal", "name.first", b"Ada")
 assert client.hget("literal", "name") is None
 assert client.hget("literal", "name.first") == b"Ada"
 client.hset("invalid", "counter", "3.5")
 try:
 client.hincrby("invalid", "counter", 1)
 except redis.ResponseError as error:
 assert "integer" in str(error)
 else:
 raise AssertionError("an integer counter cannot contain 3.5")
 assert client.hget("invalid", "counter") == b"3.5"
 client.hset("overflow", "counter", 2**63 - 1)
 try:
 client.hincrby("overflow", "counter", 1)
 except redis.ResponseError as error:
 assert "overflow" in str(error)
 else:
 raise AssertionError("signed 64-bit integer overflow must fail")
 assert client.hget("overflow", "counter") == str(2**63 - 1).encode()
 print("wrong types and invalid or overflowing counters fail; dotted fields are literal")

 assert client.object("ENCODING", "profile") == b"listpack"
 assert client.hset("profile", "fourth", "ok") == 1
 assert client.object("ENCODING", "profile") == b"listpack"
 assert client.hset("profile", "fifth", "ok") == 1
 assert client.object("ENCODING", "profile") == b"hashtable"
 client.hset("value-limit", "field", b"abcdefgh")
 assert client.object("ENCODING", "value-limit") == b"listpack"
 client.hset("value-limit", "field", b"abcdefghi")
 assert client.object("ENCODING", "value-limit") == b"hashtable"
 assert client.hget("profile", "visits") == b"1"
 assert client.hget("value-limit", "field") == b"abcdefghi"
 print("private 4-field and 8-byte thresholds each trigger listpack to hashtable conversion")

 assert client.hdel("value-limit", "absent") == 0
 assert client.hdel("value-limit", "field") == 1
 assert client.exists("value-limit") == 0
 print("deleting the last field removes the hash key")

The assertions distinguish added fields from replacements, preserve HMGET ordering and missing-versus-empty values, and verify unchanged neighboring fields. A JSON string raises WRONGTYPE, a dotted field stays literal, and invalid or overflowing counters remain unchanged after rejection.

At four short fields the profile remains listpack; the fifth switches it to hashtable. A separate hash switches when its value grows from eight to nine bytes. Reads still return the expected values, and deleting the final field removes that key. These results demonstrate configured encoding transitions and command semantics, not production memory savings, latency, concurrent-load behavior, or cluster distribution.

Sources

Redis hashes, field-oriented operations and current expiration capabilities, checked 2026-09-08.
Redis HSET, partial updates and added-field counts, checked 2026-09-08.
Redis HMGET, requested ordering and missing-field replies, checked 2026-09-08.
Redis HGETALL, full-hash reads and complexity, checked 2026-09-08.
Redis HINCRBY, integer increments and supported range, checked 2026-09-08.
Redis HDEL, removal counts and final-field deletion, checked 2026-09-08.
Redis OBJECT ENCODING, internal representations and conversion, checked 2026-09-08.
Redis memory optimization, compact encoding thresholds and measurement guidance, checked 2026-09-08.
Redis 8.10.1 configuration, default hash thresholds, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only build and allocator selection, checked 2026-09-08.

Redis pub/sub and streams are different tools

Sun, 06 Feb 2028 00:00:00 +0000

Redis Pub/Sub and Streams answer different delivery questions. Pub/Sub broadcasts to currently active subscriptions without retaining messages for later replay. A stream stores entries that readers can revisit while they remain retained. Choosing between them starts with who must receive an event and what should happen during a consumer’s absence.

Decide who receives each event

For a replaceable live dashboard update, broadcasting to connected viewers may be sufficient. Wait for subscription confirmation before assuming a new listener is active. Reconnecting restores future delivery, not missed publications. A positive PUBLISH response counts recipients according to Redis’s command semantics; it does not acknowledge application processing.

Streams offer two distinct consumption models. Ordinary XREAD lets independent readers request entries after their own supplied IDs, so both can receive the same retained history. It does not divide work or maintain an automatic checkpoint per connection. XREADGROUP within one consumer group distributes new deliveries among named workers and records pending entries unless NOACK is used.

Separate consumer groups allow separate workflows, such as billing and analytics, to progress independently over one stream. Membership within a group shares work; adding a new group creates another delivery history. Neither choosing unique consumer names nor adding workers promises equal load or business completion order. Decide whether the requirement is broadcast, independent replay, or competing work before choosing the command.

Retention is not a durability setting

XADD retains data in Redis; that does not by itself make every accepted append survive a crash or failover. RDB/AOF settings, replication behavior, key eviction, and operational recovery determine durability. Pub/Sub does not acquire a replay log by enabling persistence. Calling the difference simply latency versus durability hides both contracts.

MAXLEN trims the oldest entries, not the newest tail while preserving older history. Approximate trimming may retain extra records, while exact trimming follows the requested count. Neither makes unlimited replay possible. Default trimming can remove payloads still referenced by pending entries; select retention policies and capacity for the longest supported outage and subsequent catch-up.

Stored backlog also has an operational cost. Bound its growth, monitor unread lag and pending work separately, and define what happens to poison messages. Pub/Sub instead avoids retained history but still uses connection output buffers. A slow subscriber can be disconnected when those limits are exceeded. Neither mechanism eliminates memory planning.

When it breaks

Using Pub/Sub for required jobs silently loses the opportunity to deliver while no listener is subscribed. A dashboard that can reload authoritative state may tolerate that; a required financial action usually cannot. The appropriate replacement might be a retained stream or another durable queue, with recovery semantics designed around the action rather than assumed from the product name.

Using Streams without a recovery loop creates a different failure: an unacknowledged group delivery can remain pending while ordinary > reads continue with new entries. Acknowledging before processing risks loss; processing before acknowledgement permits duplicate effects after retry. Stable event identifiers and idempotent handling remain necessary when repeated business actions are unacceptable.

Publishing and appending the same event as separate commands is not an atomic dual-write protocol. One operation can succeed without the other if execution stops between them. The exercise deliberately compares two mechanisms in an isolated process; it does not propose that sequence as a production delivery guarantee.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0. The commands compile Redis core with libc and install its client in a temporary virtual environment, without optional modules or system-wide installation.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It creates a private Unix socket under /tmp, disables TCP, scheduled snapshots, and append-only persistence, then stops only its own process and removes its runtime directory. Downloads require network access. No default local Redis is contacted.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", "noeviction",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


def receive(subscription):
 deadline = time.monotonic() + 2
 while time.monotonic() < deadline:
 response = subscription.get_message(timeout=0.1)
 if response is not None:
 return response
 raise AssertionError("subscription response timed out")


with instance() as client:
 assert client.publish("updates", b"offline") == 0
 first_id = client.xadd("history", {"event": "offline"}, id="1-0")
 with client.pubsub() as first, client.pubsub() as second:
 for subscription in (first, second):
 subscription.subscribe("updates")
 assert receive(subscription)["type"] == "subscribe"
 assert client.publish("updates", b"live") == 2
 second_id = client.xadd("history", {"event": "live"}, id="2-0")
 for subscription in (first, second):
 response = receive(subscription)
 assert response["type"] == "message" and response["data"] == b"live"
 with client.client() as reader_one, client.client() as reader_two:
 one = reader_one.xread({"history": "0-0"})
 two = reader_two.xread({"history": "0-0"})
 assert one == two
 assert [entry[0] for entry in one[0][1]] == [first_id, second_id]
 print("Pub/Sub listeners get live only; both independent XREAD readers replay offline and live")

 client.xgroup_create("history", "workers", id="0-0")
 a = client.xreadgroup("workers", "a", {"history": ">"}, count=1)[0][1]
 b = client.xreadgroup("workers", "b", {"history": ">"}, count=1)[0][1]
 assert a[0][0] == first_id and b[0][0] == second_id
 assert client.xpending("history", "workers")["pending"] == 2
 assert client.xack("history", "workers", first_id, second_id) == 2
 assert client.xlen("history") == 2
 print("one group splits new entries between a/b; ACK clears pending state, not stored history")

 third_id = client.xadd("history", {"event": "later"}, id="3-0")
 assert client.xtrim("history", maxlen=2, approximate=False) == 1
 retained = client.xread({"history": "0-0"})[0][1]
 assert [entry[0] for entry in retained] == [second_id, third_id]
 assert client.xrange("history", first_id, first_id) == []
 print("exact MAXLEN removes the oldest entry; replay is limited to retained history")

The offline publication returns zero, while XADD keeps an entry with the same event label. Once both subscriptions are confirmed, each listener receives live, not offline. Two ordinary XREAD calls then return both stored entries, proving independent replay rather than division of work.

Creating one group changes that delivery pattern: workers a and b receive different entries, and both become pending. Acknowledging them leaves the stream length unchanged. A third append followed by exact MAXLEN removes only the oldest retained entry; reading from zero cannot recover its missing payload.

The three output summaries follow assertions on actual responses. They demonstrate subscription presence, independent replay, grouped distribution, and finite retention. This run does not benchmark latency or memory, reproduce a real crash, validate persistence, or test automatic recovery. Those operational choices remain part of the application’s delivery contract.

Sources

Redis Pub/Sub, live delivery, subscription acknowledgement, and absent replay, checked 2026-09-08.
Redis XREAD, independent reads and caller-supplied positions, checked 2026-09-08.
Redis XREADGROUP, shared work and pending entries, checked 2026-09-08.
Redis XTRIM, oldest-entry removal and reference policies, checked 2026-09-08.
Redis client handling, Pub/Sub output buffering and disconnect limits, checked 2026-09-08.
Redis persistence, RDB and AOF recovery tradeoffs, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only compilation and allocator selection, checked 2026-09-08.

Redis Streams let a group share the work

Sat, 05 Feb 2028 00:00:00 +0000

A Redis Streams consumer group distributes new entries among named consumers and records deliveries awaiting acknowledgement. It does not guarantee exactly-once effects, prevent retention loss, or recover abandoned work by itself. Understanding the group position, pending entries, and retained stream data separately is essential before treating it as a work queue.

Position and pending work differ

XGROUP CREATE jobs workers 0-0 starts before the retained history; using $ instead starts after the current end. Workers in one group share new deliveries, while separate groups maintain independent delivery state. Use distinct names for concurrent workers. The group owns a last-delivered position, not a separate offset committed by each consumer.

For ordinary XREADGROUP without CLAIM or NOACK, > requests new entries and delivery adds them to the Pending Entries List, or PEL. The group position advances when new entries are delivered, before XACK. XPENDING reports delivered but unacknowledged entries, not the backlog that has never been delivered. A group can therefore have pending count zero while unread work exists.

XACK removes specified IDs from that group’s PEL. It does not delete their stream records or advance the last-delivered position. Acknowledgement can finish out of order, and another group can still read the retained entries. XREADGROUP with 0-0 reads this named consumer’s pending history instead of requesting new work; it does not retrieve another consumer’s pending messages.

Recovery needs an explicit decision

Repeated ordinary > reads do not retry an abandoned PEL entry. XAUTOCLAIM transfers eligible pending entries to the requesting consumer. Its arguments include the stream, group, consumer, minimum idle milliseconds, and starting ID. Continue scanning with the returned cursor until 0-0; an empty claimed batch alone does not mean the scan is complete. Later scans may find entries that have since become eligible.

Redis 8.4 also introduced XREADGROUP CLAIM as an explicit combined recovery option. The example does not enable it. Its XAUTOCLAIM idle threshold of zero is solely for deterministic local observation; production thresholds must reflect processing time and liveness policy. Claiming ownership cannot stop an old worker that is still running or undo its external side effects.

When it breaks

A crash after an external effect but before acknowledgement leaves work eligible for another delivery. Retrying can repeat that effect. Acknowledging before the effect instead risks losing required work. Use stable business identifiers and transactional deduplication where possible; a delivery counter records delivery attempts, not completed database transactions. Bound retries and preserve failing payloads before retiring work.

A PEL reference does not pin the stream payload. XDEL and default KEEPREF trimming can remove it while leaving a pending ID. XAUTOCLAIM reports and clears such missing records on Redis 7 and later; that is cleanup, not recovery of their contents. Choose retention with consumer outages in mind. Stream persistence and replication also require separate durability decisions.

Monitor unread lag, pending count, idle time, and repeated deliveries together. One count cannot establish queue health. NOACK omits pending tracking for new deliveries and is unsuitable when that tracking is required for recovery. Parallel workers can finish out of insertion order even when the stream IDs themselves remain ordered.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0. The commands build Redis core with libc and install the client in a temporary virtual environment, without optional modules or system-wide installation.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It creates a private Unix socket under /tmp, disables TCP, scheduled snapshots, and append-only persistence, then terminates only its own process and removes its runtime directory. Downloads require network access. No default local Redis is contacted.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", "noeviction",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


with instance() as client:
 for entry_id in ("1-0", "2-0", "3-0"):
 client.xadd("jobs", {"job": entry_id}, id=entry_id)
 client.xgroup_create("jobs", "workers", id="0-0")
 client.xgroup_create("jobs", "tail", id="$")
 assert client.xpending("jobs", "workers")["pending"] == 0
 assert client.xreadgroup("tail", "observer", {"jobs": ">"}) == []
 first = client.xreadgroup("workers", "w1", {"jobs": ">"}, count=1)[0][1]
 second = client.xreadgroup("workers", "w2", {"jobs": ">"}, count=1)[0][1]
 assert first[0][0] == b"1-0" and second[0][0] == b"2-0"
 assert client.xpending("jobs", "workers")["pending"] == 2
 group = next(item for item in client.xinfo_groups("jobs") if item["name"] == b"workers")
 assert group["last-delivered-id"] == b"2-0"
 print("backlog starts with PEL=0; new deliveries split 1-0/2-0 and advance the group position")

 effects = [first[0][0]]
 assert client.xack("jobs", "workers", "2-0") == 1
 assert client.xlen("jobs") == 3
 group = next(item for item in client.xinfo_groups("jobs") if item["name"] == b"workers")
 assert group["last-delivered-id"] == b"2-0"
 assert client.xreadgroup("workers", "w2", {"jobs": "0-0"}) == [[b"jobs", []]]
 assert client.xreadgroup("workers", "w1", {"jobs": "0-0"})[0][1] == first
 third = client.xreadgroup("workers", "w2", {"jobs": ">"})[0][1]
 assert third[0][0] == b"3-0"
 assert client.xack("jobs", "workers", "3-0") == 1
 assert client.xreadgroup("workers", "w2", {"jobs": ">"}) == []
 print("XACK keeps stream data and read position; 0-0 reads own pending; > does not recover w1")

 cursor, claimed, deleted = client.xautoclaim(
 "jobs", "workers", "w2", min_idle_time=0, start_id="0-0", count=10,
 )
 assert cursor == b"0-0" and claimed == first and deleted == []
 pending = client.xpending_range("jobs", "workers", "-", "+", 10)
 assert len(pending) == 1 and pending[0]["consumer"] == b"w2"
 assert pending[0]["times_delivered"] >= 2
 effects.append(claimed[0][0])
 assert effects.count(b"1-0") == 2
 assert client.xack("jobs", "workers", "1-0") == 1
 assert client.xack("jobs", "workers", "1-0") == 0
 assert client.xpending("jobs", "workers")["pending"] == 0
 assert client.xlen("jobs") == 3
 print("explicit claim changes owner; simulated side effect repeats; ACK clears PEL, not the stream")

 client.xadd("jobs", {"job": "4-0"}, id="4-0")
 assert client.xreadgroup("tail", "observer", {"jobs": ">"})[0][1][0][0] == b"4-0"
 assert client.xreadgroup("workers", "w1", {"jobs": ">"})[0][1][0][0] == b"4-0"
 assert client.xdel("jobs", "4-0") == 1
 assert client.xpending("jobs", "workers")["pending"] == 1
 cursor, claimed, deleted = client.xautoclaim(
 "jobs", "workers", "w2", min_idle_time=0, start_id="0-0", count=10,
 )
 assert cursor == b"0-0" and claimed == [] and deleted == [b"4-0"]
 assert client.xpending("jobs", "workers")["pending"] == 0
 print("a $ group starts after old entries; XDEL loses pending payload; claim reports deleted ID")

The workers group begins with three unread entries and zero pending entries. Two reads distribute 1-0 and 2-0; the group position already reaches 2-0 before acknowledgement. Acknowledging the second entry leaves stream length and that position unchanged. Pending-history reads distinguish the two consumers, and a new-entry read returns 3-0 instead of retrying 1-0.

The effects list models a completed action without its acknowledgement. Explicit claiming moves 1-0 to w2, and processing it again produces two recorded effects. This is a simulation of duplicate business processing, not a transaction-safe implementation. Acknowledgement then clears the PEL while leaving the three stream records available.

The tail group sees only a later 4-0 entry. Deleting that entry while pending makes the subsequent claim return its ID in the deleted list without a payload. The checks exercise command-level state changes, not process-crash detection, automatic retry scheduling, durable deduplication, or failover recovery.

Sources

Redis XGROUP CREATE, group starting positions, checked 2026-09-08.
Redis XREADGROUP, new and pending reads, NOACK, and optional CLAIM behavior, checked 2026-09-08.
Redis XPENDING, pending ownership and delivery counts, checked 2026-09-08.
Redis XACK, acknowledgement and PEL removal, checked 2026-09-08.
Redis XAUTOCLAIM, recovery cursors, idle thresholds, and deleted entries, checked 2026-09-08.
Redis XTRIM, retention and consumer-group reference policies, checked 2026-09-08.
Redis persistence, RDB and AOF durability tradeoffs, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only compilation and allocator selection, checked 2026-09-08.

Redis geo handles locations by distance

Tue, 25 Jan 2028 00:00:00 +0000

Redis geospatial commands index named locations and answer nearby-point queries without returning every coordinate to the application. They are useful when spherical distance is an acceptable model. They do not provide road routes, altitude-aware separation, arbitrary polygon analysis, or a guarantee that every returned location is still current.

Longitude comes before latitude

GEOADD takes longitude, latitude, and member name in that order. Valid longitude runs from -180 to 180 degrees; latitude is restricted to approximately -85.05112878 through 85.05112878 degrees. The polar regions are outside this index’s supported range. Swapping two otherwise valid coordinate values may silently store a completely different location.

The index is a sorted set whose scores encode 52-bit geohashes. Do not overwrite these scores with ordinary ranking values. Members are unique: updating the same member changes its position without adding another entry. GEOADD normally counts newly added members; CH counts additions and changes. ZREM removes a location, and GEOPOS reports the stored coordinates.

Coordinates are encoded with finite precision, so GEOPOS need not reproduce the input decimals exactly. GEODIST computes distance on an idealized spherical Earth, not a straight chord through the planet or a road journey. Its default unit is meters, with kilometers, feet, and miles available. A missing endpoint returns nil; it is not a zero-distance pair.

State the search contract

GEOSEARCH, available since Redis 6.2, searches around an existing member or explicit longitude/latitude. BYRADIUS describes a circle; BYBOX describes an axis-aligned box using width and height in the selected unit. Use this command for new queries instead of the deprecated GEORADIUS family. Match units throughout the request and result interpretation.

ASC orders by increasing distance, and COUNT limits returned matches. For nearest results, use ASC with COUNT and avoid ANY. ANY allows stopping after enough matches are found, which does not guarantee the globally nearest matches within the search area. A tiny count also does not guarantee cheap execution across a densely populated, enormous search region.

FROMMEMBER can return the center member itself. An application asking for other nearby people must account for that result. WITHDIST returns distances in the chosen unit, while WITHCOORD returns coordinates. These answers reflect the index contents at query time, not independent verification that a device or business is actually at that position.

When it breaks

The documented spherical approximation can differ from Earth’s actual shape by up to about 0.5 percent in edge cases. A city-scale query is not automatically accurate to one meter. Establish acceptable error and boundary behavior for the application; surveying, safety-critical proximity, routing, or exact geofencing may require a more appropriate spatial model.

GEOSEARCHSTORE writes a result snapshot into another sorted set. By default it preserves position scores; STOREDIST instead writes distances from the query center as scores. That distance-ranked set must not be reused as a geospatial index. Later source updates or removals do not refresh either stored result automatically.

GEOADD does not attach per-location expiration or freshness timestamps. Key expiration removes the entire index, while individual cleanup requires explicit membership management such as ZREM. A fast query can therefore return stale locations perfectly correctly. Define the update and removal policy separately from the spatial search.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0. The setup builds Redis core with libc and installs its client in a temporary virtual environment, without optional modules or system-wide installation.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. A private Redis Unix socket under /tmp isolates the experiment from existing instances. TCP, scheduled snapshots, and append-only persistence are disabled. The program stops its own process and removes its runtime directory. Only dependency downloads need network access.

import math

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", "noeviction",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


with instance() as client:
 palermo = (13.361389, 38.115556)
 catania = (15.087269, 37.502669)
 assert client.geoadd("places", [
 *palermo, b"Palermo", *catania, b"Catania",
 ]) == 2
 assert client.type("places") == b"zset"
 position = client.geopos("places", "Palermo")[0]
 assert all(abs(actual - expected) < 0.00001
 for actual, expected in zip(position, palermo))
 distance = client.geodist("places", "Palermo", "Catania", unit="km")
 assert math.isclose(distance, 166.2742, abs_tol=0.001)
 assert client.geodist("places", "Palermo", "missing") is None
 print(f"longitude first; decoded coordinates are approximate; distance {distance:.4f} km")

 query = dict(longitude=15, latitude=37, radius=200, unit="km", sort="ASC")
 results = client.geosearch("places", **query, withdist=True)
 assert [item[0] for item in results] == [b"Catania", b"Palermo"]
 assert results[0][1] < results[1][1] < 200
 assert client.geosearch("places", **query, count=1) == [b"Catania"]
 assert client.geosearch(
 "places", member="Palermo", radius=1, unit="km", sort="ASC",
 ) == [b"Palermo"]
 print("ASC COUNT 1 selects Catania; FROMMEMBER also includes the center member")

 assert client.geosearchstore("subset", "places", **query) == 2
 assert client.zscore("subset", "Catania") == client.zscore("places", "Catania")
 assert client.geosearchstore("distances", "places", **query, storedist=True) == 2
 assert math.isclose(client.zscore("distances", "Catania"),
 results[0][1], abs_tol=0.001)
 assert client.zscore("distances", "Catania") != client.zscore("places", "Catania")
 print("default GEOSEARCHSTORE preserves position scores; STOREDIST stores kilometers")

 try:
 client.geoadd("places", [0, 90, b"pole"])
 except redis.ResponseError as error:
 assert "invalid longitude,latitude pair" in str(error)
 else:
 raise AssertionError("latitude 90 must be rejected")
 assert client.zcard("places") == 2
 assert client.geoadd("swapped", [
 *palermo, b"correct", palermo[1], palermo[0], b"reversed",
 ]) == 2
 assert client.geodist("swapped", "correct", "reversed", unit="km") > 1000
 print("polar coordinates are rejected; swapped but valid coordinates are not detected")

 assert client.geoadd("places", [13.4, 38.1, b"Palermo"]) == 0
 assert client.zcard("places") == 2
 assert client.geoadd("places", [13.5, 38.2, b"Palermo"], ch=True) == 1
 assert client.zrem("places", b"Catania") == 1
 assert client.geopos("places", "Catania") == [None]
 assert client.zcard("subset") == 2
 print("updates reuse members; ZREM removes locations; stored subsets are snapshots")

Palermo and Catania reproduce approximately 166.2742 kilometers, with coordinate comparisons allowing encoding precision. The specified search center ranks Catania first, while a member-centered search includes Palermo itself. Stored position scores match the source; STOREDIST scores instead match the returned kilometer distances.

Latitude 90 is rejected without inserting a member, but reversed coordinates within both valid ranges are accepted and describe a distant point. Updating a member preserves cardinality; removing Catania does not modify the previously stored subset. These results establish command semantics and data interpretation, not real-world positional accuracy, route length, stale-location prevention, or production-scale search performance.

Sources

Redis GEOADD, coordinate limits, encoding, updates, and spherical-model error, checked 2026-09-08.
Redis GEOPOS, stored coordinate replies and missing members, checked 2026-09-08.
Redis GEODIST, spherical distance, units, and absent endpoints, checked 2026-09-08.
Redis GEOSEARCH, search shapes, ordering, limits, and ANY semantics, checked 2026-09-08.
Redis GEOSEARCHSTORE, result snapshots and distance-score storage, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only compilation and allocator selection, checked 2026-09-08.

Pipelining folds many Redis round trips into one

Thu, 20 Jan 2028 00:00:00 +0000

Pipelining reduces the waits between sending Redis commands, but its benefit must be measured for a particular workload. A remote synchronous loop may mostly wait for network latency; a local loop may spend more time in the client, scheduling, and socket I/O. Neither case justifies a universal promise of several-fold improvement.

Compare equivalent work

For N commands and a batch size B, a simple synchronous model changes approximately N request-response waits into ceiling(N/B) batch waits. It does not remove command execution, response bytes, encoding, or parsing. One batch can span multiple socket writes and network packets, so this estimate is not a packet count or a measured network RTT.

Keep the command, key distribution, payload, connection count, client version, and persistence settings fixed while changing B. Otherwise a faster result might come from doing less work, removing durability, or adding concurrent clients. Use transaction=False when the question is plain pipelining; redis-py’s default pipeline also includes MULTI/EXEC.

Batch size one below uses a synchronous client loop, not a one-command pipeline. The comparison measures this application’s batching benefit, not Redis’s maximum throughput or server cost alone.

Include the costs the application pays

Time from command preparation through receipt of the final response. Ignoring request construction or response parsing makes large batches look cheaper than they are to the application. The example includes buffering, execution, and result collection in its timer, while resetting the fixture and asserting correctness outside it.

Warm each case, vary test order, repeat, and report variation. Three measured runs here provide a small reproducible comparison, not a rigorous capacity study. A laptop’s scheduler and competing work can still influence the outcome. Do not promote one run’s median into a service-level objective.

Throughput is not individual-request latency. Dividing total time by the number of commands gives an average completion cost, not a p95 or p99 latency distribution. Waiting to fill a batch can delay its first request before execution even begins. This experiment constructs batches immediately and therefore does not measure arrival-time queueing.

When it breaks

A huge batch can improve commands per second while causing unacceptable pauses and memory growth. Bound the number of commands, request and response bytes, and formation delay. Choose those limits from representative measurements rather than automatically using hundreds or thousands of entries for every payload.

If larger batches stop helping, inspect client CPU, response decoding, server command cost, bandwidth, and competing workload. Pipelining can improve achieved server throughput by amortizing I/O, but cannot remove every bottleneck. Sharding is one possible architecture decision, not the inevitable explanation for a flat graph.

Speed measurements must still check failures. Dropping responses or silently accepting errors can produce an impressive but meaningless rate. Network retries are disabled below, and every increment response is checked against the expected sequence. A real disconnect can leave execution uncertain; replaying a batch after the last known response is not automatically safe for increments.

Keep this workload on its private instance. It repeatedly resets and increments a counter, so changing the endpoint to a shared production database would change the operational risk. The local Unix socket also deliberately excludes TCP, TLS, and a remote network path.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0. The setup compiles Redis core with libc and installs the client in a temporary virtual environment, not a system service.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It starts a private Unix-socket Redis under /tmp with TCP and persistence disabled, then stops only its own process and cleans its runtime directory. Downloads require network access. Existing Redis instances are not contacted.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", "noeviction",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


from statistics import median


TOTAL = 2048
SIZES = (1, 8, 64, 256)


def measure(client, size):
 client.set("bench:counter", 0)
 started = time.perf_counter()
 responses = []
 if size == 1:
 for _ in range(TOTAL):
 responses.append(client.incr("bench:counter"))
 else:
 for offset in range(0, TOTAL, size):
 with client.pipeline(transaction=False) as batch:
 for _ in range(min(size, TOTAL - offset)):
 batch.incr("bench:counter")
 responses.extend(batch.execute())
 elapsed = time.perf_counter() - started
 assert responses == list(range(1, TOTAL + 1))
 assert client.get("bench:counter") == str(TOTAL).encode()
 return elapsed


with instance() as client:
 for size in SIZES:
 measure(client, size)
 measurements = {size: [] for size in SIZES}
 for repetition in range(3):
 order = SIZES[repetition:] + SIZES[:repetition]
 for size in order:
 measurements[size].append(measure(client, size))
 for size in SIZES:
 samples = measurements[size]
 duration = median(samples)
 waits = (TOTAL + size - 1) // size
 print(f"batch={size:3d} waits={waits:4d} median_ms={duration * 1000:.2f} "
 f"commands_per_second={TOTAL / duration:.0f} "
 f"range_ms={min(samples) * 1000:.2f}-{max(samples) * 1000:.2f}")
 print("all replies and final counters checked; no speedup threshold is asserted")

Each case performs 2,048 increments of the same counter, reusing a connection. The logical blocking-call counts are 2,048, 256, 32, and eight for sizes one, eight, 64, and 256. Those are application waits, not captured packets. Every trial checks the complete reply sequence and the final stored counter.

One recorded run produced median completion times of 35.24, 9.40, 5.61, and 5.00 milliseconds respectively. The shrinking improvement at larger sizes illustrates diminishing returns for this local workload. The program prints your measurements and their observed ranges; it deliberately has no assertion requiring a speedup.

All four configurations run once as warmup, followed by three measured repetitions with rotated order. Commands per second divides 2,048 by the measured duration. It must not be read as a prediction for a remote application with different concurrency or payloads.

This exercise validates equivalent successful work and measures local batch completion. It does not test production arrival rates, tail latency, TCP/TLS transport, persistence overhead, connection failures, or cluster scaling. Those are separate dimensions of a deployment benchmark.

Sources

Redis pipelining, round-trip amortization and I/O overhead, checked 2026-09-08.
Redis benchmarking guide, workload controls, concurrency, and measurement limitations, checked 2026-09-08.
redis-py 8.1.0 client implementation, nontransactional batches and error handling, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only compilation and allocator selection, checked 2026-09-08.

Redis Lua scripts run without interleaving

Tue, 11 Jan 2028 00:00:00 +0000

A Redis Lua script can read data, make a decision, and update it without another client’s command interleaving between those steps. It does not mean every statement succeeds, that earlier writes are undone after an error, or that the result is automatically durable.

Keep the decision beside the data

Consider deleting a token only if it still belongs to a particular owner. A client-side GET followed later by DEL has a gap: another owner can replace the value before deletion. A script can compare the current value and conditionally delete it during one execution. A stale owner then leaves the replacement intact.

Use redis.call to access Redis data. The sandbox does not provide general filesystem or network access. Pass every accessed key through KEYS and ordinary values through ARGV, rather than constructing key names from stored data. Normal multi-key use in Redis Cluster needs keys in the same hash slot.

Prefer a native atomic command when it already expresses the condition. Redis 8.4 introduced DELEX with IFEQ for conditional string deletion. The compare-and-delete script below teaches branching and response handling; it is not a reason to replace that command with custom code. Deleting an ownership token neither completes a distributed-lock protocol nor protects external resources from stale workers.

When it breaks

An unhandled redis.call error stops the script. A SET executed before a failing LPUSH remains applied, while statements after that failure are not reached. That differs from MULTI/EXEC, where an individual runtime error normally leaves later queued commands running. Neither mechanism supplies automatic rollback of earlier successful writes.

redis.pcall returns a Redis command error to the script as a table with an err field. The script can inspect that field and continue deliberately. Catching the error does not reverse previous writes or make a failed business operation successful. Validate arguments and expected types before the first write where possible, and design the remaining failure path explicitly.

A caller timeout also is not cancellation or evidence that nothing happened. Retrying a script that increments a counter may increment it twice. Keep operations bounded and use an application-level outcome protocol where repeated effects are unacceptable. SCRIPT KILL can interrupt a running script only if it has not yet modified the dataset; it is not a safe undo button for a stuck writer.

Keep source available

EVALSHA refers to a script by digest, not a permanently installed routine. The script cache is volatile and may disappear on restart, failover, or SCRIPT FLUSH. EVAL-loaded scripts can also be evicted under the policy introduced in Redis 7.4. Keep parameterized source in the application and handle a confirmed NOSCRIPT response with the client’s supported loading mechanism.

Do not generate a different script body for every value; use arguments. A confirmed NOSCRIPT means that invocation did not execute the missing script. Reloading or sending EVAL addresses missing code, not an uncertain earlier business outcome.

Bound both loop counts and the amount of data a command processes. A short script containing an unbounded collection read can still monopolize the server. Moving logic into Redis removes client round trips, not the need for latency and memory budgets.

Check it quickly

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It starts Redis on a private Unix socket under /tmp, with TCP and persistence disabled. Cleanup stops only that process and removes its runtime directory. Downloads need network access; no existing Redis instance is contacted.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", "noeviction",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


compare_delete = """
if redis.call('GET', KEYS[1]) == ARGV[1] then
 return redis.call('DEL', KEYS[1])
end
return 0
"""

with instance() as client, client.client() as rival:
 client.set("lease", "owner-a")
 assert client.get("lease") == b"owner-a"
 rival.set("lease", "owner-b")
 assert client.eval(compare_delete, 1, "lease", "owner-a") == 0
 assert client.get("lease") == b"owner-b"
 assert client.eval(compare_delete, 1, "lease", "owner-b") == 1
 assert client.eval(compare_delete, 1, "lease", "owner-b") == 0
 print("compare-and-delete preserves a new owner's token and removes only a match")

 client.set("wrong-type", "string")
 fails_after_write = """
redis.call('SET', KEYS[1], 'before')
redis.call('LPUSH', KEYS[2], 'item')
redis.call('SET', KEYS[3], 'after')
return 1
"""
 try:
 client.eval(fails_after_write, 3, "written", "wrong-type", "not-reached")
 except redis.ResponseError as error:
 assert "WRONGTYPE" in str(error)
 else:
 raise AssertionError("script should fail on the string key")
 assert client.mget("written", "wrong-type", "not-reached") == [
 b"before", b"string", None,
 ]
 print("redis.call error stops the script but keeps the earlier write")

 catches_error = """
local reply = redis.pcall('LPUSH', KEYS[1], 'item')
if type(reply) == 'table' and reply.err then
 redis.call('SET', KEYS[2], 'handled')
 return reply.err
end
return reply
"""
 response = client.eval(catches_error, 2, "wrong-type", "handled-error")
 assert response.startswith(b"WRONGTYPE")
 assert client.get("handled-error") == b"handled"
 assert client.eval(catches_error, 2, "new-list", "unused-marker") == 1
 assert client.lrange("new-list", 0, -1) == [b"item"]
 assert client.get("unused-marker") is None
 print("redis.pcall exposes errors for handling and preserves successful command replies")

 digest = client.script_load(compare_delete)
 assert client.script_exists(digest) == [True]
 client.set("lease", "owner-c")
 assert client.evalsha(digest, 1, "lease", "owner-c") == 1
 client.set("lease", "owner-d")
 assert client.script_flush(sync_type="SYNC") is True
 assert client.script_exists(digest) == [False]
 try:
 client.evalsha(digest, 1, "lease", "owner-d")
 except redis.exceptions.NoScriptError:
 pass
 else:
 raise AssertionError("flushed script remained available")
 assert client.get("lease") == b"owner-d"
 assert client.eval(compare_delete, 1, "lease", "owner-d") == 1
 print("SCRIPT FLUSH causes NOSCRIPT; EVAL with retained source can run it again")

The first case replaces owner-a with owner-b before the stale owner’s script runs. The stale token returns zero and preserves owner-b; the matching token returns one and deletes it. Another call returns zero because the key is absent. The script reads the current value rather than trusting the earlier client-side read.

The next script writes one marker, attempts LPUSH against a string, and never reaches its final marker. Assertions confirm the first write survives the error. The pcall variant handles the same error and writes its handled marker; successful LPUSH instead returns its integer result. Type-checking distinguishes that number from an error table.

Finally, SCRIPT LOAD supplies a digest that EVALSHA can execute. SCRIPT FLUSH removes that code from this private test server. The resulting NOSCRIPT leaves the token intact, and a subsequent EVAL using retained source removes it. Never run this cache-clearing exercise against a shared instance.

Four output summaries follow assertions on results and stored values. The run does not measure throughput, test a long-running script, reproduce a network failure, or establish distributed-lock, cross-shard, or durability guarantees.

Sources

Introduction to Redis Eval scripts, execution isolation and volatile script caching, checked 2026-09-08.
Redis Lua API, sandbox, command calls, and error conversion, checked 2026-09-08.
Redis EVAL, key arguments and eviction of EVAL-loaded scripts, checked 2026-09-08.
Redis SCRIPT KILL, restriction after a script performs writes, checked 2026-09-08.
Redis SCRIPT FLUSH, explicit cache removal, checked 2026-09-08.
Redis DELEX, native conditional deletion since Redis 8.4, checked 2026-09-08.
Redis Cluster specification, hash-slot boundaries for multi-key operations, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only compilation and allocator selection, checked 2026-09-08.

A Redis list and a sorted set solve different problems

Tue, 04 Jan 2028 00:00:00 +0000

A Redis list stores an ordered sequence that permits repeated values. A sorted set stores unique members ordered by numeric scores. The useful distinction is not that one is universally faster or cheaper: it is whether the application needs positional sequence operations, unique membership, score ranges, or a combination requiring a different data model.

Define what makes an entry distinct

With RPUSH, a list keeps values in append order, including duplicates. RPUSH followed by LPOP removes the oldest value; RPUSH followed by RPOP removes the newest. Other choices of ends change that order, so a list is not automatically FIFO regardless of producer commands. Removal order also does not guarantee completion order among parallel workers.

ZADD associates one score with each member. Adding an existing member again updates that score rather than creating another record. Its default return value counts new members, so zero can be a successful update. Repeated event payloads used directly as member names will collapse into one membership, which is wrong when each occurrence represents separate work.

Equal scores are allowed. Ties sort by member bytes, not arrival time or locale-aware text order; REV reverses the tie order as well. A leaderboard may accept that rule, while a timestamp index may need a distinct event identifier and an explicit tie policy. Scores are floating-point numbers, so choose units that preserve the required numeric precision.

Positions and scores select different things

Lists support more than end access. LINDEX retrieves a position and LRANGE returns an inclusive positional range. Reaching an interior position involves traversal; end access is cheap, but a large response still costs work proportional to its size. A short capped list with head or tail reads can be appropriate.

ZRANGE defaults to zero-based rank positions with an inclusive end. BYSCORE interprets the bounds as numeric scores instead; an opening parenthesis excludes a score boundary. The documented range cost is O(log N + M), with N members and M results. Large result sets and deep offset scans are not free just because the structure is indexed.

Rank-based pagination in either changing structure is not a snapshot. Insertions or score updates can shift later pages, producing repeats or omissions across separate requests. Choose the read contract before selecting a container. There is no general guarantee of similar memory usage: representation, cardinality, payload sizes, and configuration all affect the result. Measure representative data when capacity is the deciding factor.

When it breaks

A timestamp score is not a member expiration deadline. ZREMRANGEBYSCORE explicitly removes a numeric interval, while LTRIM explicitly retains a positional window. Neither cleanup happens because time passed or a read occurred. A separate append and cleanup are separate operations, so retention logic needs its own execution and concurrency policy.

A sorted set also is not a complete delayed-work scheduler. ZPOPMIN immediately removes the minimum member, even if its score encodes a future time. Reading due entries and later removing them in a second command can let multiple workers select the same job. Atomic claiming, processing acknowledgement, and recovery of abandoned work are additional requirements, not properties supplied by a timestamp score.

List pops likewise remove values before business processing finishes. A consumer failure after removal can lose the task unless the application uses a suitable processing protocol. Choose lists for sequence operations and sorted sets for score-indexed membership; do not infer delivery guarantees from either ordering scheme.

Check it quickly

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", "noeviction",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


with instance() as client:
 client.rpush("arrival", b"b", b"a", b"b")
 assert client.lrange("arrival", 0, -1) == [b"b", b"a", b"b"]
 assert client.lindex("arrival", 1) == b"a"
 assert client.llen("arrival") == 3
 client.zadd("ranked", {"b": 10})
 client.zadd("ranked", {"a": 10})
 assert client.zadd("ranked", {"b": 20}) == 0
 assert client.zcard("ranked") == 2
 assert client.zrange("ranked", 0, -1, withscores=True) == [(b"a", 10.0), (b"b", 20.0)]
 client.zadd("ranked", {"b": 10})
 assert client.zrange("ranked", 0, -1) == [b"a", b"b"]
 assert client.zrange("ranked", 0, -1, desc=True) == [b"b", b"a"]
 print("lists retain duplicates and positions; sorted sets update unique members and order ties by bytes")

 assert client.lpop("arrival") == b"b"
 assert client.rpop("arrival") == b"b"
 assert client.lrange("arrival", 0, -1) == [b"a"]
 client.rpush("fifo", b"old", b"new")
 client.rpush("lifo", b"old", b"new")
 assert client.lpop("fifo") == b"old"
 assert client.rpop("lifo") == b"new"
 print("RPUSH with LPOP removes oldest first; RPUSH with RPOP removes newest first")

 client.zadd("times", {"at-10": 10, "at-20": 20, "at-30": 30})
 assert client.zrange("times", 0, 1) == [b"at-10", b"at-20"]
 assert client.zrange("times", 10, 20, byscore=True) == [b"at-10", b"at-20"]
 assert client.zrange("times", "(10", 20, byscore=True) == [b"at-20"]
 assert client.zremrangebyscore("times", "-inf", "(20") == 1
 assert client.zrange("times", 0, -1) == [b"at-20", b"at-30"]
 assert client.ttl("times") == -1
 client.rpush("window", b"old", b"middle", b"new")
 client.ltrim("window", -2, -1)
 assert client.lrange("window", 0, -1) == [b"middle", b"new"]
 print("rank selects positions; score selects values; score cleanup and list trimming are explicit")

 now = 100
 client.zadd("schedule", {"future-job": now + 50})
 assert client.zrange("schedule", "-inf", now, byscore=True) == []
 assert client.zpopmin("schedule") == [(b"future-job", 150.0)]
 assert client.exists("schedule") == 0
 print("ZPOPMIN removes the minimum immediately even when its score represents a future deadline")

The list retains b, a, b as three entries and allows an interior LINDEX read. The sorted set contains only two members; updating b returns zero while changing its score. Equal-score ordering becomes a, b regardless of insertion order, and REV reverses it.

Separate FIFO and LIFO examples expose the choice of pop end. The range checks distinguish ranks from scores and inclusive from exclusive bounds. Score-based deletion removes only values below twenty, while a list trim retains its newest two values; the sorted-set key still has no expiration.

Finally, the numeric test clock is one hundred and the only scheduled score is one hundred fifty. The due-range query is empty, yet ZPOPMIN removes that future member immediately. This verifies command semantics, not a scheduler, memory comparison, crash-recovery protocol, or throughput benchmark.

Sources

Redis RPUSH, appending list values, checked 2026-09-08.
Redis LINDEX, positional access and traversal cost, checked 2026-09-08.
Redis LRANGE, inclusive positional ranges and cost, checked 2026-09-08.
Redis LTRIM, retaining a positional window, checked 2026-09-08.
Redis ZADD, unique members, updates, and numeric scores, checked 2026-09-08.
Redis ZRANGE, rank and score queries, bounds, and tie ordering, checked 2026-09-08.
Redis ZPOPMIN, immediate minimum-score removal, checked 2026-09-08.
Redis ZREMRANGEBYSCORE, explicit score-based cleanup, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only compilation and allocator selection, checked 2026-09-08.

Redis expiry is a deadline, eviction is a capacity policy

Sun, 26 Dec 2027 00:00:00 +0000

Expiration and eviction can both turn a successful lookup into a miss, but they answer different questions. Expiration follows a key’s deadline. Eviction removes eligible keys to respond to memory pressure. Extending a TTL cannot prevent capacity-driven eviction, and increasing memory cannot correct a deadline that the application set incorrectly.

Time and capacity are separate axes

A volatile key has an expiration attached to it. Redis checks expired keys during access and through active expiration work. This does not promise that the key survives until its deadline: explicit deletion, replacement, and an eligible eviction policy can change the outcome earlier. A cache client must handle misses regardless of how far away the original expiration appeared.

Maxmemory selects an accounting limit used by Redis’s memory-management decisions. It is not an operating-system RSS ceiling. Replication and append-only buffers can include memory excluded from eviction accounting; process overhead, allocator behavior, and persistence work also need capacity headroom. A large command can temporarily exceed the configured limit. Size the process environment as well as the cache budget.

Select candidates before debating ranking

Allkeys policies consider keys regardless of whether they have TTLs. Volatile policies restrict candidates to keys with expiration. Within those sets, LRU favors keys that have gone longer without access, LFU uses frequency information, and random policies select randomly. Redis approximates LRU and LFU rather than promising one globally exact victim order. Volatile-ttl favors shorter remaining lifetimes, not keys that have necessarily already expired.

Noeviction declines memory-growing commands when the limit cannot be accommodated instead of discarding keys for capacity. Reads can still succeed. It is not a durability guarantee: ordinary expiration, explicit deletion, process failure, and persistence configuration remain separate. Inspect the actual configured policy instead of assuming a universal deployment or cluster default.

A volatile policy without expiring candidates behaves like noeviction for this purpose. Merely choosing volatile-lru does not cause permanent keys to become disposable. Conversely, putting session or authoritative records beside rebuildable cache entries under an allkeys policy makes those records eligible too. A key prefix is a naming convention, not an eviction boundary.

When it breaks

A sudden miss spike can reflect expiration, eviction, invalidation, or changed key construction. Compare expired_keys and evicted_keys deltas with the request workload and command outcomes. These are cumulative counters, so compare a bounded interval and account for restarts or resets. A rising eviction counter alone does not reveal which application feature lost data.

Watch refused writes as well as reads. A warm cache can keep returning plausible values while its writers receive memory errors. Ignoring those errors turns stale data into an application consistency defect. Increasing TTL may then make existing values older without fixing the inability to write replacements. Retry policy must not turn memory pressure into an unlimited request storm.

Separating authoritative state from a disposable cache is an architectural decision, not a command to migrate data during an incident. Verify recoverability, persistence, and ownership before changing an existing instance’s policy. The exercise below changes configuration only inside processes it creates and tests three candidate policies without changing any production keyspace.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0. Build core Redis with libc and install the pinned client locally. The source and package downloads need the network; no system installation or optional bundled-module build is performed.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Execute the Python block with "$redis_lab/venv/bin/python". Each policy gets its own temporary Unix socket, empty dataset, and process with TCP and persistence disabled. The memory limit is the measured initial allocation plus two mebibytes. Bounded writes use 64-KiB values; TTLs are one hour where enabled. Every process and its runtime directory are cleaned up afterward.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance(policy="noeviction"):
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", policy,
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=True,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


for policy, expiring in [("allkeys-lru", True), ("noeviction", True),
 ("volatile-lru", False)]:
 with instance(policy) as client:
 baseline = client.info("memory")["used_memory"]
 client.config_set("maxmemory", baseline + 2 * 1024 * 1024)
 payload = "x" * 65536
 written, rejected = 0, False
 for index in range(128):
 try:
 client.set(f"item:{index}", payload, ex=3600 if expiring else None)
 except redis.exceptions.OutOfMemoryError:
 rejected = True
 break
 written += 1
 stats = client.info("stats")
 remaining = client.dbsize()
 assert stats["expired_keys"] == 0
 if policy == "allkeys-lru":
 assert written == 128 and not rejected
 assert stats["evicted_keys"] > 0 and remaining < written
 assert client.ttl("item:127") > 0
 else:
 assert rejected and 0 < written < 128
 assert stats["evicted_keys"] == 0 and remaining == written
 assert client.get("item:0") == payload
 if not expiring:
 assert client.ttl("item:0") == -1
 print(policy, {"written": written, "remaining": remaining,
 "evicted": stats["evicted_keys"],
 "expired": stats["expired_keys"], "rejected": rejected})

In this run, allkeys-lru accepted 128 writes, retained 23 keys, and reported 105 evictions with zero expirations. The newest surviving key still had a positive TTL. This establishes capacity-driven loss before expiration without claiming an exact LRU victim sequence.

Noeviction accepted 23 writes before rejecting the next allocation; existing reads still worked and no keys were evicted. Volatile-lru with no expiring keys produced the same refusal behavior. Exact counts depend on allocation overhead and the build, so assertions check relationships rather than those particular counts. This is not a throughput benchmark or evidence about replication, failover, persistence peaks, or an operating-system memory limit.

Sources

Redis key eviction, maxmemory accounting, candidate policies, and approximate ranking, checked 2026-09-08.
Redis EXPIRE, per-key deadlines and expiration behavior, checked 2026-09-08.
Redis INFO, memory accounting and eviction/expiration counters, checked 2026-09-08.
Redis 8.10.1 build instructions, isolated core build and allocator selection, checked 2026-09-08.

Redis Cluster splits keys across slots

Fri, 26 Nov 2027 00:00:00 +0000

Redis Cluster separates key placement from machine identity. A key maps to one of 16,384 fixed hash slots; cluster metadata maps those slots to primaries and their replicas. Adding a machine does not change the slot hash function. Operators redistribute slots, while clients discover the current topology and react to routing changes.

Keys choose slots, not contiguous machine ranges

The ordinary mapping is CRC16/XMODEM of the key modulo 16,384. A nonempty hash tag between the first opening brace and its following closing brace makes Redis hash that substring instead. Thus profile:{42} and quota:{42} share a slot even though their full names differ. Tags are placement instructions, not authorization boundaries.

A primary can own several noncontiguous slot ranges. In a stable topology each slot has an authoritative primary, and a whole key belongs to that slot: a large hash or sorted set is not automatically divided between nodes. CLUSTER SHARDS reports shard membership and slot ranges. Do not confuse “same primary” with “same slot”; they are different conditions.

Clients participate in routing

A cluster-aware client maintains a slot map and connects to the advertised nodes. Contacting the wrong primary can produce MOVED with a destination, not a missing value. MOVED tells the client to update routing; ASK is a temporary migration redirection that requires ASKING before the redirected command, without permanently assigning ownership to that target.

Migration is not an instant, globally synchronized map update. Clients may observe different topology information, and some multi-key operations can return TRYAGAIN while keys are split during migration. Use a maintained cluster client rather than assuming a single endpoint forwards every request. Advertised addresses must be reachable from the application’s network, not only between Redis processes.

Placement does not imply transaction scope

Server-side multi-key commands such as MSET require their keys to share one slot. Merely landing on the same physical primary is insufficient. Hash tags can satisfy the slot constraint, but putting every tenant under one tag concentrates load and limits distribution. A client operation implemented as several independent commands is not equivalent to one atomic multi-key command.

Sharding distributes ownership; replication creates copies for a shard. Replica reads can be stale because replication is asynchronous. Failover promotes a replica for the affected shard, not the whole cluster as one indivisible unit. Acknowledged writes can still be lost during failures, and connection loss can leave a write’s outcome unknown to its caller.

When it breaks

Missing slot coverage and insufficient reachable primaries can prevent service. With cluster-require-full-coverage yes, an unavailable shard can stop key commands even on otherwise healthy primaries. Three primaries without replicas demonstrate distribution, but cannot replace a failed primary’s data with a replica that does not exist.

Watch slot coverage, cluster state, replica health, address announcements, and client redirection errors together. Adding nodes alone does not automatically move existing keys or split a hot key. Resharding, replica placement, and client retry semantics are separate operational decisions. Test the failure you intend to tolerate instead of treating a successful cluster handshake as a high-availability result.

Check it quickly

Use a POSIX machine with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Redis 8.10.1, redis-py 8.1.0, and Python 3.14.4. Build Redis core with libc and install the client in a temporary virtual environment.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python". It starts three isolated primaries using temporary directories and six loopback-only ports for data and cluster-bus traffic. Private Unix sockets identify the created processes before configuration. Persistence is disabled. The final failure targets only its own first primary; cleanup stops the remaining processes and removes their temporary files.

from contextlib import contextmanager, ExitStack
import os
from pathlib import Path
import signal
import socket
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.crc import key_slot
from redis.exceptions import ClusterCrossSlotError, ClusterDownError, MovedError
from redis.retry import Retry


def until(check):
 deadline = time.monotonic() + 15
 while not check():
 assert time.monotonic() < deadline, "cluster state did not converge"
 time.sleep(0.05)


@contextmanager
def node(directory, port, bus):
 directory.mkdir()
 endpoint = str(directory / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--bind", "127.0.0.1", "--port", str(port),
 "--protected-mode", "yes", "--unixsocket", endpoint,
 "--unixsocketperm", "700", "--dir", str(directory),
 "--save", "", "--appendonly", "no", "--cluster-enabled", "yes",
 "--cluster-config-file", "nodes.conf", "--cluster-port", str(bus),
 "--cluster-announce-ip", "127.0.0.1", "--cluster-node-timeout", "1000",
 "--cluster-require-full-coverage", "yes",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False, protocol=2,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 assert time.monotonic() < deadline, "Redis did not start"
 time.sleep(0.02)
 assert client.info("server")["process_id"] == process.pid
 yield client, process
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode in (0, -signal.SIGKILL), errors.decode()


with TemporaryDirectory(prefix="redis-cluster-", dir="/tmp") as directory, ExitStack() as stack:
 leases = [stack.enter_context(socket.socket()) for _ in range(6)]
 for lease in leases:
 lease.bind(("127.0.0.1", 0))
 ports = [lease.getsockname()[1] for lease in leases]
 nodes = []
 for index in range(3):
 leases[2 * index].close()
 leases[2 * index + 1].close()
 nodes.append(stack.enter_context(node(
 Path(directory) / str(index), ports[2 * index], ports[2 * index + 1],
 )))
 clients = [item[0] for item in nodes]
 ranges = [[(0, 4095), (8192, 12287)], [(4096, 8191)], [(12288, 16383)]]
 for client, owned in zip(clients, ranges):
 bounds = [value for interval in owned for value in interval]
 assert client.execute_command("CLUSTER", "ADDSLOTSRANGE", *bounds) == b"OK"
 for index in (1, 2):
 assert clients[0].execute_command(
 "CLUSTER", "MEET", "127.0.0.1", ports[2 * index], ports[2 * index + 1],
 ) == b"OK"
 until(lambda: all(b"cluster_state:ok" in client.execute_command("CLUSTER", "INFO")
 for client in clients))
 shards = [dict(zip(item[::2], item[1::2], strict=True))
 for item in clients[0].execute_command("CLUSTER", "SHARDS")]
 assert len(shards) == 3
 assert sorted(interval for shard in shards
 for interval in zip(shard[b"slots"][::2], shard[b"slots"][1::2])) == [
 (0, 4095), (4096, 8191), (8192, 12287), (12288, 16383),
 ]
 first_id = clients[0].execute_command("CLUSTER", "MYID")
 first_shard = next(shard for shard in shards
 if first_id in [dict(zip(item[::2], item[1::2]))[b"id"]
 for item in shard[b"nodes"]])
 assert first_shard[b"slots"] == [0, 4095, 8192, 12287]
 print("16384 slots are covered; one primary owns two noncontiguous ranges")

 keys = [b"profile:{42}", b"quota:{42}"]
 slot = key_slot(keys[0])
 assert key_slot(keys[1]) == slot
 assert clients[0].execute_command("CLUSTER", "KEYSLOT", keys[0]) == slot
 owner = next(index for index, owned in enumerate(ranges)
 if any(start <= slot <= end for start, end in owned))
 wrong = clients[(owner + 1) % 3]
 try:
 wrong.get(keys[0])
 except MovedError as error:
 assert error.slot_id == slot
 assert error.node_addr == ("127.0.0.1", ports[2 * owner])
 else:
 raise AssertionError("a plain client on the wrong primary must receive MOVED")

 routed = redis.RedisCluster(host="127.0.0.1", port=ports[0],
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 assert routed.mset({keys[0]: "Ada", keys[1]: "10"})
 assert routed.mget(keys) == [b"Ada", b"10"]
 print("MOVED names the owner; a cluster client routes a same-slot MSET and MGET")
 other = b"unrelated:2"
 assert key_slot(other) != slot
 assert any(start <= key_slot(other) <= end for start, end in ranges[owner])
 try:
 clients[owner].execute_command("MSET", keys[0], "bad", other, "bad")
 except ClusterCrossSlotError:
 pass
 else:
 raise AssertionError("cross-slot MSET must fail")
 assert routed.get(keys[0]) == b"Ada"
 assert routed.get(other) is None
 print("cross-slot MSET fails even on the same primary, without changing either key")
 finally:
 routed.close()

 assert nodes[0][1].poll() is None
 nodes[0][1].kill()
 assert nodes[0][1].wait(timeout=5) == -signal.SIGKILL
 until(lambda: all(b"cluster_state:fail" in client.execute_command("CLUSTER", "INFO")
 for client in clients[1:]))
 try:
 clients[1].get(b"any-key")
 except ClusterDownError:
 print("without replicas, a missing primary stops key service under full coverage")
 else:
 raise AssertionError("an uncovered cluster must reject key commands")

CLUSTER SHARDS confirms complete coverage and two disjoint ranges on one primary. A plain client receives MOVED, while the cluster client reads and writes the tagged pair. A second key on that same primary but in another slot makes MSET fail without changing either value. Killing a primary then produces cluster failure and rejected key commands.

The fixture uses RESP2 for directly inspected topology replies and explicit local bus ports. It is a local routing and failure-detection test, not a production topology: there are no replicas or independent machines. ASK migration, actual resharding, replica promotion, network partitions, persistence, TLS, authentication, and production throughput are not exercised here.

Sources

Redis Cluster specification, slot hashing, hash tags, routing, migration, and write safety, checked 2026-09-08.
Redis Cluster scaling, sharding, replication, and deployment boundaries, checked 2026-09-08.
Redis CLUSTER SHARDS, shard membership and noncontiguous slot ranges, checked 2026-09-08.
Redis CLUSTER ADDSLOTSRANGE, assigning slot ranges in an empty fixture, checked 2026-09-08.
redis-py 8.1.0 cluster client, routing and client configuration, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only build and allocator selection, checked 2026-09-08.

Redis append-only log trades speed for durability

Wed, 17 Nov 2027 00:00:00 +0000

An append-only file records Redis mutations for recovery, but choosing AOF is not the same as promising that no acknowledged write can ever disappear. Durability depends on the fsync policy, effective configuration, storage behavior, and which failure you mean. Start with a recovery objective rather than the reassuring word “log.”

Execution, logging, and acknowledgment

Redis executes commands against its in-memory dataset and then propagates their effects to AOF. This is not a rule that each command must reach disk before memory changes. With appendfsync always, Redis synchronizes appended data before sending replies; commands processed together may share a write and fsync. A lost connection still leaves the client uncertain about an operation whose reply never arrived.

The file is a recovery representation, not a permanent audit trail of every original request. Redis can transform commands for persistence, and rewriting retains the current dataset rather than its entire history. A deleted value is not guaranteed to remain recoverable from a rewritten log.

Choose the synchronization contract

appendfsync everysec schedules synchronization approximately once per second. Its usual tradeoff is roughly one second of exposed writes, not a universal loss bound under arbitrary storage stalls. appendfsync no leaves flushing to the operating system; it means Redis does not request fsync, not that data never reaches disk. Neither policy makes every successful reply evidence of durable storage.

appendfsync always offers a stronger local acknowledgment contract at a latency and throughput cost, but it cannot protect against losing the disk, a faulty storage stack, or deleting the wrong data. Also inspect no-appendfsync-on-rewrite: enabling it suspends normal fsync during background persistence work, weakening durability even when the main policy says always.

Rewriting creates a recoverable file set

Since Redis 7, AOF uses a manifest plus a base file and incremental files. With aof-use-rdb-preamble enabled, the base uses RDB format; later mutations live in incremental AOF files. That base belongs to AOF and is not the independently scheduled dump.rdb snapshot.

BGREWRITEAOF starts or schedules background work; its successful reply does not certify completion. Inspect INFO persistence for the completed rewrite and its status. Copying one conveniently named file, or racing a rewrite while copying the directory, can produce an unusable backup. Preserve a consistent manifest and its referenced active files.

When it breaks

Disk errors, exhausted space, and rewrite failures require monitoring, not merely an appendonly yes setting. Check aof_last_write_status and aof_last_bgrewrite_status alongside disk capacity, rewrite state, and application errors. Background work still consumes I/O and copy-on-write memory; “background” does not mean zero latency impact.

Truncated tails and corruption in the middle have different recovery semantics. Preserve the original artifact before inspection or repair: a repair can discard subsequent commands. AOF also replays application mistakes, so maintain independently retained backups. Do not turn a process-restart demonstration into a claim about power-loss safety, replication acknowledgment, or exactly-once retries.

Check it quickly

Use a POSIX machine with a C compiler, make, curl, tar, and Python 3.10 or newer. This exercise ran on macOS with Redis 8.10.1, redis-py 8.1.0, and Python 3.14.4. The setup builds Redis core with libc and installs the client in a temporary virtual environment.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It creates a private Unix-socket server, without TCP, scheduled snapshots, or automatic AOF rewrites. AOF uses always and keeps fsync enabled during rewriting. Only the subprocess it creates is killed. Persistence files survive that process restart inside the temporary directory, which is removed after the checks.

from contextlib import contextmanager
import os
from pathlib import Path
import signal
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance(directory, aof):
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--dir", directory, "--dbfilename", "dump.rdb", "--save", "",
 "--appendonly", "yes" if aof else "no", "--appendfsync", "always",
 "--appenddirname", "appendonlydir", "--appendfilename", "appendonly.aof",
 "--aof-use-rdb-preamble", "yes", "--no-appendfsync-on-rewrite", "no",
 "--auto-aof-rewrite-percentage", "0",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client, process
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode in (0, -signal.SIGKILL), errors.decode()


def crash(process):
 assert process.poll() is None
 process.kill()
 assert process.wait(timeout=5) == -signal.SIGKILL


with TemporaryDirectory(prefix="redis-aof-", dir="/tmp") as directory:
 with instance(directory, aof=True) as (client, process):
 client.set("counter", 0)
 for expected in range(1, 5):
 assert client.incr("counter") == expected
 assert client.hset("profile", mapping={"name": "Ada", "level": "2"}) == 2
 client.set("deleted", "not retained")
 assert client.delete("deleted") == 1
 rewrites = client.info("persistence")["aof_rewrites"]
 assert client.bgrewriteaof()
 deadline = time.monotonic() + 5
 while True:
 state = client.info("persistence")
 if (state["aof_rewrites"] > rewrites
 and not state["aof_rewrite_in_progress"]
 and not state["aof_rewrite_scheduled"]):
 assert state["aof_last_bgrewrite_status"] == "ok"
 break
 assert time.monotonic() < deadline, "AOF rewrite did not finish"
 time.sleep(0.02)
 assert client.incr("counter") == 5
 assert client.info("persistence")["aof_last_write_status"] == "ok"
 assert not (Path(directory) / "dump.rdb").exists()
 print("rewrite completed; a later INCR returned 5; no standalone RDB")
 crash(process)

 aof_dir = Path(directory) / "appendonlydir"
 entries = []
 for line in (aof_dir / "appendonly.aof.manifest").read_text().splitlines():
 parts = line.split()
 entries.append(dict(zip(parts[::2], parts[1::2], strict=True)))
 active = [entry for entry in entries if entry["type"] in {"b", "i"}]
 assert {entry["type"] for entry in active} == {"b", "i"}
 for entry in active:
 assert (aof_dir / entry["file"]).is_file()
 base = next(aof_dir / entry["file"] for entry in active if entry["type"] == "b")
 assert base.read_bytes().startswith(b"REDIS")
 print("manifest references an RDB-format AOF base and an incremental AOF")

 with instance(directory, aof=True) as (client, process):
 assert client.get("counter") == b"5"
 assert client.hgetall("profile") == {b"name": b"Ada", b"level": b"2"}
 assert client.get("deleted") is None
 assert client.dbsize() == 2
 print("after SIGKILL and restart: counter=5, hash restored, deletion retained")

The test waits for an observed successful rewrite, then adds one more increment. After SIGKILL, the manifest references an RDB-format AOF base and incremental data, while no standalone dump.rdb exists. Restarting restores counter five, both hash fields, and the absence of the deleted key. Checking values and types is stronger than counting keys alone.

The manifest inspection is scoped to this fixture’s simple filenames; it is not a general backup parser. SIGKILL skips graceful shutdown but leaves the operating system running, including its filesystem cache. The result proves this local AOF recovery path, not a measured loss window for everysec or no, nor survival of hardware failure.

Sources

Redis persistence, synchronization, multipart AOF, recovery, and backup boundaries, checked 2026-09-08.
Redis BGREWRITEAOF, scheduled work versus completed rewriting, checked 2026-09-08.
Redis INFO, persistence status fields, checked 2026-09-08.
Redis 8.10.1 configuration, fsync policies, rewrite overrides, and AOF file layout, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only build and allocator selection, checked 2026-09-08.

MULTI EXEC runs a batch of commands as one

Thu, 22 Jul 2027 00:00:00 +0000

MULTI/EXEC prevents another client’s commands from interleaving while the queued commands execute. That alone does not protect a value read earlier in the application. If two buyers read the same stock count and later write their own calculated value, wrapping each SET in a transaction can still lose an update.

Watch before reading the dependency

WATCH makes EXEC conditional on specified keys remaining unmodified since they were watched. Start the watch before reading the values used in the decision. If a watched key changes before execution, Redis rejects the whole queue and returns a null reply. Rejection leaves the rival’s change intact.

The watch and transaction belong to one connection. A redis-py pipeline reserves that connection after watch; reads before multi execute immediately, so the application can inspect them. After multi, commands are buffered for execution. A context manager releases the reserved connection and resets pipeline state when the attempt finishes.

Watch every key whose value or absence affects the decision. Watching stock while ignoring a separately stored limit does not protect that limit. Even SET with identical bytes can invalidate a watch. Creating a previously absent watched key also invalidates it. Expiration and eviction count as changes in the tested Redis version.

Retry the decision, not its old output

After a confirmed conflict, start a new attempt: establish watches, reread the dependencies, recheck the business condition, and calculate the new writes. Reusing the old quantity merely repeats the stale decision. EXEC clears watches whether it executes or aborts; UNWATCH can cancel watching when the application decides not to proceed.

Bound retries and expose exhaustion. Frequent conflicts can make a hot key expensive, even when no data is lost. Backoff may reduce contention but does not repair an incorrect dependency set. If a native atomic command expresses the operation, use it; the stock arithmetic here is deliberately split to demonstrate the stale-read problem.

Successful WATCH validation does not turn a transaction into a rollback system. Individual runtime errors inside EXEC can leave other successful commands applied. Validate the relevant data model, inspect responses, and avoid treating a transaction containing several unrelated side effects as an all-or-nothing business unit.

When it breaks

Watching after the read leaves a race before the watch began. Watching one connection and committing on another protects nothing about that second connection’s queue. Sending an email or making a payment during the retry loop can repeat an external effect even when Redis later rejects the attempt.

Distinguish a server-confirmed conflict from a transport problem. In redis-py 8.1.0, WatchError can also be raised when a connection fails while watching. That exception class alone does not prove EXEC never ran. A lost reply after submission requires outcome reconciliation or an idempotent business protocol, not an unconditional retry of an irreversible action.

The example deliberately creates conflicts through a second connection and checks the pinned client’s conflict message. That is a test assertion, not a general production error classifier. Network retries are disabled for this experiment. In Redis Cluster, ordinary multi-key transactions also require the involved keys to share a hash slot; WATCH does not add a cross-shard transaction coordinator.

Check it quickly

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", "noeviction",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


with instance() as client, client.client() as rival:
 client.set("stock", 2)
 stale = int(client.get("stock"))
 assert rival.decr("stock") == 1
 with client.pipeline(transaction=True) as transaction:
 transaction.set("stock", stale - 1)
 assert transaction.execute() == [True]
 assert client.get("stock") == b"1"
 print("a stale value written inside MULTI/EXEC still loses another decrement")

 client.set("stock", 2)
 conflicts = 0
 for attempt in range(2):
 with client.pipeline(transaction=True) as transaction:
 transaction.watch("stock")
 current = int(transaction.get("stock"))
 assert current > 0
 if attempt == 0:
 assert rival.decr("stock") == 1
 transaction.multi()
 transaction.set("stock", current - 1)
 transaction.set("reservation", "accepted")
 try:
 responses = transaction.execute()
 except redis.WatchError as error:
 assert str(error) == "Watched variable changed."
 conflicts += 1
 assert client.get("stock") == b"1"
 assert client.get("reservation") is None
 continue
 assert responses == [True, True]
 break
 else:
 raise AssertionError("both attempts conflicted")
 assert conflicts == 1 and attempt == 1
 assert client.get("stock") == b"0"
 assert client.get("reservation") == b"accepted"
 print("WATCH rejects the stale queue; the second attempt rereads 1 and commits 0")

 client.set("watched", "unchanged")
 with client.pipeline(transaction=True) as transaction:
 transaction.watch("watched")
 assert transaction.get("watched") == b"unchanged"
 rival.set("watched", "unchanged")
 transaction.multi()
 transaction.set("same-value-marker", "yes")
 try:
 transaction.execute()
 except redis.WatchError as error:
 assert str(error) == "Watched variable changed."
 else:
 raise AssertionError("same-value SET did not invalidate WATCH")
 assert client.get("same-value-marker") is None
 print("WATCH detects a SET even when the resulting bytes are unchanged")

 with client.pipeline(transaction=True) as transaction:
 transaction.watch("not-yet-created")
 assert transaction.get("not-yet-created") is None
 rival.set("not-yet-created", "now-present")
 transaction.multi()
 transaction.set("absent-marker", "yes")
 try:
 transaction.execute()
 except redis.WatchError as error:
 assert str(error) == "Watched variable changed."
 else:
 raise AssertionError("creating a watched key did not invalidate WATCH")
 assert client.get("absent-marker") is None
 print("WATCH also protects a read of an absent key against later creation")

The first case reads stock as two, lets another connection decrement it to one, and writes the stale calculation inside MULTI/EXEC. Stock incorrectly remains one after two intended decrements. This deterministic ordering demonstrates that non-interleaving execution cannot repair a stale read taken outside the transaction.

The second case watches before reading. A deliberate competing decrement rejects the first queue, leaving the reservation marker absent and the rival’s stock change intact. A second attempt watches again, reads one, then commits zero with the marker. Both attempts are bounded and every resulting value is asserted.

The last cases show two less obvious conflicts: SET with identical bytes, and creating a watched key that was absent. Each prevents its queued marker from appearing. Four printed summaries follow checked responses and final state, rather than relying on timing to provoke a race.

This run tests optimistic conflict handling on an isolated server. It does not establish fair scheduling, exactly-once reservations, network-failure recovery, persistence, or cluster failover. A production reservation also needs its own business identifier and a defined response to an unknown commit outcome.

Sources

Redis transactions, optimistic concurrency and transaction failure semantics, checked 2026-09-08.
Redis WATCH, conditional execution based on watched keys, checked 2026-09-08.
Redis EXEC, null replies and execution results, checked 2026-09-08.
Redis UNWATCH, clearing connection-scoped watches, checked 2026-09-08.
redis-py 8.1.0 client implementation, immediate reads, connection pinning, and WatchError paths, checked 2026-09-08.
Redis Cluster specification, hash-slot boundaries for multi-key operations, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only compilation and allocator selection, checked 2026-09-08.

Redis streams track a consumer's own cursor

Tue, 13 Jul 2027 00:00:00 +0000

A Redis stream does not automatically track a separate cursor for every connection. Ordinary XREAD uses a position supplied by the application. A consumer group instead owns a server-side last-delivered position plus pending delivery records. Confusing those two forms of progress can make a restart skip unfinished work or replay effects already completed.

The group advances on delivery

XGROUP CREATE with 0-0 starts before retained history; $ starts after the current end. Each group maintains independent delivery state over the same stream records. Billing can have delivered two entries while analytics has delivered only one. Within a group, named consumers share new work rather than each receiving every entry.

For ordinary XREADGROUP without CLAIM or NOACK, > requests entries beyond the group’s delivery position and adds deliveries to its Pending Entries List, or PEL. That position advances before processing and acknowledgement. XACK removes specified pending IDs from one group, not stream records, and does not advance the group’s last-delivered ID.

Acknowledgements can arrive out of order. Finishing the second entry says nothing about whether the first finished. The group can have reached the end of the stream while older deliveries remain pending. Do not reconstruct successful processing from the largest delivered or acknowledged ID without an application contract that actually guarantees contiguous completion.

Pending history belongs to a consumer

Using 0-0 with XREADGROUP requests this named consumer’s pending history, not an unrestricted replay of the stream. A newly named worker sees none of another worker’s pending entries. Restarting under a different name and reading only > therefore does not recover abandoned deliveries. It can make the service look active while older work never resumes.

Recovery requires reading the original consumer’s pending history or explicitly transferring eligible work, for example with XAUTOCLAIM. Its idle threshold must account for legitimate processing time. Current Redis also supports optional XREADGROUP CLAIM; this exercise does not enable it. Ownership transfer cannot prevent a late worker from repeating an external action, so processing still needs an appropriate deduplication policy.

Use XINFO GROUPS to compare last-delivered-id, pending, and lag. Lag estimates entries awaiting delivery from the group’s current position, not unfinished business actions. It can be unavailable after certain position changes or deletions; a null metric is not zero. A group with zero lag and nonzero pending work has reached the delivery end but still has outstanding acknowledgements.

When it breaks

XGROUP SETID is a deliberate change to the group’s delivery position, not a harmless acknowledgement repair. Rewinding after completed work makes retained, previously acknowledged entries eligible for new delivery again. Moving forward can skip unread entries. It does not automatically empty existing PEL records, and neither direction changes another group’s position.

Plan a repositioning operation with workers coordinated and business effects understood. Replaying into a separate group isolates delivery bookkeeping, but not external side effects if it calls the same handler. An acknowledgement is not a permanent tombstone that forbids all future reads of that record. Avoid using a rewind as a substitute for diagnosing the pending entries.

Independent group positions do not provide independent retention. Deleting or trimming a shared payload affects every group that might need it. Default KEEPREF trimming may leave pending IDs without data. Durability across server failure is also separate from the cursor mechanism. A recorded delivery position is neither proof that the payload survives nor proof that its business work completed.

Check it quickly

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", "noeviction",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


def groups(client):
 return {group["name"]: group for group in client.xinfo_groups("events")}


with instance() as client:
 for entry_id in ("1-0", "2-0", "3-0"):
 client.xadd("events", {"event": entry_id}, id=entry_id)
 for name in ("billing", "analytics"):
 client.xgroup_create("events", name, id="0-0")
 billing = client.xreadgroup("billing", "b1", {"events": ">"}, count=2)[0][1]
 analytics = client.xreadgroup("analytics", "a1", {"events": ">"}, count=1)[0][1]
 assert [entry[0] for entry in billing] == [b"1-0", b"2-0"]
 assert [entry[0] for entry in analytics] == [b"1-0"]
 state = groups(client)
 assert state[b"billing"]["last-delivered-id"] == b"2-0"
 assert state[b"analytics"]["last-delivered-id"] == b"1-0"
 assert state[b"billing"]["pending"] == 2 and state[b"analytics"]["pending"] == 1
 assert state[b"billing"]["lag"] == 1 and state[b"analytics"]["lag"] == 2
 print("two groups read overlapping records but retain independent positions and PELs")

 assert client.xack("events", "billing", "2-0") == 1
 state = groups(client)
 assert state[b"billing"]["last-delivered-id"] == b"2-0"
 assert state[b"billing"]["pending"] == 1 and state[b"analytics"]["pending"] == 1
 assert client.xreadgroup("billing", "b2", {"events": "0-0"}) == [[b"events", []]]
 own = client.xreadgroup("billing", "b1", {"events": "0-0"})[0][1]
 assert [entry[0] for entry in own] == [b"1-0"]
 next_entry = client.xreadgroup("billing", "b2", {"events": ">"})[0][1]
 assert [entry[0] for entry in next_entry] == [b"3-0"]
 assert groups(client)[b"billing"]["lag"] == 0
 assert groups(client)[b"billing"]["pending"] == 2
 assert client.xack("events", "billing", "1-0", "3-0") == 2
 assert groups(client)[b"billing"]["pending"] == 0
 assert client.xpending("events", "analytics")["pending"] == 1
 assert client.xlen("events") == 3
 print("ACK affects only one group's PEL; a new consumer sees no other owner's pending history")

 client.xgroup_setid("events", "billing", "0-0")
 replay = client.xreadgroup("billing", "b3", {"events": ">"}, count=1)[0][1]
 assert [entry[0] for entry in replay] == [b"1-0"]
 assert groups(client)[b"analytics"]["last-delivered-id"] == b"1-0"
 assert client.xack("events", "billing", "1-0") == 1
 client.xgroup_setid("events", "analytics", "$")
 state = groups(client)
 assert state[b"analytics"]["last-delivered-id"] == b"3-0"
 assert state[b"analytics"]["pending"] == 1
 assert state[b"analytics"]["lag"] == 0
 assert client.xreadgroup("analytics", "a1", {"events": ">"}) == []
 own = client.xreadgroup("analytics", "a1", {"events": "0-0"})[0][1]
 assert [entry[0] for entry in own] == [b"1-0"]
 assert client.xack("events", "analytics", "1-0") == 1
 print("SETID rewinds acknowledged data for replay; moving forward skips unread data but keeps PEL")

Billing reads two entries and analytics reads one, producing distinct delivery positions and pending counts. Acknowledging billing’s second entry changes only its PEL. Consumer b2 cannot read b1’s pending history, but it can receive the next new entry. After all new billing entries are delivered, lag is zero while two deliveries still await acknowledgement.

Once billing work is acknowledged, explicitly rewinding its position lets b3 receive 1-0 again. Analytics remains independent. Moving analytics to $ skips its unread entries but preserves the pending 1-0 record, which its original consumer can still read and acknowledge.

All state comparisons use actual Redis replies; no timing assumptions or concurrent workers are needed. The experiment verifies group isolation, delivery positions, pending ownership, and deliberate repositioning. It does not implement crash detection, recovery scheduling, persistence testing, or transaction-safe business processing.

Sources

Redis XREAD, positions supplied by independent readers, checked 2026-09-08.
Redis XREADGROUP, group delivery positions and consumer pending history, checked 2026-09-08.
Redis XACK, group-specific pending acknowledgement, checked 2026-09-08.
Redis XINFO GROUPS, last-delivered ID, pending count, and lag limitations, checked 2026-09-08.
Redis XGROUP SETID, deliberate repositioning of a group, checked 2026-09-08.
Redis XAUTOCLAIM, explicit recovery of eligible pending entries, checked 2026-09-08.
Redis XTRIM, payload retention and consumer-group references, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only compilation and allocator selection, checked 2026-09-08.

Redis TTL lets every key carry its own clock

Tue, 29 Jun 2027 00:00:00 +0000

Redis expiration answers when a key should stop being available, not how much memory the server may consume. A deadline belongs to the key, and later commands can preserve, replace, or remove it. Reliable TTL use therefore requires testing update paths as well as the initial insertion. A successful write alone says nothing about the deadline remaining afterward.

A deadline is not a sliding timer

EXPIRE accepts seconds; PEXPIRE accepts milliseconds. Redis stores expiration as an absolute Unix timestamp, so elapsed time does not pause while the server is stopped. Stable system clocks matter, especially when restoring data on another machine. This is not a monotonic-duration promise independent of wall-clock changes. The exercise never changes the system clock.

GET does not refresh expiration. In-place changes such as INCR preserve the key’s existing timeout, whereas an ordinary successful SET replaces the value and discards the old timeout. SET with KEEPTTL preserves it. Explicitly issuing another expiration command changes the deadline again. Applications wanting sliding sessions must deliberately implement that policy rather than assuming each read extends access.

Set the value and its initial lifetime together with SET EX or SET PX when both belong to one operation. Separate SET and EXPIRE calls leave a failure window in which the value exists without expiration. A client crash between them can create a persistent entry. This is a write-protocol defect, not a slow expiration cycle.

Inspect the actual state

TTL returns remaining seconds, with -1 for an existing key without expiration and -2 for a missing key. PTTL offers millisecond resolution. PEXPIRETIME reports the absolute deadline, which makes preserving a deadline easier to test than comparing two countdown values that legitimately decrease between commands. A TTL result is a snapshot, not a reservation that guarantees the key survives until then.

EXPIRE NX only adds a deadline when none exists. GT only accepts a later deadline; a persistent key is treated as having an infinite lifetime for that comparison. PERSIST explicitly removes expiration. Check these commands’ results rather than treating every accepted command as a successful state change. A non-positive relative expiry deletes the key immediately; it is not another spelling of persistence.

When it breaks

Passive expiration checks deadlines during access, while active expiration removes expired keys that are not accessed. Delayed physical cleanup is not permission for an ordinary primary GET to keep serving an expired value. Nor does removing an entry guarantee that its allocator pages immediately return to the operating system. Visibility, data-structure cleanup, and process RSS are different observations.

A TTL is not a hard memory budget. Arrival rate, value sizes, unexpired data, command overhead, and allocator behavior still determine consumption. Eviction can also remove a key before its deadline. Do not use cache presence as the only business authorization check, or infer replication and failover guarantees from a single-instance experiment. Define the relevant deadline and consistency boundary for those workflows separately.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0. These commands build Redis core using libc and install only the client into a temporary virtual environment. Nothing is installed system-wide; optional bundled modules are not built.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It starts a disposable Redis over a private Unix socket beneath /tmp, with TCP, snapshot scheduling, and append-only persistence disabled. Requests cannot reach a default localhost Redis. The program terminates its own process and removes only its temporary runtime directory. Downloads require network access; the experiment does not.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance(policy="noeviction"):
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", policy,
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=True,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


with instance() as client:
 assert client.ttl("missing") == -2
 client.set("session", "A", px=60000)
 deadline = client.pexpiretime("session")
 assert client.get("session") == "A"
 assert client.pexpiretime("session") == deadline
 client.set("session", "B", keepttl=True)
 assert client.pexpiretime("session") == deadline
 print("GET and SET KEEPTTL preserve the absolute deadline")

 client.set("session", "C")
 assert client.ttl("session") == -1
 assert client.expire("session", 60, nx=True)
 assert not client.expire("session", 120, nx=True)
 assert client.expire("session", 120, gt=True)
 assert not client.expire("session", 30, gt=True)
 assert client.persist("session")
 assert client.ttl("session") == -1
 print("plain SET clears TTL; NX, GT and PERSIST follow their conditions")

 client.set("counter", "1", px=60000)
 deadline = client.pexpiretime("counter")
 assert client.incr("counter") == 2
 assert client.pexpiretime("counter") == deadline
 assert client.expire("counter", 0)
 assert client.get("counter") is None
 print("INCR preserves TTL; EXPIRE zero deletes immediately")

 client.pexpire("session", 100)
 limit = time.monotonic() + 3
 while client.get("session") is not None:
 assert time.monotonic() < limit, "key did not expire"
 time.sleep(0.01)
 assert client.ttl("session") == -2
 print("after expiry: GET is None and TTL is -2")

The absolute deadline survives GET and SET KEEPTTL, but plain SET leaves TTL -1. The conditional expiry assertions distinguish successful updates from rejected ones, and PERSIST returns the key to the persistent state. INCR retains its original deadline; EXPIRE zero removes that separate counter immediately.

Finally, a 100-millisecond timeout leads to GET returning None and TTL -2. A bounded polling loop observes actual disappearance without demanding an exact scheduler latency. These results verify command semantics and ordinary expired reads, not memory reclamation timing, active-cycle throughput, replication behavior, failover, or secure deletion of previously copied data.

Sources

Redis EXPIRE, deadline updates, conditional options, clocks, and active/passive expiration, checked 2026-09-08.
Redis SET, atomic value/lifetime assignment and KEEPTTL, checked 2026-09-08.
Redis TTL, remaining lifetime and the -1/-2 results, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only compilation and allocator selection, checked 2026-09-08.

Two ways Redis keeps data safe

Tue, 15 Jun 2027 00:00:00 +0000

RDB and AOF preserve different representations of Redis state. Choosing between them starts with two questions: how much recent data may be lost, and how long recovery may take? A persistence setting alone answers neither. You need a recoverable artifact, an understood acknowledgment policy, and a restore procedure tested against actual values.

A snapshot captures a point, not an interval guarantee

An RDB file is a compact point-in-time dataset. BGSAVE forks a child that writes a temporary file and replaces the previous snapshot when complete. The parent continues serving requests, but forking, copy-on-write memory, and disk work still have costs. SAVE instead writes synchronously and blocks other clients; reserve it here for the tiny isolated exercise.

A rule such as save 60 1000 requires both elapsed time and a change threshold. It does not promise a new snapshot every sixty seconds regardless of activity, nor successful completion on schedule. Recovery exposure follows the age of a usable snapshot, including missed or failed saves. Monitor successful saves and their results, not just the configured interval.

AOF changes the recovery representation

AOF records mutations and synchronizes them according to appendfsync. everysec normally exposes roughly the most recent second, while no delegates flushing to the operating system. always synchronizes before replies under its effective configuration, at additional cost. These policies do not protect against losing the storage device or guarantee that a promoted replica contains every locally persisted write.

Modern AOF uses a manifest, a base, and incremental files. Its base can be RDB-formatted, so “AOF always replays a huge textual history” is misleading. Rewriting removes unnecessary historical operations. File size, startup time, rewrite overhead, and latency must be measured for your dataset rather than inferred from the format’s name.

Running both is not an automatic merge

With ordinary startup settings and AOF enabled, Redis loads AOF rather than independently loading dump.rdb and replaying a separate log on top. The RDB-formatted base inside AOF is part of that file set. An independently retained RDB remains useful for backup and deliberate recovery, but it does not magically fill holes in incomplete AOF artifacts.

Do not switch an existing RDB deployment by merely editing appendonly and restarting. Follow Redis’s documented live conversion procedure, wait for successful initial AOF creation, and persist the configuration. The two cases below start with separate empty directories; they are not a migration recipe for an existing database.

When it breaks

A successful BGSAVE request is not a successful completed snapshot. Failed storage writes, unavailable fork resources, or inadequate disk space can leave the usable recovery point old. Inspect INFO persistence, error logs, and resource headroom. Background persistence can also worsen tail latency even while Redis remains responsive.

Neither file format is an independent backup if its only copy shares the server’s failure domain. Retain consistent copies outside that machine and test their restore. AOF can faithfully persist an accidental deletion; snapshots can already contain the same mistake. Retention history and application validation matter as much as a boot that accepts connections.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Redis 8.10.1, redis-py 8.1.0, and Python 3.14.4. Build only Redis core with libc and install the client in the temporary virtual environment.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block using "$redis_lab/venv/bin/python" in the same shell. Each case creates its own Unix-socket server, with no TCP listener or scheduled snapshots. Both explicitly save a tiny snapshot; only the second enables AOF with always and no fsync suppression during rewriting. SIGKILL targets only the created subprocess. Its files stay available for restart, then the temporary directory is removed.

from contextlib import contextmanager
import os
from pathlib import Path
import signal
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance(directory, aof):
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--dir", directory, "--dbfilename", "dump.rdb", "--save", "",
 "--appendonly", "yes" if aof else "no", "--appendfsync", "always",
 "--appenddirname", "appendonlydir", "--appendfilename", "appendonly.aof",
 "--aof-use-rdb-preamble", "yes", "--no-appendfsync-on-rewrite", "no",
 "--auto-aof-rewrite-percentage", "0",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client, process
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode in (0, -signal.SIGKILL), errors.decode()


def crash(process):
 assert process.poll() is None
 process.kill()
 assert process.wait(timeout=5) == -signal.SIGKILL


for aof in (False, True):
 with TemporaryDirectory(prefix="redis-restore-", dir="/tmp") as directory:
 with instance(directory, aof) as (client, process):
 assert client.set("version", "snapshot")
 assert client.incr("counter") == 1
 assert client.save()
 snapshot = (Path(directory) / "dump.rdb").read_bytes()
 assert snapshot.startswith(b"REDIS")
 assert client.set("version", "after-snapshot")
 assert client.set("later", "present")
 assert client.incr("counter") == 2
 if aof:
 assert client.info("persistence")["aof_last_write_status"] == "ok"
 crash(process)

 assert (Path(directory) / "dump.rdb").read_bytes() == snapshot
 with instance(directory, aof) as (client, process):
 expected = ([b"after-snapshot", b"present", b"2"] if aof
 else [b"snapshot", None, b"1"])
 assert client.mget("version", "later", "counter") == expected
 assert client.dbsize() == (3 if aof else 2)
 assert (Path(directory) / "dump.rdb").read_bytes() == snapshot
 print("RDB + AOF" if aof else "RDB only", expected)
print("an unchanged standalone RDB does not override enabled AOF recovery")

RDB-only recovers version snapshot, no later key, and counter one. The AOF case recovers version after-snapshot, the later key, and counter two. In both cases, the standalone RDB bytes remain unchanged across the crash and checked restart. The counter is deliberately incremented without an initial SET: replaying both increments over the snapshot would incorrectly produce three.

These assertions compare recovered state, not just whether Redis starts. SIGKILL bypasses graceful shutdown but does not crash the operating system or discard its cache. The experiment establishes snapshot boundaries and ordinary AOF startup selection, not power-loss survival, backup transport integrity, replica failover behavior, or production recovery time. Define those separate acceptance tests before making a stronger promise.

Sources

Redis persistence, snapshots, AOF startup selection, live conversion, and backups, checked 2026-09-08.
Redis SAVE, synchronous snapshot behavior and blocking cost, checked 2026-09-08.
Redis BGSAVE, background saving and request acceptance, checked 2026-09-08.
Redis INFO, persistence progress and result fields, checked 2026-09-08.
Redis 8.10.1 configuration, snapshot triggers and AOF policy, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only build and allocator selection, checked 2026-09-08.

Redis memory fills with fragmentation over time

Thu, 03 Jun 2027 00:00:00 +0000

A Redis process can keep a substantial resident footprint after application keys are deleted. That observation deserves explanation, but it does not by itself prove a leak or allocator fragmentation. The useful question is which bytes remain allocated, which pages remain resident, and whether that footprint threatens the actual capacity budget.

Separate the measurements

INFO memory exposes used_memory for Redis-accounted allocations and used_memory_rss for the resident process footprint observed by the operating system. Neither is simply the sum of application payload lengths. Keys, objects, metadata, clients, and other overhead consume memory too. MEMORY USAGE provides per-key allocation estimates, including overhead; it does not measure that key’s independent contribution to process RSS.

Mem_fragmentation_ratio broadly compares resident memory with Redis-accounted allocations. Code, shared libraries, stacks, and allocator retention also contribute to the difference. The ratio can rise when the denominator falls after deletion, even if RSS barely changes. Inspect absolute bytes alongside ratios: a small empty instance can have a high ratio without consuming a dangerous amount of RAM.

Used_memory_peak helps explain a footprint left after an earlier workload peak. Used_memory_dataset is another accounting view, not a promise that an empty keyspace leaves zero process allocations. Measurements are not all sampled at exactly the instant INFO runs. Do not require an exact equality between a rounded ratio and independently observed values during active writes.

Check the allocator and version

Allocator-specific metrics can help distinguish fragmentation from retained pages and non-heap overhead, but their meaning depends on the build. Redis 8.10.1’s source computes allocator fragmentation from small-bin counters with a Lua adjustment; blindly reconstructing every version’s metric as active bytes divided by allocated bytes is unsafe. Check the versioned implementation when a general documentation formula disagrees.

This exercise deliberately builds with libc. That build lacks jemalloc’s detailed accounting; Redis substitutes some allocator fields from other measurements, and the fragmentation counter remains zero. An allocator_frag_ratio of one is therefore not proof that this process has no fragmentation. Always record mem_allocator before interpreting the more specialized fields or choosing an intervention.

When it breaks

A rising RSS curve alongside a growing dataset calls for workload and capacity analysis, not an automatic fragmentation diagnosis. A falling dataset with stable RSS suggests retained footprint, but still does not identify every retained page. Likewise, one snapshot cannot rule out a leak. Compare the same workload over time, value-size distribution, peak usage, and relevant process limits.

MEMORY PURGE attempts allocator reclamation where implemented. Redis documents it as a benign no-op for allocators other than jemalloc, so a successful reply is not evidence of recovered memory. It also does not remove application keys. Active defragmentation is a different mechanism and requires supported builds; enabling it can add CPU and latency costs that need measurement.

Restarting merely to lower a ratio can introduce downtime, cold-cache load, or data loss when persistence and recovery are inadequate. Establish a measured capacity problem and a rehearsed recovery path before treating restart as remediation. Maxmemory is not an RSS ceiling, and reducing key count does not guarantee immediate page return to the operating system.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1 using libc, and redis-py 8.1.0. The commands build Redis core and install the client in a temporary virtual environment. Downloads require the network; nothing is installed system-wide and optional bundled modules are not built.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python". It creates a dedicated Redis process and private Unix socket under /tmp, disables TCP and persistence, and writes only its own 320 sample keys. Alternating values contain 16 or 64 KiB. Short pauses permit metric sampling, without assuming exact refresh timing. The process and temporary runtime directory are removed afterward.

from contextlib import contextmanager
import json
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance(policy="noeviction"):
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", policy,
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=True,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


with instance() as client:
 def snapshot(stage):
 time.sleep(0.2)
 info = client.info("memory")
 fields = ["used_memory", "used_memory_dataset", "used_memory_rss",
 "used_memory_peak", "mem_fragmentation_ratio", "mem_allocator",
 "allocator_frag_ratio", "allocator_frag_bytes"]
 print(stage, json.dumps({"keys": client.dbsize(),
 **{name: info[name] for name in fields}}, sort_keys=True))
 return info

 empty = snapshot("empty")
 for index in range(320):
 client.set(f"sample:{index}", "x" * (16384 if index % 2 == 0 else 65536))
 assert client.dbsize() == 320
 assert client.memory_usage("sample:0") >= 16384
 loaded = snapshot("loaded")
 assert loaded["used_memory"] > empty["used_memory"] + 10 * 1024 * 1024
 for index in range(1, 320, 2):
 assert client.delete(f"sample:{index}") == 1
 assert client.dbsize() == 160
 half = snapshot("large values deleted")
 assert half["used_memory"] < loaded["used_memory"] - 8 * 1024 * 1024
 for index in range(0, 320, 2):
 assert client.delete(f"sample:{index}") == 1
 assert client.dbsize() == 0
 assert client.memory_usage("sample:0") is None
 cleared = snapshot("all values deleted")
 assert cleared["used_memory"] < half["used_memory"]
 assert cleared["used_memory_peak"] >= loaded["used_memory"]
 assert client.memory_purge()
 snapshot("after MEMORY PURGE")
 print("PASS: logical allocations fell; no RSS or purge-effect threshold is assumed")

One run reported about 16.7 MiB of used_memory after loading. Deleting all keys reduced that to about 1.04 MiB, while RSS remained around 20.8 MiB. The overall ratio consequently rose to roughly twenty. MEMORY USAGE confirmed the named key no longer existed; logical allocations fell, but resident memory did not track them proportionally.

The allocator-specific fragmentation ratio stayed at one throughout this libc run, and MEMORY PURGE returned success without a meaningful RSS drop. Counts and logical-allocation relationships are asserted; exact RSS values are intentionally observations, not pass thresholds. This experiment demonstrates accounting differences and the no-op limitation, not jemalloc defragmentation effectiveness, Linux memory-pressure behavior, or a leak diagnosis.

Sources

Redis INFO, process, allocation, peak, and allocator memory fields, checked 2026-09-08.
Redis MEMORY USAGE, per-key allocation estimates and overhead, checked 2026-09-08.
Redis MEMORY PURGE, jemalloc support and other allocators’ no-op behavior, checked 2026-09-08.
Redis 8.10.1 object accounting, version-specific fragmentation calculation, checked 2026-09-08.
Redis 8.10.1 memory sampling, cached measurements and substituted allocator fields, checked 2026-09-08.
Redis 8.10.1 build instructions, core build and allocator selection, checked 2026-09-08.

Redis keys read well with clear naming

Tue, 01 Jun 2027 00:00:00 +0000

Redis keys are binary-safe names in a flat keyspace, not filesystem paths or schema declarations. Colons are a useful convention for separating ownership, environment, entity, and identifier. A name such as cart:prod:string:42 can explain intent, but Redis does not interpret those labels as a type check or access restriction.

Make the convention unambiguous

Agree on field order and identifier encoding across writers, readers, maintenance jobs, and migrations. Distinct prefixes separate names, but identical encoded names still collide. A convention helps people cooperate; it does not create an independently managed namespace.

A type label is documentation. SET can create a string under a key whose name contains hash, and HSET will then report WRONGTYPE. Use TYPE to inspect data, and keep serialization and type expectations in the application contract. Names cannot prove value types.

Prefer short, readable names, not cryptic abbreviations or entire documents embedded in keys. Names consume memory and travel in command arguments, but object overhead and payloads also matter. A naming change is not evidence of faster queries without measurement. Encode identifiers so separator characters, glob metacharacters, and cluster braces cannot acquire unintended meaning.

Match patterns without inventing deletion semantics

SCAN MATCH accepts a glob pattern. Start with cursor zero, reuse each returned cursor, and continue until zero returns again. COUNT is a work hint, not an exact result count or memory ceiling. A call may return no matches while the cursor remains nonzero. Each call still performs server work; incremental iteration does not mean zero blocking cost.

A full scan returns matching keys present throughout it, but it may return duplicates. Keys added or removed during the scan have weaker guarantees. Deduplicate when necessary or make processing safe to repeat. This is not a snapshot, and an empty page is not the end of the collection.

DEL has different semantics: it receives exact key names, not patterns. Sending an argument ending in an asterisk does not delete matching siblings. It only deletes a key literally containing that asterisk if one exists. Shell expansion is another layer, so quote arguments when using command-line tools; do not mistake shell behavior for Redis pattern matching.

When it breaks

A maintenance script that assumes DEL expands a prefix may remove nothing and falsely report cleanup. The opposite mistake is collecting a broad SCAN result and deleting every match without checking ownership. Review exact targets and the mutation policy. A key can be recreated between discovery and deletion, so a previously reviewed name alone may not identify the same logical object later.

Prefixes are not security boundaries. Restrict access with appropriate ACL key patterns and command permissions, or with separate deployment boundaries when required. Key-pattern permissions do not constrain commands that operate on an entire database without key arguments. A visible service prefix also does not reserve memory or isolate eviction.

Redis Cluster hash tags are an intentional exception to otherwise conventional naming. A valid brace-delimited hash tag selects the bytes used for slot calculation. Different service prefixes can therefore share a slot when their tags match. This can support related multi-key work, but putting every key under one constant tag can concentrate load instead of distributing it.

Check it quickly

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", "noeviction",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


with instance() as client:
 client.set("cart:prod:string:42", "cart-value")
 client.set("billing:prod:string:42", "billing-value")
 assert client.get("cart:prod:string:42") == b"cart-value"
 assert client.get("billing:prod:string:42") == b"billing-value"
 client.set("cart:prod:hash:42", "not-a-hash")
 assert client.type("cart:prod:hash:42") == b"string"
 try:
 client.hset("cart:prod:hash:42", "item", "value")
 except redis.ResponseError as error:
 assert "WRONGTYPE" in str(error)
 else:
 raise AssertionError("a type label unexpectedly enforced a schema")
 print("namespaces separate exact names; a hash label does not create a hash value")

 prefix = "cart:prod:string:"
 expected = {f"{prefix}{number}".encode() for number in range(24)}
 for key in expected:
 client.set(key, "value")
 expected.add(b"cart:prod:string:42")
 cursor, found, pages = 0, set(), 0
 while True:
 cursor, keys = client.scan(cursor=cursor, match=prefix + "*", count=3)
 assert all(key.startswith(prefix.encode()) for key in keys)
 found.update(keys)
 pages += 1
 assert pages < 1000, "unexpected nontermination in this bounded test"
 if cursor == 0:
 break
 assert found == expected
 print("SCAN reaches cursor zero and finds the stable fixture, with duplicates deduplicated")

 assert client.delete(prefix + "*") == 0
 assert all(client.exists(key) for key in expected)
 client.set(prefix + "*", "literal-star")
 assert client.delete(prefix + "*") == 1
 assert all(client.exists(key) for key in expected)
 print("DEL treats the asterisk literally; it does not expand a key pattern")

 from redis.crc import key_slot

 assert key_slot(b"cart:{customer-42}:items") == key_slot(b"billing:{customer-42}:total")
 assert key_slot(b"cart:{customer-42}:items") == key_slot(b"customer-42")
 print("matching hash tags select the same slot across different service prefixes")

The first case stores different values under cart and billing prefixes, then writes a string under a name labeled hash. TYPE reports string and HSET fails, proving that readable labels do not enforce a schema. These are separate exact names, not nested directories.

A stable fixture contains 25 matching string keys. The scan loop accumulates them until cursor zero, tolerating page size variation and deduplicating results. Its bounded test guard detects unexpected nontermination for this small fixture; it is not a universal production scan limit or evidence about concurrently changing keys.

Deleting the literal prefix-plus-asterisk first returns zero while all 25 keys remain. Creating exactly that literal name makes the next deletion return one, still leaving the siblings intact. No wildcard cleanup or shared data deletion is performed.

Finally, the pinned client’s key_slot helper confirms that two differently prefixed names with customer-42 in braces compute the same slot. This is a client-side calculation, not a live Redis Cluster test. Four summaries follow assertions; the exercise does not validate ACL enforcement, concurrent deletion safety, cluster routing, or memory savings.

Sources

Redis keys and values, binary-safe keys and naming conventions, checked 2026-09-08.
Redis SCAN, cursor guarantees, duplicates, and COUNT hints, checked 2026-09-08.
Redis DEL, deleting exact key arguments, checked 2026-09-08.
Redis TYPE, inspecting the actual value type, checked 2026-09-08.
Redis ACL documentation, key patterns and separate command permissions, checked 2026-09-08.
Redis Cluster specification, hash tags and slot placement, checked 2026-09-08.
redis-py 8.1.0 key-slot calculation, client-side hashing used in the exercise, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only compilation and allocator selection, checked 2026-09-08.

A read-through cache loads on a miss by itself

Wed, 26 May 2027 00:00:00 +0000

Read-through caching gives callers one reading interface: a hit returns a stored value, while a miss invokes a loader and stores its result. The important boundary is who owns that loader. Moving it behind the cache interface can remove duplicated lookup logic, but does not automatically solve concurrency, writes, authorization, or consistency between independent copies.

Identify the loading layer

In cache-aside, application code explicitly checks the cache, queries the source on a miss, and fills the cache. With read-through, the caching component owns that miss path. A library can provide this interface inside an application, or a service can provide it remotely. Those deployment choices still determine which processes need updating when loader behavior changes.

A Redis GET alone is not a read-through loader. For a missing string key it returns nil; it does not discover and query your database. A client library, caching service, or application adapter must supply the loading behavior. The exercise below uses a memoizing function wrapper as that interface, not Redis, and keeps the source as an in-memory fixture.

Loading does not define writing

Read-through says nothing about where writes go. Write-through is a separate arrangement, and even a shared read/write component must define what happens when source updates and cache updates fail independently. Writers that bypass it can leave cached values unchanged. Identify all real update paths instead of assuming the pattern makes every writer participate.

TTL restricts reuse of a particular cached entry; it does not guarantee that a later reload obtains current truth from a lagging replica or buggy loader. Nor does expiration necessarily remove bytes from memory immediately. Choose lifetimes from actual data requirements. A short TTL on user-specific data is not permission isolation, and a high hit ratio cannot establish correctness.

When it breaks

An absent record and an unavailable source are different results. Caching a legitimate absence can reduce repeated misses, but can also hide a record created afterward until expiry or invalidation. Converting a timeout into an empty successful result can therefore create a poisoned entry. Decide which outcomes are cacheable and how callers distinguish absence from failure.

Concurrent loading is another independent contract. Cachetools documents that a lock protects cache access while the wrapped function executes outside it; a lock alone does not suppress duplicate loading. Its condition option addresses same-key concurrent execution. The single-threaded exercise deliberately uses neither and must not be treated as a thread-safe or distributed implementation.

Invalidation can race with a load already reading an older source version. Removing a key does not inherently prevent that pending result from being inserted later. Select coordination appropriate to the real writer and loader protocol, and test their ordering. Avoid broad key-pattern deletion as a default response: it creates cold entries without proving that the underlying defect is fixed.

Check it quickly

Use Python 3.10 or newer; this ran with Python 3.14.4 and cachetools 7.1.8. Install the pinned dependency, then run the Python block with "$readthrough_lab/venv/bin/python". The library implements storage, expiry, and memoization. A manually advanced test clock makes the deadline deterministic without sleeping or changing system time. No Redis server or external database is contacted.

readthrough_lab=$(mktemp -d)
python3 -m venv "$readthrough_lab/venv"
"$readthrough_lab/venv/bin/python" -m pip install cachetools==7.1.8

from collections import Counter

from cachetools import TTLCache, cached

clock = [0]
cache = TTLCache(maxsize=8, ttl=10, timer=lambda: clock[0])
source = {"tea": 100}
failures = set()
loads = Counter()


@cached(cache=cache, info=True)
def price(sku):
 loads[sku] += 1
 if sku in failures:
 raise OSError("source unavailable")
 return source.get(sku)


assert price("tea") == price("tea") == 100
assert loads["tea"] == 1
source["tea"] = 120
clock[0] = 9
assert price("tea") == 100 and loads["tea"] == 1
clock[0] = 10
assert price("tea") == 120 and loads["tea"] == 2
print("hit does not extend TTL: old value at t=9; reload at t=10")

source["tea"] = 130
cache.pop(price.cache_key("tea"))
assert price("tea") == 130 and loads["tea"] == 3
print("explicit eviction: next read loads the updated source")

assert price("coffee") is None
source["coffee"] = 200
assert price("coffee") is None and loads["coffee"] == 1
cache.pop(price.cache_key("coffee"))
assert price("coffee") == 200 and loads["coffee"] == 2
print("None is cached: newly created data needs expiry or invalidation")

cache.pop(price.cache_key("tea"))
failures.add("tea")
for _ in range(2):
 try:
 price("tea")
 except OSError:
 pass
 else:
 raise AssertionError("failure was hidden")
assert loads["tea"] == 5
assert price.cache_key("tea") not in cache
failures.remove("tea")
assert price("tea") == price("tea") == 130
assert loads["tea"] == 6
print("exceptions are not cached: two failed loads, then one recovery load")
print(price.cache_info())

The first two reads execute one loader. Changing the source does not change the still-live cached price: it remains 100 at time nine. At time ten the entry expires, and the wrapper loads 120. This checks expiration after insertion, not sliding expiration after access. Explicitly removing the wrapper’s actual cache key then exposes the next source update, 130.

The missing coffee price is cached as None. Creating the source record alone does not replace that cached absence; explicit removal allows a subsequent load of 200. Two raised source errors each invoke the loader and leave no cached tea entry. After recovery, one successful load supplies two reads. The reported totals are four hits and eight misses, including failed loads.

These assertions prove this wrapper’s observed miss, expiry, absence, and error behavior. They do not prove atomic writes, concurrent miss suppression, eviction under memory pressure, or distributed consistency. For a real service, record loader counts and outcomes alongside cache statistics; two faster HTTP requests or global Redis hit counters cannot identify which application loader actually ran.

Sources

Cachetools documentation, TTLCache, cached, synchronization options, and exception handling, checked 2026-09-07.
Redis GET, missing keys return nil rather than loading another data source, checked 2026-09-07.
AWS Builders’ Library: caching challenges and strategies, cold caches, negative caching, and source-load risks, checked 2026-09-07.

A Redis stream keeps events in order for consumers

Thu, 06 May 2027 00:00:00 +0000

A Redis stream stores entries under ordered IDs so readers can revisit retained history. Unlike Pub/Sub, the reader need not be online when the producer appends. That difference provides replay, not unconditional durability: deleted entries, retention limits, and persistence settings still determine what remains available after delay or failure.

Separate entry order from event time

XADD with * generates an ID containing a millisecond component and a sequence component. IDs increase within one stream, including when Redis must compensate for a backward clock movement. Explicit IDs must also advance beyond the stream’s last generated ID. Compare their numeric components, not arbitrary string order.

The ID orders insertion into that stream. It does not establish a global order across independent streams or prove when an external event happened. Store the business timestamp separately when needed. Two ordinary XADD calls with identical fields still create separate entries: automatic IDs identify records, not duplicate business operations.

Save the last processed position

XRANGE includes both boundaries by default. Starting it at the last ID already processed returns that entry again; an opening parenthesis makes a range boundary exclusive. XREAD instead returns entries strictly greater than the supplied ID. With COUNT, save the final returned ID and continue from that position.

Ordinary XREAD does not maintain a server-side offset for each reader or remove delivered entries. Independent readers can read the same range, and repeating a request repeats retained data. The application owns its checkpoint. Saving it before a side effect risks skipping work after failure; saving it afterward risks repeating completed work. Coordinate checkpointing with idempotent processing where that distinction matters.

A dollar ID means start after the stream’s current end when Redis handles this call. It is useful for entering a live feed, not for resuming a saved position. Reusing $ after every response can skip entries appended between calls. After receiving data, continue with the actual last ID. Consumer groups add separate delivery and acknowledgement state; XACK is not an offset commit for ordinary XREAD.

When it breaks

A reader can fall behind retention. XTRIM MAXLEN removes the oldest, lowest-ID entries and keeps newer ones; it does not trim the newest end. Exact trimming enforces the requested count, while approximate trimming may retain extra entries for efficiency. redis-py’s xtrim defaults to approximate trimming, so the exercise explicitly disables it.

XDEL removes selected records. Neither deletion nor a short retention limit becomes safe merely because a consumer has not acknowledged an entry. Default KEEPREF trimming can leave pending references without payloads. Redis 8.2 added explicit reference policies, including ACKED, but retention remains an application decision. A saved ID cannot reconstruct deleted data.

Streams are Redis data, subject to RDB/AOF configuration, replication behavior, key deletion, and possible eviction. A successful append alone does not establish lossless failover or exactly-once processing. Size retention for expected consumer outages and catch-up capacity, monitor lag, and test recovery using the actual persistence configuration. This experiment deliberately disables persistence and makes no restart-survival claim.

Check it quickly

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It creates a private Unix socket under /tmp, disables TCP, scheduled snapshots, and append-only persistence, then terminates only its own process and removes its runtime directory. Downloads require network access. No default local Redis is contacted.

from concurrent.futures import ThreadPoolExecutor

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", "noeviction",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


def ids(entries):
 return [entry[0] for entry in entries]


with instance() as client:
 for entry_id in ("1-0", "1-1", "2-0"):
 assert client.xadd("events", {"value": entry_id}, id=entry_id) == entry_id.encode()
 try:
 client.xadd("events", {"value": "duplicate"}, id="1-1")
 except redis.ResponseError as error:
 assert "equal or smaller" in str(error)
 else:
 raise AssertionError("XADD must reject an ID below the stream's last ID")
 generated = [client.xadd("auto", {"value": "same"}) for _ in range(2)]
 assert tuple(map(int, generated[0].split(b"-"))) < tuple(map(int, generated[1].split(b"-")))
 assert client.xlen("auto") == 2
 print("explicit IDs advance; duplicate/lower ID rejected; equal payloads get distinct auto IDs")

 assert ids(client.xrange("events", "1-1", "+")) == [b"1-1", b"2-0"]
 assert ids(client.xrange("events", "(1-1", "+")) == [b"2-0"]
 first = client.xread({"events": "0-0"}, count=2)
 second = client.xread({"events": "0-0"}, count=2)
 assert first == second and ids(first[0][1]) == [b"1-0", b"1-1"]
 saved = first[0][1][-1][0]
 remaining = client.xread({"events": saved})
 assert ids(remaining[0][1]) == [b"2-0"]
 saved = remaining[0][1][-1][0]
 print("XRANGE includes its start; XREAD excludes the saved ID; independent reads replay")

 client.xadd("events", {"value": "between-calls"}, id="3-0")
 assert client.xread({"events": "$"}, block=100) == []
 assert ids(client.xread({"events": saved})[0][1]) == [b"3-0"]

 def read_live():
 with client.client() as connection:
 connection.client_setname("stream-lab-reader")
 return connection.xread({"events": "$"}, block=1000)

 with ThreadPoolExecutor(max_workers=1) as worker:
 future = worker.submit(read_live)
 deadline = time.monotonic() + 1
 while not any(
 entry["name"] == "stream-lab-reader" and "b" in entry["flags"]
 for entry in client.client_list()
 ):
 assert time.monotonic() < deadline, "reader never blocked"
 time.sleep(0.01)
 client.xadd("events", {"value": "live"}, id="4-0")
 assert ids(future.result(timeout=2)[0][1]) == [b"4-0"]
 print("$ skips existing entries; a saved ID catches up; observed blocking receives a new entry")

 assert client.xdel("events", "2-0") == 1
 assert client.xrange("events", "2-0", "2-0") == []
 assert client.xtrim("events", maxlen=2, approximate=False) == 2
 assert ids(client.xrange("events")) == [b"3-0", b"4-0"]
 assert ids(client.xread({"events": "0-0"})[0][1]) == [b"3-0", b"4-0"]
 print("XDEL removes a record; exact MAXLEN keeps the newest two; old IDs cannot restore deleted data")

Explicit IDs produce a deterministic three-entry log. An older ID fails, while two automatic appends with the same payload produce increasing distinct IDs. The boundary assertions show XRANGE including 1-1 and XREAD continuing strictly after it. Two independent reads from zero return identical data without consuming it.

An entry added before a $ call is skipped and that call times out; reading from the saved ID finds it. For live delivery, the producer waits until CLIENT LIST reports the named reader as blocked before appending 4-0. The example asserts the received record, not exact scheduler timing.

Finally, XDEL removes 2-0 and exact MAXLEN keeps only 3-0 and 4-0. Reading from zero now returns those retained records alone. These results demonstrate ID order, cursor boundaries, live waiting, and retention loss. They do not validate crash durability, consumer-group recovery, cross-stream ordering, or production throughput.

Sources

Redis XADD, entry IDs and monotonic ordering, checked 2026-09-08.
Redis XREAD, exclusive saved positions, blocking, and the dollar ID, checked 2026-09-08.
Redis XRANGE, inclusive and exclusive range bounds, checked 2026-09-08.
Redis XTRIM, retention limits and pending-reference policies, checked 2026-09-08.
Redis XDEL, explicit removal of retained entries, checked 2026-09-08.
Redis persistence, RDB and AOF durability tradeoffs, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only compilation and allocator selection, checked 2026-09-08.

Redis bitmaps flip bits instead of storing numbers

Thu, 29 Apr 2027 00:00:00 +0000

A Redis bitmap is a string interpreted as individual bits, not a separate key type. It is useful for dense yes/no membership such as daily activity. Ten million contiguous flags need 1,250,000 payload bytes, but that estimate excludes keys, object metadata, allocation overhead, and any mapping from business identifiers to bit positions.

Define the offset contract

SETBIT writes zero or one at a zero-based offset and returns the previous bit. Setting an already-set bit again does not increase membership. Offset zero is the most significant bit of the first byte; offset eight starts the second byte. Reading the same key with GET returns its underlying bytes, which need not be valid text.

GETBIT returns zero for a missing key or an offset beyond the current string without expanding it. Zero therefore does not distinguish an explicitly inactive user from an unrecorded one. Decide whether those states may share a representation. SETBIT and GETBIT require a string value and reject a hash rather than converting its fields into bits; BITOP instead writes a new string at its destination.

Memory follows the highest written offset, not the number of ones. Writing offset n requires at least floor(n / 8) + 1 payload bytes, even when the requested value is zero. Clearing that bit later does not shorten the string. Sparse external identifiers can therefore create a huge allocation for almost no membership. Validate the offset range and use a stable, suitably dense mapping.

Count and combine deliberately

BITCOUNT scans all bytes by default. Its optional start and end arguments are inclusive byte positions unless BIT is specified; that modifier is available from Redis 7.0. Changing units changes the selected interval, not the representation stored at the key. The command returns a population count, not the highest occupied identifier.

BITOP OR combines membership, while AND computes intersection. Shorter inputs and missing keys behave as zero-padded inputs up to the longest input length. The destination is overwritten with a new string; an existing destination expiration is removed. Derived results need their own deliberate lifetime policy, and recomputing them does not preserve a previously attached deadline.

When it breaks

BITCOUNT, BITPOS, and BITOP can scan substantial data. Optimized bit operations do not make full-bitmap work constant-time, and extending a string must allocate the intervening space. Evaluate offset span, query frequency, result lifetimes, and peak allocation together. A tiny number of active users says little about the resource cost of a sparse bitmap.

BITPOS returns an absolute bit offset even when searching a subrange. Searching for zero without an explicit end can return the first position beyond an all-one string, treating the right side as zero-padded. With both bounds specified, no matching bit in that interval returns -1. That difference matters when the bitmap represents a fixed-capacity resource pool.

Finding a zero is not reserving it. Two callers can read the same free position before either changes it. A separate BITPOS and SETBIT sequence is not one atomic claim, and ignoring SETBIT’s old-value result can double-assign a slot. Likewise, OR-ing two day keys is only meaningful when both use the same identifier mapping.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0. Build only Redis core with libc and install the client in a temporary virtual environment; no optional modules or system-wide installation are required.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It uses its own Redis Unix socket under /tmp, disables TCP and persistence, and removes only its runtime directory after stopping its process. Downloading dependencies requires network access. The largest bitmap in this experiment has one MiB of payload; it does not allocate the enormous example offsets discussed above.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", "noeviction",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


with instance() as client:
 assert client.getbit("missing", 100) == 0
 assert client.exists("missing") == 0
 assert client.setbit("active", 0, 1) == 0
 assert client.setbit("active", 2, 1) == 0
 assert client.setbit("active", 2, 1) == 1
 assert client.type("active") == b"string"
 assert client.get("active") == b"\xa0"
 assert client.bitcount("active") == 2
 print("offset zero is the high bit; repeated SETBIT returns old 1 without adding membership")

 highest = 8 * 1024 * 1024 - 1
 assert client.setbit("sparse", highest, 0) == 0
 assert client.strlen("sparse") == 1024 * 1024
 assert client.bitcount("sparse") == 0
 assert client.setbit("sparse", highest, 1) == 0
 assert client.setbit("sparse", highest, 0) == 1
 assert client.strlen("sparse") == 1024 * 1024
 print("writing a distant zero allocates 1048576 payload bytes; clearing does not shrink")

 client.set("range", b"\x80\x01")
 assert client.bitcount("range", 0, 1) == 2
 assert client.bitcount("range", 0, 1, mode="BIT") == 1
 assert client.bitpos("range", 1, 1, 1) == 15
 client.set("full", b"\xff")
 assert client.bitpos("full", 0) == 8
 assert client.bitpos("full", 0, 0, 0) == -1
 print("ranges default to bytes; BIT changes units; BITPOS returns absolute bit offsets")

 client.set("other", b"\x50\x01")
 client.set("union", b"obsolete", px=60000)
 assert client.bitop("OR", "union", "active", "other") == 2
 assert client.get("union") == b"\xf0\x01"
 assert client.bitcount("union") == 5
 assert client.ttl("union") == -1
 assert client.bitop("AND", "intersection", "active", "other") == 2
 assert client.get("intersection") == b"\x00\x00"
 print("BITOP zero-pads shorter inputs; OR has 5 flags; destination TTL is removed")

 client.hset("hash", "field", b"value")
 try:
 client.setbit("hash", 0, 1)
 except redis.ResponseError as error:
 assert "WRONGTYPE" in str(error)
 else:
 raise AssertionError("a hash is not converted into a bitmap")
 assert client.hget("hash", "field") == b"value"
 print("bitmap commands reject a hash without converting or replacing it")

Bits zero and two produce byte 0xa0 and count two. Writing a distant zero still creates 1,048,576 payload bytes; toggling and clearing it leaves that length unchanged. Byte-range counting and bit-range counting return two and one respectively, while the bounded BITPOS result remains an absolute offset.

An all-one byte yields position eight for an unbounded zero search but -1 for a bounded search. OR produces five flags across two bytes and removes the destination TTL; AND produces two zero bytes. Wrong-type access leaves the hash intact. These results verify representation and command semantics, not total process memory, concurrent allocation correctness, or production throughput.

Sources

Redis SETBIT, previous-bit replies and string growth, checked 2026-09-08.
Redis GETBIT, missing-key and out-of-range reads, checked 2026-09-08.
Redis BITCOUNT, population counts and byte/bit ranges, checked 2026-09-08.
Redis BITPOS, absolute offsets and bounded zero searches, checked 2026-09-08.
Redis BITOP, output strings, zero-padding, and operation cost, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only compilation and allocator selection, checked 2026-09-08.

A Redis list is a queue, a stack, or both

Sun, 18 Apr 2027 00:00:00 +0000

A Redis list is an ordered sequence of byte values with operations at both ends. Choosing the push and pop ends determines whether it behaves as a queue or a stack. Blocking operations wait for data; they are not distributed locks, acknowledgements, or a complete message-processing protocol.

Choose the ends deliberately

RPUSH followed by LPOP provides FIFO removal: older entries leave first. RPUSH followed by RPOP provides LIFO removal. Multiple values in one LPUSH are inserted at the left in argument order, so pushing a, b, c leaves c, b, a from left to right. Test actual producer commands, not just the queue diagram.

Lists allow duplicate values. Two identical messages occupy two entries; list insertion does not provide deduplication. Use distinct job identifiers when separate deliveries must remain distinguishable. LRANGE uses inclusive endpoints, with -1 denoting the final element. Removing the last entry removes the list key, and popping a missing list returns nil rather than creating an empty placeholder.

End operations are inexpensive per element, but returning a large range or removing many entries still requires proportional work. LTRIM can retain a recent window, but truncating pending work deliberately discards entries. A producer push and a separate trim are not one atomic operation. Select retention boundaries according to the data’s meaning.

Wait without monopolizing the server

BLPOP waits on the connection until an item becomes available or the timeout expires. Other connections continue operating, including the producer that supplies the item. With redis-py, success returns the key and value; timeout returns None. A zero timeout waits indefinitely. A client’s socket deadline must allow the intended server wait and response.

Blocking consumers occupy connections even when they do no work. Allocate a bounded connection budget and avoid sharing one connection between a waiting consumer and its producer. A FIFO list also orders removal, not completion: parallel workers can finish later jobs first. Blocking behavior alone does not enforce exclusive ownership of a business resource.

When it breaks

LPOP and BLPOP remove a job before its business work finishes. If the consumer crashes or loses the response, the job is no longer on the ready list. Redis does not automatically restore it or require acknowledgement. Persistence settings do not repair this application-level handoff by themselves.

LMOVE, available since Redis 6.2, atomically transfers an item between a ready list and a processing list. BLMOVE is its blocking variant. After successful processing, LREM can acknowledge one uniquely identified job by removing it from processing. The application still needs a recovery policy for abandoned work; a list does not attach a lease, timestamp, or retry count to each item.

If work succeeds but acknowledgement fails, recovery can repeat the side effect. Unique job identifiers, idempotent processing, and a policy for late workers matter; moving between lists does not establish exactly-once processing. The exercise deliberately requeues one known job to demonstrate the mechanics, not an automatic failure detector or production recovery scheduler.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0. These commands build Redis core with libc and install the client in a temporary virtual environment, without optional modules or system-wide installation.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It creates a private Redis Unix socket under /tmp, disables TCP, scheduled snapshots, and append-only persistence, then terminates only its own process and removes its runtime directory. Dependency downloads need network access. No default local Redis is contacted.

from concurrent.futures import ThreadPoolExecutor

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", "noeviction",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


with instance() as client:
 client.rpush("fifo", b"a", b"b", b"c")
 assert [client.lpop("fifo") for _ in range(3)] == [b"a", b"b", b"c"]
 assert client.exists("fifo") == 0
 assert client.lpop("fifo") is None
 client.rpush("stack", b"a", b"b", b"c")
 assert [client.rpop("stack") for _ in range(3)] == [b"c", b"b", b"a"]
 client.lpush("left", b"a", b"b", b"c")
 assert client.lrange("left", 0, -1) == [b"c", b"b", b"a"]
 client.rpush("duplicates", b"same", b"same")
 assert client.llen("duplicates") == 2
 print("FIFO a,b,c; LIFO c,b,a; multi-value LPUSH reverses argument order; duplicates stay")

 def consume():
 with client.client() as connection:
 connection.client_setname("list-lab-consumer")
 return connection.blpop("channel", timeout=1)

 with ThreadPoolExecutor(max_workers=1) as worker:
 future = worker.submit(consume)
 deadline = time.monotonic() + 1
 while not any(
 entry["name"] == "list-lab-consumer" and "b" in entry["flags"]
 for entry in client.client_list()
 ):
 assert time.monotonic() < deadline, "consumer never blocked"
 time.sleep(0.01)
 client.rpush("channel", b"job:1")
 assert future.result(timeout=2) == (b"channel", b"job:1")
 assert client.blpop("empty", timeout=0.1) is None
 print("observed a blocked connection; RPUSH wakes it; empty BLPOP times out")

 client.rpush("ready", b"job:2", b"job:3")
 job = client.lmove("ready", "processing", "LEFT", "RIGHT")
 assert job == b"job:2"
 assert client.lrange("ready", 0, -1) == [b"job:3"]
 assert client.lrange("processing", 0, -1) == [b"job:2"]
 assert client.lmove("processing", "ready", "LEFT", "LEFT") == job
 assert client.lrange("ready", 0, -1) == [b"job:2", b"job:3"]
 assert client.lmove("ready", "processing", "LEFT", "RIGHT") == job
 assert client.lrem("processing", 1, job) == 1
 assert client.exists("processing") == 0
 print("LMOVE retains an in-flight job; explicit requeue and LREM acknowledgement work")

 client.set("wrong", b"not a list")
 try:
 client.lmove("ready", "wrong", "LEFT", "RIGHT")
 except redis.ResponseError as error:
 assert "WRONGTYPE" in str(error)
 else:
 raise AssertionError("LMOVE must reject a non-list destination")
 assert client.lrange("ready", 0, -1) == [b"job:3"]
 assert client.get("wrong") == b"not a list"
 client.rpush("recent", b"1", b"2", b"3", b"4")
 client.ltrim("recent", -2, -1)
 assert client.lrange("recent", 0, -1) == [b"3", b"4"]
 print("wrong-type move preserves source; LTRIM keeps only the requested window")

The first assertions distinguish FIFO, LIFO, multi-value LPUSH order, and duplicate entries. The producer waits until CLIENT LIST actually reports a blocked consumer before submitting work; a separate empty pop returns None after its finite timeout. This observes real blocking without asserting exact scheduler latency.

LMOVE leaves the claimed job in processing, an explicit reverse move requeues it, and LREM removes the later acknowledgement. A wrong-type destination is rejected without losing the source entry. Finally, LTRIM retains two selected values. These checks verify command behavior, not crash durability, automatic recovery, throughput, or distributed locking.

Sources

Redis LPUSH, multi-value insertion order, checked 2026-09-08.
Redis LRANGE, inclusive range bounds and operation cost, checked 2026-09-08.
Redis BLPOP, connection blocking, timeouts, and destructive delivery, checked 2026-09-08.
Redis LMOVE, atomic transfer and the processing-list pattern, checked 2026-09-08.
Redis LREM, removal by value and count, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only compilation and allocator selection, checked 2026-09-08.

Redis · XB Field Notes

A Redis hash stores many fields in one key

Read the fields your caller actually needs

Updates have precise return values

Compact representation is an implementation choice

When it breaks

Check it quickly

Sources

Redis pub/sub and streams are different tools

Decide who receives each event

Retention is not a durability setting

When it breaks

Check it quickly

Sources

Redis Streams let a group share the work

Position and pending work differ

Recovery needs an explicit decision

When it breaks

Check it quickly

Sources

Redis geo handles locations by distance

Longitude comes before latitude

State the search contract

When it breaks

Check it quickly

Sources

Pipelining folds many Redis round trips into one

Compare equivalent work

Include the costs the application pays

When it breaks

Check it quickly

Sources

Redis Lua scripts run without interleaving

Keep the decision beside the data

When it breaks

Keep source available

Check it quickly

Sources

A Redis list and a sorted set solve different problems

Define what makes an entry distinct

Positions and scores select different things

When it breaks

Check it quickly

Sources

Redis expiry is a deadline, eviction is a capacity policy

Time and capacity are separate axes

Select candidates before debating ranking

When it breaks

Check it quickly

Further reading

Sources

Redis Cluster splits keys across slots

Keys choose slots, not contiguous machine ranges

Clients participate in routing

Placement does not imply transaction scope

When it breaks

Check it quickly

Sources

Redis append-only log trades speed for durability

Execution, logging, and acknowledgment

Choose the synchronization contract

Rewriting creates a recoverable file set

When it breaks

Check it quickly

Sources

MULTI EXEC runs a batch of commands as one

Watch before reading the dependency

Retry the decision, not its old output

When it breaks

Check it quickly

Sources

Redis streams track a consumer's own cursor

The group advances on delivery

Pending history belongs to a consumer

When it breaks

Check it quickly

Sources

Redis TTL lets every key carry its own clock

A deadline is not a sliding timer

Inspect the actual state