Streams · XB Field Notes

Redis pub/sub and streams are different tools

Sun, 06 Feb 2028 00:00:00 +0000

Redis Pub/Sub and Streams answer different delivery questions. Pub/Sub broadcasts to currently active subscriptions without retaining messages for later replay. A stream stores entries that readers can revisit while they remain retained. Choosing between them starts with who must receive an event and what should happen during a consumer’s absence.

Decide who receives each event

For a replaceable live dashboard update, broadcasting to connected viewers may be sufficient. Wait for subscription confirmation before assuming a new listener is active. Reconnecting restores future delivery, not missed publications. A positive PUBLISH response counts recipients according to Redis’s command semantics; it does not acknowledge application processing.

Streams offer two distinct consumption models. Ordinary XREAD lets independent readers request entries after their own supplied IDs, so both can receive the same retained history. It does not divide work or maintain an automatic checkpoint per connection. XREADGROUP within one consumer group distributes new deliveries among named workers and records pending entries unless NOACK is used.

Separate consumer groups allow separate workflows, such as billing and analytics, to progress independently over one stream. Membership within a group shares work; adding a new group creates another delivery history. Neither choosing unique consumer names nor adding workers promises equal load or business completion order. Decide whether the requirement is broadcast, independent replay, or competing work before choosing the command.

Retention is not a durability setting

XADD retains data in Redis; that does not by itself make every accepted append survive a crash or failover. RDB/AOF settings, replication behavior, key eviction, and operational recovery determine durability. Pub/Sub does not acquire a replay log by enabling persistence. Calling the difference simply latency versus durability hides both contracts.

MAXLEN trims the oldest entries, not the newest tail while preserving older history. Approximate trimming may retain extra records, while exact trimming follows the requested count. Neither makes unlimited replay possible. Default trimming can remove payloads still referenced by pending entries; select retention policies and capacity for the longest supported outage and subsequent catch-up.

Stored backlog also has an operational cost. Bound its growth, monitor unread lag and pending work separately, and define what happens to poison messages. Pub/Sub instead avoids retained history but still uses connection output buffers. A slow subscriber can be disconnected when those limits are exceeded. Neither mechanism eliminates memory planning.

When it breaks

Using Pub/Sub for required jobs silently loses the opportunity to deliver while no listener is subscribed. A dashboard that can reload authoritative state may tolerate that; a required financial action usually cannot. The appropriate replacement might be a retained stream or another durable queue, with recovery semantics designed around the action rather than assumed from the product name.

Using Streams without a recovery loop creates a different failure: an unacknowledged group delivery can remain pending while ordinary > reads continue with new entries. Acknowledging before processing risks loss; processing before acknowledgement permits duplicate effects after retry. Stable event identifiers and idempotent handling remain necessary when repeated business actions are unacceptable.

Publishing and appending the same event as separate commands is not an atomic dual-write protocol. One operation can succeed without the other if execution stops between them. The exercise deliberately compares two mechanisms in an isolated process; it does not propose that sequence as a production delivery guarantee.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0. The commands compile Redis core with libc and install its client in a temporary virtual environment, without optional modules or system-wide installation.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It creates a private Unix socket under /tmp, disables TCP, scheduled snapshots, and append-only persistence, then stops only its own process and removes its runtime directory. Downloads require network access. No default local Redis is contacted.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", "noeviction",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


def receive(subscription):
 deadline = time.monotonic() + 2
 while time.monotonic() < deadline:
 response = subscription.get_message(timeout=0.1)
 if response is not None:
 return response
 raise AssertionError("subscription response timed out")


with instance() as client:
 assert client.publish("updates", b"offline") == 0
 first_id = client.xadd("history", {"event": "offline"}, id="1-0")
 with client.pubsub() as first, client.pubsub() as second:
 for subscription in (first, second):
 subscription.subscribe("updates")
 assert receive(subscription)["type"] == "subscribe"
 assert client.publish("updates", b"live") == 2
 second_id = client.xadd("history", {"event": "live"}, id="2-0")
 for subscription in (first, second):
 response = receive(subscription)
 assert response["type"] == "message" and response["data"] == b"live"
 with client.client() as reader_one, client.client() as reader_two:
 one = reader_one.xread({"history": "0-0"})
 two = reader_two.xread({"history": "0-0"})
 assert one == two
 assert [entry[0] for entry in one[0][1]] == [first_id, second_id]
 print("Pub/Sub listeners get live only; both independent XREAD readers replay offline and live")

 client.xgroup_create("history", "workers", id="0-0")
 a = client.xreadgroup("workers", "a", {"history": ">"}, count=1)[0][1]
 b = client.xreadgroup("workers", "b", {"history": ">"}, count=1)[0][1]
 assert a[0][0] == first_id and b[0][0] == second_id
 assert client.xpending("history", "workers")["pending"] == 2
 assert client.xack("history", "workers", first_id, second_id) == 2
 assert client.xlen("history") == 2
 print("one group splits new entries between a/b; ACK clears pending state, not stored history")

 third_id = client.xadd("history", {"event": "later"}, id="3-0")
 assert client.xtrim("history", maxlen=2, approximate=False) == 1
 retained = client.xread({"history": "0-0"})[0][1]
 assert [entry[0] for entry in retained] == [second_id, third_id]
 assert client.xrange("history", first_id, first_id) == []
 print("exact MAXLEN removes the oldest entry; replay is limited to retained history")

The offline publication returns zero, while XADD keeps an entry with the same event label. Once both subscriptions are confirmed, each listener receives live, not offline. Two ordinary XREAD calls then return both stored entries, proving independent replay rather than division of work.

Creating one group changes that delivery pattern: workers a and b receive different entries, and both become pending. Acknowledging them leaves the stream length unchanged. A third append followed by exact MAXLEN removes only the oldest retained entry; reading from zero cannot recover its missing payload.

The three output summaries follow assertions on actual responses. They demonstrate subscription presence, independent replay, grouped distribution, and finite retention. This run does not benchmark latency or memory, reproduce a real crash, validate persistence, or test automatic recovery. Those operational choices remain part of the application’s delivery contract.

Sources

Redis Pub/Sub, live delivery, subscription acknowledgement, and absent replay, checked 2026-09-08.
Redis XREAD, independent reads and caller-supplied positions, checked 2026-09-08.
Redis XREADGROUP, shared work and pending entries, checked 2026-09-08.
Redis XTRIM, oldest-entry removal and reference policies, checked 2026-09-08.
Redis client handling, Pub/Sub output buffering and disconnect limits, checked 2026-09-08.
Redis persistence, RDB and AOF recovery tradeoffs, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only compilation and allocator selection, checked 2026-09-08.

Redis Streams let a group share the work

Sat, 05 Feb 2028 00:00:00 +0000

A Redis Streams consumer group distributes new entries among named consumers and records deliveries awaiting acknowledgement. It does not guarantee exactly-once effects, prevent retention loss, or recover abandoned work by itself. Understanding the group position, pending entries, and retained stream data separately is essential before treating it as a work queue.

Position and pending work differ

XGROUP CREATE jobs workers 0-0 starts before the retained history; using $ instead starts after the current end. Workers in one group share new deliveries, while separate groups maintain independent delivery state. Use distinct names for concurrent workers. The group owns a last-delivered position, not a separate offset committed by each consumer.

For ordinary XREADGROUP without CLAIM or NOACK, > requests new entries and delivery adds them to the Pending Entries List, or PEL. The group position advances when new entries are delivered, before XACK. XPENDING reports delivered but unacknowledged entries, not the backlog that has never been delivered. A group can therefore have pending count zero while unread work exists.

XACK removes specified IDs from that group’s PEL. It does not delete their stream records or advance the last-delivered position. Acknowledgement can finish out of order, and another group can still read the retained entries. XREADGROUP with 0-0 reads this named consumer’s pending history instead of requesting new work; it does not retrieve another consumer’s pending messages.

Recovery needs an explicit decision

Repeated ordinary > reads do not retry an abandoned PEL entry. XAUTOCLAIM transfers eligible pending entries to the requesting consumer. Its arguments include the stream, group, consumer, minimum idle milliseconds, and starting ID. Continue scanning with the returned cursor until 0-0; an empty claimed batch alone does not mean the scan is complete. Later scans may find entries that have since become eligible.

Redis 8.4 also introduced XREADGROUP CLAIM as an explicit combined recovery option. The example does not enable it. Its XAUTOCLAIM idle threshold of zero is solely for deterministic local observation; production thresholds must reflect processing time and liveness policy. Claiming ownership cannot stop an old worker that is still running or undo its external side effects.

When it breaks

A crash after an external effect but before acknowledgement leaves work eligible for another delivery. Retrying can repeat that effect. Acknowledging before the effect instead risks losing required work. Use stable business identifiers and transactional deduplication where possible; a delivery counter records delivery attempts, not completed database transactions. Bound retries and preserve failing payloads before retiring work.

A PEL reference does not pin the stream payload. XDEL and default KEEPREF trimming can remove it while leaving a pending ID. XAUTOCLAIM reports and clears such missing records on Redis 7 and later; that is cleanup, not recovery of their contents. Choose retention with consumer outages in mind. Stream persistence and replication also require separate durability decisions.

Monitor unread lag, pending count, idle time, and repeated deliveries together. One count cannot establish queue health. NOACK omits pending tracking for new deliveries and is unsuitable when that tracking is required for recovery. Parallel workers can finish out of insertion order even when the stream IDs themselves remain ordered.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0. The commands build Redis core with libc and install the client in a temporary virtual environment, without optional modules or system-wide installation.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It creates a private Unix socket under /tmp, disables TCP, scheduled snapshots, and append-only persistence, then terminates only its own process and removes its runtime directory. Downloads require network access. No default local Redis is contacted.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", "noeviction",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


with instance() as client:
 for entry_id in ("1-0", "2-0", "3-0"):
 client.xadd("jobs", {"job": entry_id}, id=entry_id)
 client.xgroup_create("jobs", "workers", id="0-0")
 client.xgroup_create("jobs", "tail", id="$")
 assert client.xpending("jobs", "workers")["pending"] == 0
 assert client.xreadgroup("tail", "observer", {"jobs": ">"}) == []
 first = client.xreadgroup("workers", "w1", {"jobs": ">"}, count=1)[0][1]
 second = client.xreadgroup("workers", "w2", {"jobs": ">"}, count=1)[0][1]
 assert first[0][0] == b"1-0" and second[0][0] == b"2-0"
 assert client.xpending("jobs", "workers")["pending"] == 2
 group = next(item for item in client.xinfo_groups("jobs") if item["name"] == b"workers")
 assert group["last-delivered-id"] == b"2-0"
 print("backlog starts with PEL=0; new deliveries split 1-0/2-0 and advance the group position")

 effects = [first[0][0]]
 assert client.xack("jobs", "workers", "2-0") == 1
 assert client.xlen("jobs") == 3
 group = next(item for item in client.xinfo_groups("jobs") if item["name"] == b"workers")
 assert group["last-delivered-id"] == b"2-0"
 assert client.xreadgroup("workers", "w2", {"jobs": "0-0"}) == [[b"jobs", []]]
 assert client.xreadgroup("workers", "w1", {"jobs": "0-0"})[0][1] == first
 third = client.xreadgroup("workers", "w2", {"jobs": ">"})[0][1]
 assert third[0][0] == b"3-0"
 assert client.xack("jobs", "workers", "3-0") == 1
 assert client.xreadgroup("workers", "w2", {"jobs": ">"}) == []
 print("XACK keeps stream data and read position; 0-0 reads own pending; > does not recover w1")

 cursor, claimed, deleted = client.xautoclaim(
 "jobs", "workers", "w2", min_idle_time=0, start_id="0-0", count=10,
 )
 assert cursor == b"0-0" and claimed == first and deleted == []
 pending = client.xpending_range("jobs", "workers", "-", "+", 10)
 assert len(pending) == 1 and pending[0]["consumer"] == b"w2"
 assert pending[0]["times_delivered"] >= 2
 effects.append(claimed[0][0])
 assert effects.count(b"1-0") == 2
 assert client.xack("jobs", "workers", "1-0") == 1
 assert client.xack("jobs", "workers", "1-0") == 0
 assert client.xpending("jobs", "workers")["pending"] == 0
 assert client.xlen("jobs") == 3
 print("explicit claim changes owner; simulated side effect repeats; ACK clears PEL, not the stream")

 client.xadd("jobs", {"job": "4-0"}, id="4-0")
 assert client.xreadgroup("tail", "observer", {"jobs": ">"})[0][1][0][0] == b"4-0"
 assert client.xreadgroup("workers", "w1", {"jobs": ">"})[0][1][0][0] == b"4-0"
 assert client.xdel("jobs", "4-0") == 1
 assert client.xpending("jobs", "workers")["pending"] == 1
 cursor, claimed, deleted = client.xautoclaim(
 "jobs", "workers", "w2", min_idle_time=0, start_id="0-0", count=10,
 )
 assert cursor == b"0-0" and claimed == [] and deleted == [b"4-0"]
 assert client.xpending("jobs", "workers")["pending"] == 0
 print("a $ group starts after old entries; XDEL loses pending payload; claim reports deleted ID")

The workers group begins with three unread entries and zero pending entries. Two reads distribute 1-0 and 2-0; the group position already reaches 2-0 before acknowledgement. Acknowledging the second entry leaves stream length and that position unchanged. Pending-history reads distinguish the two consumers, and a new-entry read returns 3-0 instead of retrying 1-0.

The effects list models a completed action without its acknowledgement. Explicit claiming moves 1-0 to w2, and processing it again produces two recorded effects. This is a simulation of duplicate business processing, not a transaction-safe implementation. Acknowledgement then clears the PEL while leaving the three stream records available.

The tail group sees only a later 4-0 entry. Deleting that entry while pending makes the subsequent claim return its ID in the deleted list without a payload. The checks exercise command-level state changes, not process-crash detection, automatic retry scheduling, durable deduplication, or failover recovery.

Sources

Redis XGROUP CREATE, group starting positions, checked 2026-09-08.
Redis XREADGROUP, new and pending reads, NOACK, and optional CLAIM behavior, checked 2026-09-08.
Redis XPENDING, pending ownership and delivery counts, checked 2026-09-08.
Redis XACK, acknowledgement and PEL removal, checked 2026-09-08.
Redis XAUTOCLAIM, recovery cursors, idle thresholds, and deleted entries, checked 2026-09-08.
Redis XTRIM, retention and consumer-group reference policies, checked 2026-09-08.
Redis persistence, RDB and AOF durability tradeoffs, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only compilation and allocator selection, checked 2026-09-08.

Redirection reorders the streams

Wed, 29 Sep 2027 00:00:00 +0000

Redirection changes a command’s file descriptors before the command executes. The shell processes redirections from left to right, opening or duplicating a descriptor as it reaches each operation. Descriptors 0, 1, and 2 conventionally represent standard input, standard output, and standard error. Reading this sequence as concrete changes to those handles explains both misplaced diagnostics and files that become empty before their contents can be read.

Opening a file is already a side effect

>output opens a file for writing and normally truncates an existing regular file. >>output opens it for appending. <input opens a file for reading. These actions belong to the shell, not to the program named by the command. If opening a required target fails, the command is not executed; successful earlier redirections can still have created or truncated other files.

Consequently, transforming a file with sed expression input >input destroys the original before sed can read it. Opening input first with <input does not rescue the data if a later redirection truncates the same underlying file. Write to a separate temporary file, check the transformation’s status and result, and only then perform a deliberate replacement appropriate to the file’s ownership and filesystem.

Duplication captures the current destination

>log 2>&1 first sends descriptor 1 to the log, then makes descriptor 2 refer to that same open destination. 2>&1 >log first duplicates the original standard output onto descriptor 2, then changes only descriptor 1. Standard error therefore remains at the original destination, such as a terminal or a capturing pipe. Duplication copies a descriptor reference, not file contents, and it does not create a live connection between descriptor numbers.

Bash’s &>log is shorthand for redirecting both streams, but >log 2>&1 also works in portable shell scripts. Opening the same filename twice with >log 2>log is not equivalent: the two opens have independent offsets and can overwrite one another’s output. Even when descriptors share one destination, application buffering can affect the order in which messages become visible.

Pipelines connect standard output to the next stage before command-specific redirections are applied. In producer >file | consumer, the producer’s output consequently goes to the file, leaving the consumer without that data. Conversely, producer 2>&1 | consumer sends diagnostics through the pipe too. When the consumer expects machine-readable records, merging human-readable errors into that stream can corrupt otherwise valid input.

When it breaks

Quote variable filenames so spaces do not turn a target into multiple words. Keep the descriptor number adjacent to its operator: 2>errors redirects standard error, while 2 >errors supplies an ordinary argument containing 2 and redirects standard output. A misplaced space can therefore change both program input and output routing.

Bash’s set -C enables noclobber, rejecting ordinary output redirection to an existing regular file. It is useful protection against accidental replacement, not a backup system or a universal guarantee about every file type. >| explicitly overrides it. Also distinguish a command’s temporary redirections from exec >log, which changes the current shell’s descriptors and can redirect every subsequent command. Local tests should not unexpectedly silence the surrounding session.

Check it quickly

Run the following in Bash with standard Unix utilities. Only freshly created temporary files are truncated or removed. The output compares both orderings, demonstrates same-file truncation, and checks that a failed redirection prevents a function body from running.

bash --noprofile --norc <<'BASH'
set -eu
demo_dir=$(mktemp -d)
trap 'rm -f "$demo_dir/both" "$demo_dir/out" "$demo_dir/input"; rmdir "$demo_dir"' EXIT
emit() { printf 'out\n'; printf 'err\n' >&2; }
emit > "$demo_dir/both" 2>&1
demo_captured=$(emit 2>&1 > "$demo_dir/out")
printf 'both:\n'
cat "$demo_dir/both"
printf 'out-file=[%s] captured=[%s]\n' \
 "$(cat "$demo_dir/out")" "$demo_captured"
printf 'seed\n' > "$demo_dir/input"
sed 's/seed/replaced/' "$demo_dir/input" > "$demo_dir/input"
if [ ! -s "$demo_dir/input" ]; then printf 'input truncated\n'; fi
if (set -C; emit > "$demo_dir/both") 2>/dev/null; then
 printf 'unexpected overwrite\n'
else
 printf 'noclobber rejected overwrite\n'
fi
demo_ran=no
mark_run() { demo_ran=yes; }
if mark_run 2>/dev/null > "$demo_dir/missing/output"; then
 printf 'unexpected redirection\n'
else
 printf 'function-ran=%s\n' "$demo_ran"
fi
BASH

The combined file prints out and err. Reversed ordering leaves only out in the output file and captures err separately. The transformation reports an empty input because truncation preceded execution. The remaining lines confirm that noclobber rejected replacement and the failed open left function-ran=no.

These checks observe file contents and whether the command actually ran, rather than relying on a successful final print. In a real script, retain the producer’s exit status alongside its logs: a perfectly routed error message does not turn a failed operation into success. Keeping output routing and operation status separate makes recovery decisions reliable.

Sources

Bash reference manual: redirections, left-to-right descriptor changes and noclobber behavior, checked 2026-09-07.
POSIX shell redirection, portable redirection operators and file-opening semantics, checked 2026-09-07.

Redis streams track a consumer's own cursor

Tue, 13 Jul 2027 00:00:00 +0000

A Redis stream does not automatically track a separate cursor for every connection. Ordinary XREAD uses a position supplied by the application. A consumer group instead owns a server-side last-delivered position plus pending delivery records. Confusing those two forms of progress can make a restart skip unfinished work or replay effects already completed.

The group advances on delivery

XGROUP CREATE with 0-0 starts before retained history; $ starts after the current end. Each group maintains independent delivery state over the same stream records. Billing can have delivered two entries while analytics has delivered only one. Within a group, named consumers share new work rather than each receiving every entry.

For ordinary XREADGROUP without CLAIM or NOACK, > requests entries beyond the group’s delivery position and adds deliveries to its Pending Entries List, or PEL. That position advances before processing and acknowledgement. XACK removes specified pending IDs from one group, not stream records, and does not advance the group’s last-delivered ID.

Acknowledgements can arrive out of order. Finishing the second entry says nothing about whether the first finished. The group can have reached the end of the stream while older deliveries remain pending. Do not reconstruct successful processing from the largest delivered or acknowledged ID without an application contract that actually guarantees contiguous completion.

Pending history belongs to a consumer

Using 0-0 with XREADGROUP requests this named consumer’s pending history, not an unrestricted replay of the stream. A newly named worker sees none of another worker’s pending entries. Restarting under a different name and reading only > therefore does not recover abandoned deliveries. It can make the service look active while older work never resumes.

Recovery requires reading the original consumer’s pending history or explicitly transferring eligible work, for example with XAUTOCLAIM. Its idle threshold must account for legitimate processing time. Current Redis also supports optional XREADGROUP CLAIM; this exercise does not enable it. Ownership transfer cannot prevent a late worker from repeating an external action, so processing still needs an appropriate deduplication policy.

Use XINFO GROUPS to compare last-delivered-id, pending, and lag. Lag estimates entries awaiting delivery from the group’s current position, not unfinished business actions. It can be unavailable after certain position changes or deletions; a null metric is not zero. A group with zero lag and nonzero pending work has reached the delivery end but still has outstanding acknowledgements.

When it breaks

XGROUP SETID is a deliberate change to the group’s delivery position, not a harmless acknowledgement repair. Rewinding after completed work makes retained, previously acknowledged entries eligible for new delivery again. Moving forward can skip unread entries. It does not automatically empty existing PEL records, and neither direction changes another group’s position.

Plan a repositioning operation with workers coordinated and business effects understood. Replaying into a separate group isolates delivery bookkeeping, but not external side effects if it calls the same handler. An acknowledgement is not a permanent tombstone that forbids all future reads of that record. Avoid using a rewind as a substitute for diagnosing the pending entries.

Independent group positions do not provide independent retention. Deleting or trimming a shared payload affects every group that might need it. Default KEEPREF trimming may leave pending IDs without data. Durability across server failure is also separate from the cursor mechanism. A recorded delivery position is neither proof that the payload survives nor proof that its business work completed.

Check it quickly

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", "noeviction",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


def groups(client):
 return {group["name"]: group for group in client.xinfo_groups("events")}


with instance() as client:
 for entry_id in ("1-0", "2-0", "3-0"):
 client.xadd("events", {"event": entry_id}, id=entry_id)
 for name in ("billing", "analytics"):
 client.xgroup_create("events", name, id="0-0")
 billing = client.xreadgroup("billing", "b1", {"events": ">"}, count=2)[0][1]
 analytics = client.xreadgroup("analytics", "a1", {"events": ">"}, count=1)[0][1]
 assert [entry[0] for entry in billing] == [b"1-0", b"2-0"]
 assert [entry[0] for entry in analytics] == [b"1-0"]
 state = groups(client)
 assert state[b"billing"]["last-delivered-id"] == b"2-0"
 assert state[b"analytics"]["last-delivered-id"] == b"1-0"
 assert state[b"billing"]["pending"] == 2 and state[b"analytics"]["pending"] == 1
 assert state[b"billing"]["lag"] == 1 and state[b"analytics"]["lag"] == 2
 print("two groups read overlapping records but retain independent positions and PELs")

 assert client.xack("events", "billing", "2-0") == 1
 state = groups(client)
 assert state[b"billing"]["last-delivered-id"] == b"2-0"
 assert state[b"billing"]["pending"] == 1 and state[b"analytics"]["pending"] == 1
 assert client.xreadgroup("billing", "b2", {"events": "0-0"}) == [[b"events", []]]
 own = client.xreadgroup("billing", "b1", {"events": "0-0"})[0][1]
 assert [entry[0] for entry in own] == [b"1-0"]
 next_entry = client.xreadgroup("billing", "b2", {"events": ">"})[0][1]
 assert [entry[0] for entry in next_entry] == [b"3-0"]
 assert groups(client)[b"billing"]["lag"] == 0
 assert groups(client)[b"billing"]["pending"] == 2
 assert client.xack("events", "billing", "1-0", "3-0") == 2
 assert groups(client)[b"billing"]["pending"] == 0
 assert client.xpending("events", "analytics")["pending"] == 1
 assert client.xlen("events") == 3
 print("ACK affects only one group's PEL; a new consumer sees no other owner's pending history")

 client.xgroup_setid("events", "billing", "0-0")
 replay = client.xreadgroup("billing", "b3", {"events": ">"}, count=1)[0][1]
 assert [entry[0] for entry in replay] == [b"1-0"]
 assert groups(client)[b"analytics"]["last-delivered-id"] == b"1-0"
 assert client.xack("events", "billing", "1-0") == 1
 client.xgroup_setid("events", "analytics", "$")
 state = groups(client)
 assert state[b"analytics"]["last-delivered-id"] == b"3-0"
 assert state[b"analytics"]["pending"] == 1
 assert state[b"analytics"]["lag"] == 0
 assert client.xreadgroup("analytics", "a1", {"events": ">"}) == []
 own = client.xreadgroup("analytics", "a1", {"events": "0-0"})[0][1]
 assert [entry[0] for entry in own] == [b"1-0"]
 assert client.xack("events", "analytics", "1-0") == 1
 print("SETID rewinds acknowledged data for replay; moving forward skips unread data but keeps PEL")

Billing reads two entries and analytics reads one, producing distinct delivery positions and pending counts. Acknowledging billing’s second entry changes only its PEL. Consumer b2 cannot read b1’s pending history, but it can receive the next new entry. After all new billing entries are delivered, lag is zero while two deliveries still await acknowledgement.

Once billing work is acknowledged, explicitly rewinding its position lets b3 receive 1-0 again. Analytics remains independent. Moving analytics to $ skips its unread entries but preserves the pending 1-0 record, which its original consumer can still read and acknowledge.

All state comparisons use actual Redis replies; no timing assumptions or concurrent workers are needed. The experiment verifies group isolation, delivery positions, pending ownership, and deliberate repositioning. It does not implement crash detection, recovery scheduling, persistence testing, or transaction-safe business processing.

Sources

Redis XREAD, positions supplied by independent readers, checked 2026-09-08.
Redis XREADGROUP, group delivery positions and consumer pending history, checked 2026-09-08.
Redis XACK, group-specific pending acknowledgement, checked 2026-09-08.
Redis XINFO GROUPS, last-delivered ID, pending count, and lag limitations, checked 2026-09-08.
Redis XGROUP SETID, deliberate repositioning of a group, checked 2026-09-08.
Redis XAUTOCLAIM, explicit recovery of eligible pending entries, checked 2026-09-08.
Redis XTRIM, payload retention and consumer-group references, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only compilation and allocator selection, checked 2026-09-08.

Streams · XB Field Notes

Redis pub/sub and streams are different tools

Decide who receives each event

Retention is not a durability setting

When it breaks

Check it quickly

Sources

Redis Streams let a group share the work

Position and pending work differ

Recovery needs an explicit decision

When it breaks

Check it quickly

Sources

Redirection reorders the streams

Opening a file is already a side effect

Duplication captures the current destination

When it breaks

Check it quickly

Further reading

Sources

Redis streams track a consumer's own cursor

The group advances on delivery

Pending history belongs to a consumer

When it breaks

Check it quickly

Sources