Consumer · XB Field Notes

Redis Streams let a group share the work

Sat, 05 Feb 2028 00:00:00 +0000

A Redis Streams consumer group distributes new entries among named consumers and records deliveries awaiting acknowledgement. It does not guarantee exactly-once effects, prevent retention loss, or recover abandoned work by itself. Understanding the group position, pending entries, and retained stream data separately is essential before treating it as a work queue.

Position and pending work differ

XGROUP CREATE jobs workers 0-0 starts before the retained history; using $ instead starts after the current end. Workers in one group share new deliveries, while separate groups maintain independent delivery state. Use distinct names for concurrent workers. The group owns a last-delivered position, not a separate offset committed by each consumer.

For ordinary XREADGROUP without CLAIM or NOACK, > requests new entries and delivery adds them to the Pending Entries List, or PEL. The group position advances when new entries are delivered, before XACK. XPENDING reports delivered but unacknowledged entries, not the backlog that has never been delivered. A group can therefore have pending count zero while unread work exists.

XACK removes specified IDs from that group’s PEL. It does not delete their stream records or advance the last-delivered position. Acknowledgement can finish out of order, and another group can still read the retained entries. XREADGROUP with 0-0 reads this named consumer’s pending history instead of requesting new work; it does not retrieve another consumer’s pending messages.

Recovery needs an explicit decision

Repeated ordinary > reads do not retry an abandoned PEL entry. XAUTOCLAIM transfers eligible pending entries to the requesting consumer. Its arguments include the stream, group, consumer, minimum idle milliseconds, and starting ID. Continue scanning with the returned cursor until 0-0; an empty claimed batch alone does not mean the scan is complete. Later scans may find entries that have since become eligible.

Redis 8.4 also introduced XREADGROUP CLAIM as an explicit combined recovery option. The example does not enable it. Its XAUTOCLAIM idle threshold of zero is solely for deterministic local observation; production thresholds must reflect processing time and liveness policy. Claiming ownership cannot stop an old worker that is still running or undo its external side effects.

When it breaks

A crash after an external effect but before acknowledgement leaves work eligible for another delivery. Retrying can repeat that effect. Acknowledging before the effect instead risks losing required work. Use stable business identifiers and transactional deduplication where possible; a delivery counter records delivery attempts, not completed database transactions. Bound retries and preserve failing payloads before retiring work.

A PEL reference does not pin the stream payload. XDEL and default KEEPREF trimming can remove it while leaving a pending ID. XAUTOCLAIM reports and clears such missing records on Redis 7 and later; that is cleanup, not recovery of their contents. Choose retention with consumer outages in mind. Stream persistence and replication also require separate durability decisions.

Monitor unread lag, pending count, idle time, and repeated deliveries together. One count cannot establish queue health. NOACK omits pending tracking for new deliveries and is unsuitable when that tracking is required for recovery. Parallel workers can finish out of insertion order even when the stream IDs themselves remain ordered.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0. The commands build Redis core with libc and install the client in a temporary virtual environment, without optional modules or system-wide installation.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It creates a private Unix socket under /tmp, disables TCP, scheduled snapshots, and append-only persistence, then terminates only its own process and removes its runtime directory. Downloads require network access. No default local Redis is contacted.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", "noeviction",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


with instance() as client:
 for entry_id in ("1-0", "2-0", "3-0"):
 client.xadd("jobs", {"job": entry_id}, id=entry_id)
 client.xgroup_create("jobs", "workers", id="0-0")
 client.xgroup_create("jobs", "tail", id="$")
 assert client.xpending("jobs", "workers")["pending"] == 0
 assert client.xreadgroup("tail", "observer", {"jobs": ">"}) == []
 first = client.xreadgroup("workers", "w1", {"jobs": ">"}, count=1)[0][1]
 second = client.xreadgroup("workers", "w2", {"jobs": ">"}, count=1)[0][1]
 assert first[0][0] == b"1-0" and second[0][0] == b"2-0"
 assert client.xpending("jobs", "workers")["pending"] == 2
 group = next(item for item in client.xinfo_groups("jobs") if item["name"] == b"workers")
 assert group["last-delivered-id"] == b"2-0"
 print("backlog starts with PEL=0; new deliveries split 1-0/2-0 and advance the group position")

 effects = [first[0][0]]
 assert client.xack("jobs", "workers", "2-0") == 1
 assert client.xlen("jobs") == 3
 group = next(item for item in client.xinfo_groups("jobs") if item["name"] == b"workers")
 assert group["last-delivered-id"] == b"2-0"
 assert client.xreadgroup("workers", "w2", {"jobs": "0-0"}) == [[b"jobs", []]]
 assert client.xreadgroup("workers", "w1", {"jobs": "0-0"})[0][1] == first
 third = client.xreadgroup("workers", "w2", {"jobs": ">"})[0][1]
 assert third[0][0] == b"3-0"
 assert client.xack("jobs", "workers", "3-0") == 1
 assert client.xreadgroup("workers", "w2", {"jobs": ">"}) == []
 print("XACK keeps stream data and read position; 0-0 reads own pending; > does not recover w1")

 cursor, claimed, deleted = client.xautoclaim(
 "jobs", "workers", "w2", min_idle_time=0, start_id="0-0", count=10,
 )
 assert cursor == b"0-0" and claimed == first and deleted == []
 pending = client.xpending_range("jobs", "workers", "-", "+", 10)
 assert len(pending) == 1 and pending[0]["consumer"] == b"w2"
 assert pending[0]["times_delivered"] >= 2
 effects.append(claimed[0][0])
 assert effects.count(b"1-0") == 2
 assert client.xack("jobs", "workers", "1-0") == 1
 assert client.xack("jobs", "workers", "1-0") == 0
 assert client.xpending("jobs", "workers")["pending"] == 0
 assert client.xlen("jobs") == 3
 print("explicit claim changes owner; simulated side effect repeats; ACK clears PEL, not the stream")

 client.xadd("jobs", {"job": "4-0"}, id="4-0")
 assert client.xreadgroup("tail", "observer", {"jobs": ">"})[0][1][0][0] == b"4-0"
 assert client.xreadgroup("workers", "w1", {"jobs": ">"})[0][1][0][0] == b"4-0"
 assert client.xdel("jobs", "4-0") == 1
 assert client.xpending("jobs", "workers")["pending"] == 1
 cursor, claimed, deleted = client.xautoclaim(
 "jobs", "workers", "w2", min_idle_time=0, start_id="0-0", count=10,
 )
 assert cursor == b"0-0" and claimed == [] and deleted == [b"4-0"]
 assert client.xpending("jobs", "workers")["pending"] == 0
 print("a $ group starts after old entries; XDEL loses pending payload; claim reports deleted ID")

The workers group begins with three unread entries and zero pending entries. Two reads distribute 1-0 and 2-0; the group position already reaches 2-0 before acknowledgement. Acknowledging the second entry leaves stream length and that position unchanged. Pending-history reads distinguish the two consumers, and a new-entry read returns 3-0 instead of retrying 1-0.

The effects list models a completed action without its acknowledgement. Explicit claiming moves 1-0 to w2, and processing it again produces two recorded effects. This is a simulation of duplicate business processing, not a transaction-safe implementation. Acknowledgement then clears the PEL while leaving the three stream records available.

The tail group sees only a later 4-0 entry. Deleting that entry while pending makes the subsequent claim return its ID in the deleted list without a payload. The checks exercise command-level state changes, not process-crash detection, automatic retry scheduling, durable deduplication, or failover recovery.

Sources

Redis XGROUP CREATE, group starting positions, checked 2026-09-08.
Redis XREADGROUP, new and pending reads, NOACK, and optional CLAIM behavior, checked 2026-09-08.
Redis XPENDING, pending ownership and delivery counts, checked 2026-09-08.
Redis XACK, acknowledgement and PEL removal, checked 2026-09-08.
Redis XAUTOCLAIM, recovery cursors, idle thresholds, and deleted entries, checked 2026-09-08.
Redis XTRIM, retention and consumer-group reference policies, checked 2026-09-08.
Redis persistence, RDB and AOF durability tradeoffs, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only compilation and allocator selection, checked 2026-09-08.

Redis streams track a consumer's own cursor

Tue, 13 Jul 2027 00:00:00 +0000

A Redis stream does not automatically track a separate cursor for every connection. Ordinary XREAD uses a position supplied by the application. A consumer group instead owns a server-side last-delivered position plus pending delivery records. Confusing those two forms of progress can make a restart skip unfinished work or replay effects already completed.

The group advances on delivery

XGROUP CREATE with 0-0 starts before retained history; $ starts after the current end. Each group maintains independent delivery state over the same stream records. Billing can have delivered two entries while analytics has delivered only one. Within a group, named consumers share new work rather than each receiving every entry.

For ordinary XREADGROUP without CLAIM or NOACK, > requests entries beyond the group’s delivery position and adds deliveries to its Pending Entries List, or PEL. That position advances before processing and acknowledgement. XACK removes specified pending IDs from one group, not stream records, and does not advance the group’s last-delivered ID.

Acknowledgements can arrive out of order. Finishing the second entry says nothing about whether the first finished. The group can have reached the end of the stream while older deliveries remain pending. Do not reconstruct successful processing from the largest delivered or acknowledged ID without an application contract that actually guarantees contiguous completion.

Pending history belongs to a consumer

Using 0-0 with XREADGROUP requests this named consumer’s pending history, not an unrestricted replay of the stream. A newly named worker sees none of another worker’s pending entries. Restarting under a different name and reading only > therefore does not recover abandoned deliveries. It can make the service look active while older work never resumes.

Recovery requires reading the original consumer’s pending history or explicitly transferring eligible work, for example with XAUTOCLAIM. Its idle threshold must account for legitimate processing time. Current Redis also supports optional XREADGROUP CLAIM; this exercise does not enable it. Ownership transfer cannot prevent a late worker from repeating an external action, so processing still needs an appropriate deduplication policy.

Use XINFO GROUPS to compare last-delivered-id, pending, and lag. Lag estimates entries awaiting delivery from the group’s current position, not unfinished business actions. It can be unavailable after certain position changes or deletions; a null metric is not zero. A group with zero lag and nonzero pending work has reached the delivery end but still has outstanding acknowledgements.

When it breaks

XGROUP SETID is a deliberate change to the group’s delivery position, not a harmless acknowledgement repair. Rewinding after completed work makes retained, previously acknowledged entries eligible for new delivery again. Moving forward can skip unread entries. It does not automatically empty existing PEL records, and neither direction changes another group’s position.

Plan a repositioning operation with workers coordinated and business effects understood. Replaying into a separate group isolates delivery bookkeeping, but not external side effects if it calls the same handler. An acknowledgement is not a permanent tombstone that forbids all future reads of that record. Avoid using a rewind as a substitute for diagnosing the pending entries.

Independent group positions do not provide independent retention. Deleting or trimming a shared payload affects every group that might need it. Default KEEPREF trimming may leave pending IDs without data. Durability across server failure is also separate from the cursor mechanism. A recorded delivery position is neither proof that the payload survives nor proof that its business work completed.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0. The commands compile Redis core with libc and install its client in a temporary virtual environment, without optional modules or system-wide installation.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It creates a private Unix socket under /tmp, disables TCP, scheduled snapshots, and append-only persistence, then stops only its own process and removes its runtime directory. Downloads require network access. No default local Redis is contacted.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", "noeviction",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


def groups(client):
 return {group["name"]: group for group in client.xinfo_groups("events")}


with instance() as client:
 for entry_id in ("1-0", "2-0", "3-0"):
 client.xadd("events", {"event": entry_id}, id=entry_id)
 for name in ("billing", "analytics"):
 client.xgroup_create("events", name, id="0-0")
 billing = client.xreadgroup("billing", "b1", {"events": ">"}, count=2)[0][1]
 analytics = client.xreadgroup("analytics", "a1", {"events": ">"}, count=1)[0][1]
 assert [entry[0] for entry in billing] == [b"1-0", b"2-0"]
 assert [entry[0] for entry in analytics] == [b"1-0"]
 state = groups(client)
 assert state[b"billing"]["last-delivered-id"] == b"2-0"
 assert state[b"analytics"]["last-delivered-id"] == b"1-0"
 assert state[b"billing"]["pending"] == 2 and state[b"analytics"]["pending"] == 1
 assert state[b"billing"]["lag"] == 1 and state[b"analytics"]["lag"] == 2
 print("two groups read overlapping records but retain independent positions and PELs")

 assert client.xack("events", "billing", "2-0") == 1
 state = groups(client)
 assert state[b"billing"]["last-delivered-id"] == b"2-0"
 assert state[b"billing"]["pending"] == 1 and state[b"analytics"]["pending"] == 1
 assert client.xreadgroup("billing", "b2", {"events": "0-0"}) == [[b"events", []]]
 own = client.xreadgroup("billing", "b1", {"events": "0-0"})[0][1]
 assert [entry[0] for entry in own] == [b"1-0"]
 next_entry = client.xreadgroup("billing", "b2", {"events": ">"})[0][1]
 assert [entry[0] for entry in next_entry] == [b"3-0"]
 assert groups(client)[b"billing"]["lag"] == 0
 assert groups(client)[b"billing"]["pending"] == 2
 assert client.xack("events", "billing", "1-0", "3-0") == 2
 assert groups(client)[b"billing"]["pending"] == 0
 assert client.xpending("events", "analytics")["pending"] == 1
 assert client.xlen("events") == 3
 print("ACK affects only one group's PEL; a new consumer sees no other owner's pending history")

 client.xgroup_setid("events", "billing", "0-0")
 replay = client.xreadgroup("billing", "b3", {"events": ">"}, count=1)[0][1]
 assert [entry[0] for entry in replay] == [b"1-0"]
 assert groups(client)[b"analytics"]["last-delivered-id"] == b"1-0"
 assert client.xack("events", "billing", "1-0") == 1
 client.xgroup_setid("events", "analytics", "$")
 state = groups(client)
 assert state[b"analytics"]["last-delivered-id"] == b"3-0"
 assert state[b"analytics"]["pending"] == 1
 assert state[b"analytics"]["lag"] == 0
 assert client.xreadgroup("analytics", "a1", {"events": ">"}) == []
 own = client.xreadgroup("analytics", "a1", {"events": "0-0"})[0][1]
 assert [entry[0] for entry in own] == [b"1-0"]
 assert client.xack("events", "analytics", "1-0") == 1
 print("SETID rewinds acknowledged data for replay; moving forward skips unread data but keeps PEL")

Billing reads two entries and analytics reads one, producing distinct delivery positions and pending counts. Acknowledging billing’s second entry changes only its PEL. Consumer b2 cannot read b1’s pending history, but it can receive the next new entry. After all new billing entries are delivered, lag is zero while two deliveries still await acknowledgement.

Once billing work is acknowledged, explicitly rewinding its position lets b3 receive 1-0 again. Analytics remains independent. Moving analytics to $ skips its unread entries but preserves the pending 1-0 record, which its original consumer can still read and acknowledge.

All state comparisons use actual Redis replies; no timing assumptions or concurrent workers are needed. The experiment verifies group isolation, delivery positions, pending ownership, and deliberate repositioning. It does not implement crash detection, recovery scheduling, persistence testing, or transaction-safe business processing.

Sources

Redis XREAD, positions supplied by independent readers, checked 2026-09-08.
Redis XREADGROUP, group delivery positions and consumer pending history, checked 2026-09-08.
Redis XACK, group-specific pending acknowledgement, checked 2026-09-08.
Redis XINFO GROUPS, last-delivered ID, pending count, and lag limitations, checked 2026-09-08.
Redis XGROUP SETID, deliberate repositioning of a group, checked 2026-09-08.
Redis XAUTOCLAIM, explicit recovery of eligible pending entries, checked 2026-09-08.
Redis XTRIM, payload retention and consumer-group references, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only compilation and allocator selection, checked 2026-09-08.