A Redis stream keeps events in order for consumers

Thu, 06 May 2027 00:00:00 +0000

A Redis stream stores entries under ordered IDs so readers can revisit retained history. Unlike Pub/Sub, the reader need not be online when the producer appends. That difference provides replay, not unconditional durability: deleted entries, retention limits, and persistence settings still determine what remains available after delay or failure.

Separate entry order from event time

XADD with * generates an ID containing a millisecond component and a sequence component. IDs increase within one stream, including when Redis must compensate for a backward clock movement. Explicit IDs must also advance beyond the stream’s last generated ID. Compare their numeric components, not arbitrary string order.

The ID orders insertion into that stream. It does not establish a global order across independent streams or prove when an external event happened. Store the business timestamp separately when needed. Two ordinary XADD calls with identical fields still create separate entries: automatic IDs identify records, not duplicate business operations.

Save the last processed position

XRANGE includes both boundaries by default. Starting it at the last ID already processed returns that entry again; an opening parenthesis makes a range boundary exclusive. XREAD instead returns entries strictly greater than the supplied ID. With COUNT, save the final returned ID and continue from that position.

Ordinary XREAD does not maintain a server-side offset for each reader or remove delivered entries. Independent readers can read the same range, and repeating a request repeats retained data. The application owns its checkpoint. Saving it before a side effect risks skipping work after failure; saving it afterward risks repeating completed work. Coordinate checkpointing with idempotent processing where that distinction matters.

A dollar ID means start after the stream’s current end when Redis handles this call. It is useful for entering a live feed, not for resuming a saved position. Reusing $ after every response can skip entries appended between calls. After receiving data, continue with the actual last ID. Consumer groups add separate delivery and acknowledgement state; XACK is not an offset commit for ordinary XREAD.

When it breaks

A reader can fall behind retention. XTRIM MAXLEN removes the oldest, lowest-ID entries and keeps newer ones; it does not trim the newest end. Exact trimming enforces the requested count, while approximate trimming may retain extra entries for efficiency. redis-py’s xtrim defaults to approximate trimming, so the exercise explicitly disables it.

XDEL removes selected records. Neither deletion nor a short retention limit becomes safe merely because a consumer has not acknowledged an entry. Default KEEPREF trimming can leave pending references without payloads. Redis 8.2 added explicit reference policies, including ACKED, but retention remains an application decision. A saved ID cannot reconstruct deleted data.

Streams are Redis data, subject to RDB/AOF configuration, replication behavior, key deletion, and possible eviction. A successful append alone does not establish lossless failover or exactly-once processing. Size retention for expected consumer outages and catch-up capacity, monitor lag, and test recovery using the actual persistence configuration. This experiment deliberately disables persistence and makes no restart-survival claim.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0. The commands build Redis core with libc and install the client in a temporary virtual environment, without optional modules or system-wide installation.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It creates a private Unix socket under /tmp, disables TCP, scheduled snapshots, and append-only persistence, then terminates only its own process and removes its runtime directory. Downloads require network access. No default local Redis is contacted.

from concurrent.futures import ThreadPoolExecutor

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", "noeviction",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


def ids(entries):
 return [entry[0] for entry in entries]


with instance() as client:
 for entry_id in ("1-0", "1-1", "2-0"):
 assert client.xadd("events", {"value": entry_id}, id=entry_id) == entry_id.encode()
 try:
 client.xadd("events", {"value": "duplicate"}, id="1-1")
 except redis.ResponseError as error:
 assert "equal or smaller" in str(error)
 else:
 raise AssertionError("XADD must reject an ID below the stream's last ID")
 generated = [client.xadd("auto", {"value": "same"}) for _ in range(2)]
 assert tuple(map(int, generated[0].split(b"-"))) < tuple(map(int, generated[1].split(b"-")))
 assert client.xlen("auto") == 2
 print("explicit IDs advance; duplicate/lower ID rejected; equal payloads get distinct auto IDs")

 assert ids(client.xrange("events", "1-1", "+")) == [b"1-1", b"2-0"]
 assert ids(client.xrange("events", "(1-1", "+")) == [b"2-0"]
 first = client.xread({"events": "0-0"}, count=2)
 second = client.xread({"events": "0-0"}, count=2)
 assert first == second and ids(first[0][1]) == [b"1-0", b"1-1"]
 saved = first[0][1][-1][0]
 remaining = client.xread({"events": saved})
 assert ids(remaining[0][1]) == [b"2-0"]
 saved = remaining[0][1][-1][0]
 print("XRANGE includes its start; XREAD excludes the saved ID; independent reads replay")

 client.xadd("events", {"value": "between-calls"}, id="3-0")
 assert client.xread({"events": "$"}, block=100) == []
 assert ids(client.xread({"events": saved})[0][1]) == [b"3-0"]

 def read_live():
 with client.client() as connection:
 connection.client_setname("stream-lab-reader")
 return connection.xread({"events": "$"}, block=1000)

 with ThreadPoolExecutor(max_workers=1) as worker:
 future = worker.submit(read_live)
 deadline = time.monotonic() + 1
 while not any(
 entry["name"] == "stream-lab-reader" and "b" in entry["flags"]
 for entry in client.client_list()
 ):
 assert time.monotonic() < deadline, "reader never blocked"
 time.sleep(0.01)
 client.xadd("events", {"value": "live"}, id="4-0")
 assert ids(future.result(timeout=2)[0][1]) == [b"4-0"]
 print("$ skips existing entries; a saved ID catches up; observed blocking receives a new entry")

 assert client.xdel("events", "2-0") == 1
 assert client.xrange("events", "2-0", "2-0") == []
 assert client.xtrim("events", maxlen=2, approximate=False) == 2
 assert ids(client.xrange("events")) == [b"3-0", b"4-0"]
 assert ids(client.xread({"events": "0-0"})[0][1]) == [b"3-0", b"4-0"]
 print("XDEL removes a record; exact MAXLEN keeps the newest two; old IDs cannot restore deleted data")

Explicit IDs produce a deterministic three-entry log. An older ID fails, while two automatic appends with the same payload produce increasing distinct IDs. The boundary assertions show XRANGE including 1-1 and XREAD continuing strictly after it. Two independent reads from zero return identical data without consuming it.

An entry added before a $ call is skipped and that call times out; reading from the saved ID finds it. For live delivery, the producer waits until CLIENT LIST reports the named reader as blocked before appending 4-0. The example asserts the received record, not exact scheduler timing.

Finally, XDEL removes 2-0 and exact MAXLEN keeps only 3-0 and 4-0. Reading from zero now returns those retained records alone. These results demonstrate ID order, cursor boundaries, live waiting, and retention loss. They do not validate crash durability, consumer-group recovery, cross-stream ordering, or production throughput.

Sources