Pubsub · XB Field Notes

Redis pub/sub and streams are different tools

Sun, 06 Feb 2028 00:00:00 +0000

Redis Pub/Sub and Streams answer different delivery questions. Pub/Sub broadcasts to currently active subscriptions without retaining messages for later replay. A stream stores entries that readers can revisit while they remain retained. Choosing between them starts with who must receive an event and what should happen during a consumer’s absence.

Decide who receives each event

For a replaceable live dashboard update, broadcasting to connected viewers may be sufficient. Wait for subscription confirmation before assuming a new listener is active. Reconnecting restores future delivery, not missed publications. A positive PUBLISH response counts recipients according to Redis’s command semantics; it does not acknowledge application processing.

Streams offer two distinct consumption models. Ordinary XREAD lets independent readers request entries after their own supplied IDs, so both can receive the same retained history. It does not divide work or maintain an automatic checkpoint per connection. XREADGROUP within one consumer group distributes new deliveries among named workers and records pending entries unless NOACK is used.

Separate consumer groups allow separate workflows, such as billing and analytics, to progress independently over one stream. Membership within a group shares work; adding a new group creates another delivery history. Neither choosing unique consumer names nor adding workers promises equal load or business completion order. Decide whether the requirement is broadcast, independent replay, or competing work before choosing the command.

Retention is not a durability setting

XADD retains data in Redis; that does not by itself make every accepted append survive a crash or failover. RDB/AOF settings, replication behavior, key eviction, and operational recovery determine durability. Pub/Sub does not acquire a replay log by enabling persistence. Calling the difference simply latency versus durability hides both contracts.

MAXLEN trims the oldest entries, not the newest tail while preserving older history. Approximate trimming may retain extra records, while exact trimming follows the requested count. Neither makes unlimited replay possible. Default trimming can remove payloads still referenced by pending entries; select retention policies and capacity for the longest supported outage and subsequent catch-up.

Stored backlog also has an operational cost. Bound its growth, monitor unread lag and pending work separately, and define what happens to poison messages. Pub/Sub instead avoids retained history but still uses connection output buffers. A slow subscriber can be disconnected when those limits are exceeded. Neither mechanism eliminates memory planning.

When it breaks

Using Pub/Sub for required jobs silently loses the opportunity to deliver while no listener is subscribed. A dashboard that can reload authoritative state may tolerate that; a required financial action usually cannot. The appropriate replacement might be a retained stream or another durable queue, with recovery semantics designed around the action rather than assumed from the product name.

Using Streams without a recovery loop creates a different failure: an unacknowledged group delivery can remain pending while ordinary > reads continue with new entries. Acknowledging before processing risks loss; processing before acknowledgement permits duplicate effects after retry. Stable event identifiers and idempotent handling remain necessary when repeated business actions are unacceptable.

Publishing and appending the same event as separate commands is not an atomic dual-write protocol. One operation can succeed without the other if execution stops between them. The exercise deliberately compares two mechanisms in an isolated process; it does not propose that sequence as a production delivery guarantee.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0. The commands compile Redis core with libc and install its client in a temporary virtual environment, without optional modules or system-wide installation.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It creates a private Unix socket under /tmp, disables TCP, scheduled snapshots, and append-only persistence, then stops only its own process and removes its runtime directory. Downloads require network access. No default local Redis is contacted.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", "noeviction",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


def receive(subscription):
 deadline = time.monotonic() + 2
 while time.monotonic() < deadline:
 response = subscription.get_message(timeout=0.1)
 if response is not None:
 return response
 raise AssertionError("subscription response timed out")


with instance() as client:
 assert client.publish("updates", b"offline") == 0
 first_id = client.xadd("history", {"event": "offline"}, id="1-0")
 with client.pubsub() as first, client.pubsub() as second:
 for subscription in (first, second):
 subscription.subscribe("updates")
 assert receive(subscription)["type"] == "subscribe"
 assert client.publish("updates", b"live") == 2
 second_id = client.xadd("history", {"event": "live"}, id="2-0")
 for subscription in (first, second):
 response = receive(subscription)
 assert response["type"] == "message" and response["data"] == b"live"
 with client.client() as reader_one, client.client() as reader_two:
 one = reader_one.xread({"history": "0-0"})
 two = reader_two.xread({"history": "0-0"})
 assert one == two
 assert [entry[0] for entry in one[0][1]] == [first_id, second_id]
 print("Pub/Sub listeners get live only; both independent XREAD readers replay offline and live")

 client.xgroup_create("history", "workers", id="0-0")
 a = client.xreadgroup("workers", "a", {"history": ">"}, count=1)[0][1]
 b = client.xreadgroup("workers", "b", {"history": ">"}, count=1)[0][1]
 assert a[0][0] == first_id and b[0][0] == second_id
 assert client.xpending("history", "workers")["pending"] == 2
 assert client.xack("history", "workers", first_id, second_id) == 2
 assert client.xlen("history") == 2
 print("one group splits new entries between a/b; ACK clears pending state, not stored history")

 third_id = client.xadd("history", {"event": "later"}, id="3-0")
 assert client.xtrim("history", maxlen=2, approximate=False) == 1
 retained = client.xread({"history": "0-0"})[0][1]
 assert [entry[0] for entry in retained] == [second_id, third_id]
 assert client.xrange("history", first_id, first_id) == []
 print("exact MAXLEN removes the oldest entry; replay is limited to retained history")

The offline publication returns zero, while XADD keeps an entry with the same event label. Once both subscriptions are confirmed, each listener receives live, not offline. Two ordinary XREAD calls then return both stored entries, proving independent replay rather than division of work.

Creating one group changes that delivery pattern: workers a and b receive different entries, and both become pending. Acknowledging them leaves the stream length unchanged. A third append followed by exact MAXLEN removes only the oldest retained entry; reading from zero cannot recover its missing payload.

The three output summaries follow assertions on actual responses. They demonstrate subscription presence, independent replay, grouped distribution, and finite retention. This run does not benchmark latency or memory, reproduce a real crash, validate persistence, or test automatic recovery. Those operational choices remain part of the application’s delivery contract.

Sources

Redis Pub/Sub, live delivery, subscription acknowledgement, and absent replay, checked 2026-09-08.
Redis XREAD, independent reads and caller-supplied positions, checked 2026-09-08.
Redis XREADGROUP, shared work and pending entries, checked 2026-09-08.
Redis XTRIM, oldest-entry removal and reference policies, checked 2026-09-08.
Redis client handling, Pub/Sub output buffering and disconnect limits, checked 2026-09-08.
Redis persistence, RDB and AOF recovery tradeoffs, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only compilation and allocator selection, checked 2026-09-08.

Pub/sub delivers a message to everyone listening

Wed, 17 Feb 2027 00:00:00 +0000

Redis Pub/Sub broadcasts to subscriptions that are active when a message is published. It does not retain an inbox for a disconnected listener. This suits replaceable live notifications, but a positive PUBLISH response is not evidence that a worker finished a job or that an offline consumer can recover it.

Subscription is a handshake

SUBSCRIBE produces a subscription confirmation before that connection starts receiving subsequent publications. Wait for this response when coordinating a publisher and a new listener. Sending the subscription request is not enough: another connection can publish before Redis has processed it. Reconnecting and subscribing again restores future delivery, not missing history.

Within the live connection, subscribers receive messages in publication order. That order reflects Redis processing, not business timestamps from independent producers. Dispatching received events to parallel handlers can also reorder completion. Saying Pub/Sub has no ordering is incorrect; saying ordered delivery guarantees ordered business effects is equally misleading.

Multiple subscribers each receive the broadcast, unlike workers competing for one list item. A channel subscription and a matching pattern subscription are separate delivery paths. Subscribing to events and event* can produce both a message and a pmessage for one publication. Overlapping patterns can do the same. At-most-once delivery describes the lack of retry, not application-level deduplication across subscriptions.

Interpret the response narrowly

On the tested standalone server, PUBLISH counts matching deliveries, including channel and pattern matches separately. Two client connections can therefore produce a return value of three. The pinned server implementation increments the counter along both paths. In Redis Cluster, the documented count includes only subscribers connected to the publishing node, so it is not a cluster-wide audience census.

No response waits for a handler to commit a database transaction, render an update, or acknowledge completion. A zero response means no matching recipients were counted at that moment; a positive response cannot prove business success. Put a stable event identifier in messages when the application must recognize duplicates caused by overlapping subscriptions or repeated producer attempts.

When it breaks

A subscription gap loses publications made during that gap. A reconnect loop cannot request an offset because Pub/Sub has no retained log or replay cursor. For cache invalidation, losing a notification can leave a stale value until another mechanism refreshes it. For billing or job dispatch, that gap can lose required work. Choose a retained queue or stream with explicit recovery and appropriate persistence when replay is required.

A slow subscriber is not immediately handled by arbitrarily dropping individual reads. Redis buffers outgoing data; configured hard or sustained soft output-buffer limits can close the connection. That prevents unbounded client buffering but creates an unrecoverable subscription gap. Observe connection churn and buffer usage, keep callbacks bounded, and do not treat increasing buffer limits as a delivery guarantee.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0. The commands build Redis core with libc and install the client in a temporary virtual environment, without optional modules or system-wide installation.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It creates a private Unix socket under /tmp, disables TCP, scheduled snapshots, and append-only persistence, then terminates only its own process and removes its runtime directory. Downloads require network access. No default local Redis is contacted.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", "noeviction",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


def receive(subscription):
 deadline = time.monotonic() + 2
 while time.monotonic() < deadline:
 message = subscription.get_message(timeout=0.1)
 if message is not None:
 return message
 raise AssertionError("subscription response timed out")


with instance() as client:
 assert client.publish("events", b"before-subscribe") == 0
 with client.pubsub() as first, client.pubsub() as second:
 for subscription in (first, second):
 subscription.subscribe("events")
 response = receive(subscription)
 assert response["type"] == "subscribe"
 assert response["channel"] == b"events" and response["data"] == 1

 for value in (b"one", b"two"):
 assert client.publish("events", value) == 2
 for subscription in (first, second):
 messages = [receive(subscription) for _ in range(2)]
 assert [message["type"] for message in messages] == ["message"] * 2
 assert [message["data"] for message in messages] == [b"one", b"two"]
 print("before subscription: 0; two listeners receive one,two in order")

 second.unsubscribe("events")
 response = receive(second)
 assert response["type"] == "unsubscribe" and response["data"] == 0
 assert client.publish("events", b"missed") == 1
 assert receive(first)["data"] == b"missed"
 second.subscribe("events")
 assert receive(second)["type"] == "subscribe"
 assert client.publish("events", b"after-resubscribe") == 2
 for subscription in (first, second):
 assert receive(subscription)["data"] == b"after-resubscribe"
 print("resubscription receives the new marker, not the missed message")

 first.psubscribe("event*")
 response = receive(first)
 assert response["type"] == "psubscribe" and response["data"] == 2
 assert client.publish("events", b"overlap") == 3
 messages = [receive(first) for _ in range(2)]
 assert {message["type"] for message in messages} == {"message", "pmessage"}
 assert all(message["data"] == b"overlap" for message in messages)
 assert receive(second)["data"] == b"overlap"
 print("two clients, three matching deliveries; one client receives two copies")

The first publication returns zero before either listener subscribes. After both confirmations, two sequential publications reach both listeners as one, two. The example checks message types as well as payload order, so a subscription confirmation cannot accidentally count as a business event. It uses bounded response waits rather than guessing when subscriptions become active.

The second listener explicitly unsubscribes while missed is published, then resubscribes. Its next payload is after-resubscribe, not the missing event. This models a subscription gap without claiming to reproduce a network failure. Finally, overlapping channel and pattern subscriptions yield three counted deliveries across two connections, with two copies at the first listener.

All three printed summaries follow assertions against actual replies. The exercise verifies local command semantics and resubscription behavior, not overload thresholds, cluster propagation, crash recovery, or processing throughput. An application still needs to decide which notifications are expendable and how authoritative state will be refreshed when one never arrives.

Sources

Redis Pub/Sub, subscription acknowledgements, ordering, at-most-once delivery, and overlapping subscriptions, checked 2026-09-08.
Redis PUBLISH, command response and cluster counting scope, checked 2026-09-08.
Redis client handling, output buffering and disconnection limits, checked 2026-09-08.
Redis 8.10.1 Pub/Sub implementation, separate channel and pattern receiver counting, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only compilation and allocator selection, checked 2026-09-08.