Cluster · XB Field Notes

Redis Cluster splits keys across slots

Fri, 26 Nov 2027 00:00:00 +0000

Redis Cluster separates key placement from machine identity. A key maps to one of 16,384 fixed hash slots; cluster metadata maps those slots to primaries and their replicas. Adding a machine does not change the slot hash function. Operators redistribute slots, while clients discover the current topology and react to routing changes.

Keys choose slots, not contiguous machine ranges

The ordinary mapping is CRC16/XMODEM of the key modulo 16,384. A nonempty hash tag between the first opening brace and its following closing brace makes Redis hash that substring instead. Thus profile:{42} and quota:{42} share a slot even though their full names differ. Tags are placement instructions, not authorization boundaries.

A primary can own several noncontiguous slot ranges. In a stable topology each slot has an authoritative primary, and a whole key belongs to that slot: a large hash or sorted set is not automatically divided between nodes. CLUSTER SHARDS reports shard membership and slot ranges. Do not confuse “same primary” with “same slot”; they are different conditions.

Clients participate in routing

A cluster-aware client maintains a slot map and connects to the advertised nodes. Contacting the wrong primary can produce MOVED with a destination, not a missing value. MOVED tells the client to update routing; ASK is a temporary migration redirection that requires ASKING before the redirected command, without permanently assigning ownership to that target.

Migration is not an instant, globally synchronized map update. Clients may observe different topology information, and some multi-key operations can return TRYAGAIN while keys are split during migration. Use a maintained cluster client rather than assuming a single endpoint forwards every request. Advertised addresses must be reachable from the application’s network, not only between Redis processes.

Placement does not imply transaction scope

Server-side multi-key commands such as MSET require their keys to share one slot. Merely landing on the same physical primary is insufficient. Hash tags can satisfy the slot constraint, but putting every tenant under one tag concentrates load and limits distribution. A client operation implemented as several independent commands is not equivalent to one atomic multi-key command.

Sharding distributes ownership; replication creates copies for a shard. Replica reads can be stale because replication is asynchronous. Failover promotes a replica for the affected shard, not the whole cluster as one indivisible unit. Acknowledged writes can still be lost during failures, and connection loss can leave a write’s outcome unknown to its caller.

When it breaks

Missing slot coverage and insufficient reachable primaries can prevent service. With cluster-require-full-coverage yes, an unavailable shard can stop key commands even on otherwise healthy primaries. Three primaries without replicas demonstrate distribution, but cannot replace a failed primary’s data with a replica that does not exist.

Watch slot coverage, cluster state, replica health, address announcements, and client redirection errors together. Adding nodes alone does not automatically move existing keys or split a hot key. Resharding, replica placement, and client retry semantics are separate operational decisions. Test the failure you intend to tolerate instead of treating a successful cluster handshake as a high-availability result.

Check it quickly

Use a POSIX machine with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Redis 8.10.1, redis-py 8.1.0, and Python 3.14.4. Build Redis core with libc and install the client in a temporary virtual environment.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python". It starts three isolated primaries using temporary directories and six loopback-only ports for data and cluster-bus traffic. Private Unix sockets identify the created processes before configuration. Persistence is disabled. The final failure targets only its own first primary; cleanup stops the remaining processes and removes their temporary files.

from contextlib import contextmanager, ExitStack
import os
from pathlib import Path
import signal
import socket
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.crc import key_slot
from redis.exceptions import ClusterCrossSlotError, ClusterDownError, MovedError
from redis.retry import Retry


def until(check):
 deadline = time.monotonic() + 15
 while not check():
 assert time.monotonic() < deadline, "cluster state did not converge"
 time.sleep(0.05)


@contextmanager
def node(directory, port, bus):
 directory.mkdir()
 endpoint = str(directory / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--bind", "127.0.0.1", "--port", str(port),
 "--protected-mode", "yes", "--unixsocket", endpoint,
 "--unixsocketperm", "700", "--dir", str(directory),
 "--save", "", "--appendonly", "no", "--cluster-enabled", "yes",
 "--cluster-config-file", "nodes.conf", "--cluster-port", str(bus),
 "--cluster-announce-ip", "127.0.0.1", "--cluster-node-timeout", "1000",
 "--cluster-require-full-coverage", "yes",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False, protocol=2,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 assert time.monotonic() < deadline, "Redis did not start"
 time.sleep(0.02)
 assert client.info("server")["process_id"] == process.pid
 yield client, process
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode in (0, -signal.SIGKILL), errors.decode()


with TemporaryDirectory(prefix="redis-cluster-", dir="/tmp") as directory, ExitStack() as stack:
 leases = [stack.enter_context(socket.socket()) for _ in range(6)]
 for lease in leases:
 lease.bind(("127.0.0.1", 0))
 ports = [lease.getsockname()[1] for lease in leases]
 nodes = []
 for index in range(3):
 leases[2 * index].close()
 leases[2 * index + 1].close()
 nodes.append(stack.enter_context(node(
 Path(directory) / str(index), ports[2 * index], ports[2 * index + 1],
 )))
 clients = [item[0] for item in nodes]
 ranges = [[(0, 4095), (8192, 12287)], [(4096, 8191)], [(12288, 16383)]]
 for client, owned in zip(clients, ranges):
 bounds = [value for interval in owned for value in interval]
 assert client.execute_command("CLUSTER", "ADDSLOTSRANGE", *bounds) == b"OK"
 for index in (1, 2):
 assert clients[0].execute_command(
 "CLUSTER", "MEET", "127.0.0.1", ports[2 * index], ports[2 * index + 1],
 ) == b"OK"
 until(lambda: all(b"cluster_state:ok" in client.execute_command("CLUSTER", "INFO")
 for client in clients))
 shards = [dict(zip(item[::2], item[1::2], strict=True))
 for item in clients[0].execute_command("CLUSTER", "SHARDS")]
 assert len(shards) == 3
 assert sorted(interval for shard in shards
 for interval in zip(shard[b"slots"][::2], shard[b"slots"][1::2])) == [
 (0, 4095), (4096, 8191), (8192, 12287), (12288, 16383),
 ]
 first_id = clients[0].execute_command("CLUSTER", "MYID")
 first_shard = next(shard for shard in shards
 if first_id in [dict(zip(item[::2], item[1::2]))[b"id"]
 for item in shard[b"nodes"]])
 assert first_shard[b"slots"] == [0, 4095, 8192, 12287]
 print("16384 slots are covered; one primary owns two noncontiguous ranges")

 keys = [b"profile:{42}", b"quota:{42}"]
 slot = key_slot(keys[0])
 assert key_slot(keys[1]) == slot
 assert clients[0].execute_command("CLUSTER", "KEYSLOT", keys[0]) == slot
 owner = next(index for index, owned in enumerate(ranges)
 if any(start <= slot <= end for start, end in owned))
 wrong = clients[(owner + 1) % 3]
 try:
 wrong.get(keys[0])
 except MovedError as error:
 assert error.slot_id == slot
 assert error.node_addr == ("127.0.0.1", ports[2 * owner])
 else:
 raise AssertionError("a plain client on the wrong primary must receive MOVED")

 routed = redis.RedisCluster(host="127.0.0.1", port=ports[0],
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 assert routed.mset({keys[0]: "Ada", keys[1]: "10"})
 assert routed.mget(keys) == [b"Ada", b"10"]
 print("MOVED names the owner; a cluster client routes a same-slot MSET and MGET")
 other = b"unrelated:2"
 assert key_slot(other) != slot
 assert any(start <= key_slot(other) <= end for start, end in ranges[owner])
 try:
 clients[owner].execute_command("MSET", keys[0], "bad", other, "bad")
 except ClusterCrossSlotError:
 pass
 else:
 raise AssertionError("cross-slot MSET must fail")
 assert routed.get(keys[0]) == b"Ada"
 assert routed.get(other) is None
 print("cross-slot MSET fails even on the same primary, without changing either key")
 finally:
 routed.close()

 assert nodes[0][1].poll() is None
 nodes[0][1].kill()
 assert nodes[0][1].wait(timeout=5) == -signal.SIGKILL
 until(lambda: all(b"cluster_state:fail" in client.execute_command("CLUSTER", "INFO")
 for client in clients[1:]))
 try:
 clients[1].get(b"any-key")
 except ClusterDownError:
 print("without replicas, a missing primary stops key service under full coverage")
 else:
 raise AssertionError("an uncovered cluster must reject key commands")

CLUSTER SHARDS confirms complete coverage and two disjoint ranges on one primary. A plain client receives MOVED, while the cluster client reads and writes the tagged pair. A second key on that same primary but in another slot makes MSET fail without changing either value. Killing a primary then produces cluster failure and rejected key commands.

The fixture uses RESP2 for directly inspected topology replies and explicit local bus ports. It is a local routing and failure-detection test, not a production topology: there are no replicas or independent machines. ASK migration, actual resharding, replica promotion, network partitions, persistence, TLS, authentication, and production throughput are not exercised here.

Sources

Redis Cluster specification, slot hashing, hash tags, routing, migration, and write safety, checked 2026-09-08.
Redis Cluster scaling, sharding, replication, and deployment boundaries, checked 2026-09-08.
Redis CLUSTER SHARDS, shard membership and noncontiguous slot ranges, checked 2026-09-08.
Redis CLUSTER ADDSLOTSRANGE, assigning slot ranges in an empty fixture, checked 2026-09-08.
redis-py 8.1.0 cluster client, routing and client configuration, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only build and allocator selection, checked 2026-09-08.

Namespaces scope objects; policies establish other boundaries

Wed, 25 Aug 2027 00:00:00 +0000

A namespace scopes many Kubernetes objects within one cluster. Two teams can each create a ConfigMap named settings because those objects have different namespace-qualified identities. This organizational boundary is useful, but it does not automatically provide network isolation, a dedicated kernel, reserved node capacity, or complete tenant separation. Identify which additional policies enforce the boundaries your workload actually requires.

Names are scoped by resource type

Pods, Services, ConfigMaps, and many workload controllers are namespaced. Nodes, namespaces themselves, and other cluster-level resources are not divided into separate copies merely because a client specifies a namespace. Check the resource’s API scope rather than assuming every object lives inside a room. Namespaces also do not form a nested hierarchy of independent child clusters.

Within a Pod, a short Service name normally resolves in the Pod’s namespace. A qualified name can target a Service elsewhere, such as api.other.svc.cluster.local in a cluster using the default domain. This naming convention locates a destination; it does not authorize or prohibit the connection. The cluster domain can be configured differently, so do not mistake the example suffix for a universal constant.

Namespace choice must remain explicit in automation. A correct object name combined with the wrong active namespace still addresses the wrong resource. Use deliberate kubeconfig, context, and namespace inputs, especially for mutation. Namespace scoping makes identity clearer, but an administrator with broad permissions can still modify objects across those scopes.

When it breaks

A namespace without an applicable NetworkPolicy remains connected according to the cluster network’s behavior. Merely creating a namespace or qualifying a DNS name does not deny other Pods. Network isolation requires suitable policies and a network implementation that enforces them. Application authentication remains another layer; successful name resolution says nothing about who should be allowed to read a response.

RBAC controls API operations separately. A RoleBinding can grant an account access to ConfigMaps in one namespace without granting the same access elsewhere. Permissions are additive, so another broad binding can still grant additional access. Test actual allowed and denied operations for the intended identity rather than assuming that adding a narrow Role cancels wider grants.

ResourceQuota primarily enforces admission limits, including object counts and aggregate resource requests or limits. It does not itself throttle a running process or reserve physical capacity. LimitRange can supply defaults and bounds for admitted workloads, while runtime resource enforcement has other mechanisms. These policies must be configured; the namespace alone does not make a noisy neighbor harmless.

Check it quickly

Use a disposable Kubernetes cluster with the default cluster.local DNS domain, kubectl administrator access for impersonation, and python:3.11-slim-bookworm available on its nodes. Save this as namespaces.py and run python3 namespaces.py KUBECONFIG_PATH CONTEXT with explicit test configuration. The recorded cluster ran Kubernetes 1.34.11 with kind networking. The script creates two unique namespaces and removes both afterward.

import json
import subprocess
import sys
import time
from uuid import uuid4

assert len(sys.argv) == 3, 'usage: python3 example.py DISPOSABLE_KUBECONFIG CONTEXT'
base = ['kubectl', '--kubeconfig', sys.argv[1], '--context', sys.argv[2], '--request-timeout=30s']
image = 'python:3.11-slim-bookworm'


def k(*args, payload=None, expected=0):
 result = subprocess.run(base + list(args), input=json.dumps(payload) if payload is not None else None,
 capture_output=True, text=True, timeout=150)
 if expected is not None:
 assert result.returncode == expected, result.stderr
 return result


def obj(namespace, kind, name=None, selector=None):
 args = ['get', kind, '-n', namespace, '-o', 'json']
 if name:
 args.append(name)
 if selector:
 args += ['-l', selector]
 return json.loads(k(*args).stdout)


def apply(namespace, data):
 return k('apply', '-n', namespace, '-f', '-', payload=data)


def wait_for(read, ready, seconds=120):
 until = time.monotonic() + seconds
 while time.monotonic() < until:
 value = read()
 if ready(value):
 return value
 time.sleep(0.5)
 raise AssertionError('expected Kubernetes condition did not arrive')


def pod_ready(pod):
 return any(c['type'] == 'Ready' and c['status'] == 'True'
 for c in pod.get('status', {}).get('conditions', []))


def spec(script='import time; time.sleep(600)'):
 return {'automountServiceAccountToken': False, 'terminationGracePeriodSeconds': 1,
 'containers': [{'name': 'app', 'image': image, 'imagePullPolicy': 'IfNotPresent',
 'command': ['python3', '-c', script],
 'resources': {'requests': {'cpu': '10m', 'memory': '24Mi'},
 'limits': {'cpu': '200m', 'memory': '96Mi'}}}]}

suffix = uuid4().hex[:8]
first, second = 'notes-a-' + suffix, 'notes-b-' + suffix
created = []
try:
 for namespace, value in [(first, 'A'), (second, 'B')]:
 k('create', 'namespace', namespace)
 created.append(namespace)
 apply(namespace, {'apiVersion': 'v1', 'kind': 'ConfigMap', 'metadata': {'name': 'settings'}, 'data': {'scope': value}})
 script = '''from http.server import BaseHTTPRequestHandler,HTTPServer
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 data=VALUE.encode();self.send_response(200);self.send_header('Content-Length',str(len(data)));self.end_headers();self.wfile.write(data)
 def log_message(self,*args): pass
HTTPServer(('0.0.0.0',8080),Handler).serve_forever()
'''.replace('VALUE', repr(value))
 podspec = spec(script)
 podspec['containers'][0]['readinessProbe'] = {'httpGet': {'path': '/', 'port': 8080}, 'periodSeconds': 1}
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Pod', 'metadata': {'name': 'web', 'labels': {'app': 'web'}}, 'spec': podspec})
 apply(namespace, {'apiVersion': 'v1', 'kind': 'Service', 'metadata': {'name': 'api'},
 'spec': {'selector': {'app': 'web'}, 'ports': [{'port': 80, 'targetPort': 8080}]}})
 wait_for(lambda namespace=namespace: obj(namespace, 'pod', 'web'), pod_ready)
 wait_for(lambda namespace=namespace: obj(namespace, 'endpointslices', selector='kubernetes.io/service-name=api')['items'],
 lambda slices: any(e.get('conditions', {}).get('ready') for s in slices for e in s.get('endpoints', [])))
 assert obj(first, 'configmap', 'settings')['data']['scope'] == 'A'
 assert obj(second, 'configmap', 'settings')['data']['scope'] == 'B'
 nodes_a = {n['metadata']['uid'] for n in obj(first, 'nodes')['items']}
 nodes_b = {n['metadata']['uid'] for n in obj(second, 'nodes')['items']}
 assert nodes_a == nodes_b and nodes_a
 apply(first, {'apiVersion': 'v1', 'kind': 'ServiceAccount', 'metadata': {'name': 'reader'}, 'automountServiceAccountToken': False})
 apply(first, {'apiVersion': 'rbac.authorization.k8s.io/v1', 'kind': 'Role', 'metadata': {'name': 'read-config'},
 'rules': [{'apiGroups': [''], 'resources': ['configmaps'], 'verbs': ['get']}]})
 apply(first, {'apiVersion': 'rbac.authorization.k8s.io/v1', 'kind': 'RoleBinding', 'metadata': {'name': 'read-config'},
 'subjects': [{'kind': 'ServiceAccount', 'name': 'reader', 'namespace': first}],
 'roleRef': {'apiGroup': 'rbac.authorization.k8s.io', 'kind': 'Role', 'name': 'read-config'}})
 identity = 'system:serviceaccount:' + first + ':reader'
 assert k('--as', identity, 'get', 'configmap', 'settings', '-n', first).returncode == 0
 denied = k('--as', identity, 'get', 'configmap', 'settings', '-n', second, expected=None)
 assert denied.returncode != 0 and 'Forbidden' in denied.stderr
 apply(first, {'apiVersion': 'v1', 'kind': 'ResourceQuota', 'metadata': {'name': 'one-service'},
 'spec': {'hard': {'services': '1'}}})
 wait_for(lambda: obj(first, 'resourcequota', 'one-service'), lambda q: q.get('status', {}).get('used', {}).get('services') == '1')
 extra = {'apiVersion': 'v1', 'kind': 'Service', 'metadata': {'name': 'extra', 'namespace': first},
 'spec': {'ports': [{'port': 80}]}}
 quota_denial = k('create', '-f', '-', payload=extra, expected=None)
 assert quota_denial.returncode != 0 and 'exceeded quota' in quota_denial.stderr
 assert not obj(first, 'networkpolicies')['items'] and not obj(second, 'networkpolicies')['items']
 remote = 'api.' + second + '.svc.cluster.local'
 probe = '''import json,socket
from urllib.request import urlopen
print(json.dumps({'short_name':socket.gethostbyname('api'),
 'qualified_name':socket.gethostbyname(REMOTE),
 'cross_namespace_body':urlopen('http://'+REMOTE+'/',timeout=3).read().decode()}))'''.replace('REMOTE', repr(remote))
 network = json.loads(k('exec', '-n', first, 'web', '--', 'python3', '-c', probe).stdout)
 assert network['short_name'] == obj(first, 'service', 'api')['spec']['clusterIP']
 assert network['qualified_name'] == obj(second, 'service', 'api')['spec']['clusterIP']
 assert network['cross_namespace_body'] == 'B'
 print(json.dumps({'same_names_different_configmaps': True, 'nodes_are_cluster_scoped': True,
 'rbac_allows_local_and_denies_other_namespace': True,
 'quota_rejects_second_service': True, 'short_and_qualified_dns_match_services': True,
 'cross_namespace_http_succeeds_without_network_policy': True}))
finally:
 for namespace in reversed(created):
 k('delete', 'namespace', namespace, '--wait=true', '--timeout=90s')

Each namespace receives a ConfigMap named settings, a Pod named web, and a Service named api. The ConfigMaps retain different A and B values. Node queries made with either namespace return the same cluster-scoped node identities. This verifies object-name scope without inventing separate node pools.

A test service account is granted ConfigMap reads only in the first namespace. Requests impersonating it succeed there and return Forbidden in the second. A ResourceQuota then permits one Service in the first namespace; creating another is rejected after the existing usage is observed. This exercises a real admission decision rather than only checking that a quota object exists.

Finally, a request from the first Pod resolves api to its local Service, resolves the qualified second Service to that Service’s address, and receives B over HTTP. Neither namespace has a NetworkPolicy. All assertions passed. The experiment verifies naming, API authorization, one quota dimension, and network reachability. It does not demonstrate NetworkPolicy enforcement, CPU fairness, or full hostile-tenant isolation. Review and test each required boundary explicitly instead of making namespace membership stand in for all of them.

Sources

Kubernetes: Namespaces, object-name scope and cluster-scoped resources, checked 2026-09-13.
Kubernetes: Resource Quotas, namespace admission limits and object counts, checked 2026-09-13.
Kubernetes: RBAC authorization, namespace bindings and additive permissions, checked 2026-09-13.
Kubernetes: Network Policies, network isolation requires policy and an enforcing implementation, checked 2026-09-13.
Kubernetes: DNS for Services and Pods, short names, namespace-qualified Service names and cluster domains, checked 2026-09-13.