Ops · XB Field Notes

Alertmanager grouping batches alerts while silences suppress notifications

Tue, 27 Jul 2027 00:00:00 +0000

One incident can produce many alerts, but a group is not proof that those alerts have one cause. Alertmanager groups alerts according to configured labels and routing context so their notifications can be handled together. The original alert identities remain available inside the group. Silences add another operation: they suppress matching notifications for a time window without resolving the underlying conditions.

A group key determines which details stay together

With group_by: [alertname, service], two QueueHigh alerts for the checkout service can share a group even when their instance labels differ. A QueueHigh alert for search belongs to another group. A DiskFull alert for checkout also belongs to another group because its alert name differs. Grouping therefore depends on the selected label values, not on all alerts having an identical fingerprint.

Adding instance to the group key creates more groups and may undo the desired batching across replicas. Omitting service can combine unrelated owners or workloads. Choose the key around the investigation and ownership model, then inspect the member list to retain individual context. The special grouping value ... groups by all labels and effectively disables aggregation into broader notification groups.

Routing context matters too. Alerts routed to different receivers are not necessarily one notification merely because their selected grouping labels match. Test the combination of route and group configuration. A receiver responsible for a whole service may benefit from batching several instances, while another integration can require a different grouping contract. Neither setting automatically determines whether the alerts share a root cause.

Waiting, changes and repetition use different timers

group_wait delays the first notification for a new group so nearby arrivals can join it. An alert that resolves before that initial wait ends may never generate a notification. group_interval governs checks for subsequent group updates, including newly firing or resolved members. It is not simply a promise to resend every still-open group at that frequency.

repeat_interval controls reminders when the group’s notification content has not otherwise changed. Its eligibility is checked on the group interval, so choose a compatible multiple instead of assuming an independent precise alarm clock. Delivery processing and failures can add further delay. Set these values from the response objective, and keep the rule’s pending duration separate from notification scheduling.

When it breaks

Grouping too narrowly turns a replica outage into many notifications; grouping too broadly can hide which service needs attention. A dashboard showing one group does not mean only one alert remains. Expand the members and inspect the group key before treating a lower notification count as evidence that the incident has improved.

A silence with overly broad matchers can suppress alerts beyond the planned maintenance scope. Use a bounded time window and inspect which active alerts match it. Silence expiration restores notification eligibility; it does not fix the application or guarantee an immediate message. Inhibition is a separate configured relationship where an active source alert can suppress matching target notifications, rather than a time-bounded manual silence.

Check it quickly

Save this as grouping.py and run python3 grouping.py with Docker available. It uses Alertmanager 0.28.1, fetching the image if needed. The actual API receives four synthetic alerts. The receiver has no delivery integrations and the initial notification wait is deliberately long, so the experiment observes grouping and suppression state without sending messages.

import http.client
import io
import json
import subprocess
import time
import uuid

name = "xbcatm-monitoring-" + uuid.uuid4().hex[:12]


class ResponseSocket:
 def __init__(self, data):
 self.data = data

 def makefile(self, mode):
 return io.BytesIO(self.data)


def request(method, path, body="", content_type="application/json"):
 body = body.encode()
 headers = (f"{method} {path} HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n"
 f"Connection: close\r\nContent-Type: {content_type}\r\n"
 f"Content-Length: {len(body)}\r\n\r\n").encode()
 result = subprocess.run(
 ["docker", "exec", "-i", name, "/bin/nc", "-w", "3", "127.0.0.1", str(port)],
 input=headers + body, capture_output=True, timeout=5,
 )
 if result.returncode:
 raise RuntimeError(result.stderr.decode())
 response = http.client.HTTPResponse(ResponseSocket(result.stdout))
 response.begin()
 return response.status, response.read().decode()


def get_json(path):
 status, body = request("GET", path)
 assert status == 200, (status, body)
 return json.loads(body)


def eventually(check):
 for attempt in range(150):
 try:
 value = check()
 if value:
 return value
 except (RuntimeError, AssertionError, http.client.HTTPException):
 pass
 time.sleep(0.1)
 raise AssertionError("Expected local server state did not appear")


from datetime import datetime, timedelta, timezone

port = 9093
script = r"""set -eu
cat > /tmp/alertmanager.yml <<'ALERT_CONFIG'
route:
 receiver: preview
 group_by: [alertname, service]
 group_wait: 1h
 group_interval: 5m
 repeat_interval: 4h
receivers:
 - name: preview
ALERT_CONFIG
exec /bin/alertmanager --config.file=/tmp/alertmanager.yml --storage.path=/tmp/data \
 --web.listen-address=127.0.0.1:9093 --cluster.listen-address=
"""
try:
 subprocess.run(
 ["docker", "run", "-d", "--rm", "--name", name, "--network", "none",
 "--entrypoint", "/bin/sh", "prom/alertmanager:v0.28.1", "-c", script],
 check=True, capture_output=True, text=True, timeout=20,
 )
 eventually(lambda: request("GET", "/-/ready")[0] == 200)
 labels = [
 {"alertname": "QueueHigh", "service": "checkout", "instance": "api-a"},
 {"alertname": "QueueHigh", "service": "checkout", "instance": "api-b"},
 {"alertname": "QueueHigh", "service": "search", "instance": "api-c"},
 {"alertname": "DiskFull", "service": "checkout", "instance": "api-a"},
 ]
 assert request("POST", "/api/v2/alerts", json.dumps([{"labels": x} for x in labels]))[0] == 200
 groups = eventually(lambda: (groups if len(groups := get_json("/api/v2/alerts/groups")) == 3 else False))
 sizes = {g["labels"]["service"] + "/" + g["labels"]["alertname"]: len(g["alerts"]) for g in groups}
 assert sizes == {"checkout/QueueHigh": 2, "checkout/DiskFull": 1, "search/QueueHigh": 1}
 now = datetime.now(timezone.utc)
 silence = {"matchers": [{"name": "service", "value": "checkout", "isRegex": False, "isEqual": True}],
 "startsAt": (now - timedelta(seconds=1)).isoformat(),
 "endsAt": (now + timedelta(minutes=10)).isoformat(),
 "createdBy": "local-fixture", "comment": "Disposable grouping test"}
 status, body = request("POST", "/api/v2/silences", json.dumps(silence))
 assert status == 200
 silence_id = json.loads(body)["silenceID"]
 def silenced_state():
 alerts = get_json("/api/v2/alerts")
 count = sum(silence_id in x["status"]["silencedBy"] for x in alerts)
 return alerts if count == 3 else None
 alerts = eventually(silenced_state)
 assert len(alerts) == 4
 assert all((silence_id in x["status"]["silencedBy"]) == (x["labels"]["service"] == "checkout") for x in alerts)
 assert request("DELETE", "/api/v2/silence/" + silence_id)[0] == 200
 def unsilenced_state():
 alerts = get_json("/api/v2/alerts")
 return len(alerts) == 4 and all(silence_id not in x["status"]["silencedBy"] for x in alerts)
 eventually(unsilenced_state)
 print(json.dumps({"group_sizes": sizes, "silenced_alerts": 3,
 "alerts_remain_visible": 4, "silence_expired": True}, indent=2))
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

The output must report group sizes of two for checkout/QueueHigh, one for checkout/DiskFull, and one for search/QueueHigh. A temporary silence matching checkout suppresses three alerts, while all four remain visible through the API. The search alert stays outside that silence. The program then expires the silence and verifies that its identifier no longer appears in the alerts’ suppression lists.

All requests stay on container loopback, with no host ports or mounts; cleanup removes the unique container. These assertions test membership and silence lifecycle. They do not measure notification timer accuracy, validate inhibition, or establish downstream delivery. Those behaviors require tests using the actual routing, scheduling and receiver configuration.

Sources

Alertmanager: configuration, group keys, notification intervals and inhibition rules, checked 2026-09-13.
Prometheus: Alertmanager, grouping and silence semantics, checked 2026-09-13.
Alertmanager 0.28.1: README, API v2 and local Alertmanager operation, checked 2026-09-13.

Structured nginx logs need escaping and a value-type contract

Thu, 15 Jul 2027 00:00:00 +0000

A log_format that resembles JSON is not necessarily a reliable structured event format. nginx substitutes variable values into the text, and those values can contain quotes, backslashes, control characters, or no data at all. Correct string escaping and correct value types are separate requirements. Testing emitted records with a real parser catches failures that a visually convincing configuration or successful nginx -t cannot establish.

When it breaks

Raw interpolation of a User-Agent into a quoted JSON field fails when the header contains a quote. Disabling escaping does not make the field safer; it allows the request’s characters to alter the surrounding record syntax. nginx’s default log escaping has a different problem for JSON: it can use hexadecimal \xXX sequences, which are not valid JSON string escapes.

Adding escape=json handles JSON string escaping, but it does not automatically add quotes, supply null, or convert a variable into a number. An upstream timing can be unavailable for a locally generated response. Emitting that optional value without a defined representation can leave the field syntactically incomplete, even though every character that was present was escaped correctly.

Missing-value representation also depends on the escape path in the tested version. nginx 1.28.0’s default escaping writes a dash for a variable it cannot find. Its JSON escaping writes no variable text, so a surrounding pair of quotes becomes an empty string. A parser must distinguish that case from a real variable value that itself contains a dash or a list of upstream attempts.

Check it quickly

Run the script with host Python, Docker, and the pinned nginx image. The recorded environment used nginx 1.28.0 on Linux arm64 and Docker 29.3.1. It builds a temporary configuration with four log formats, validates syntax, makes two real HTTP requests, parses the resulting files, and removes its own container, image tags, and network. All header values are public test strings:

from contextlib import contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import http.client,json,socket,subprocess,time,uuid

NGINX='nginx@sha256:552e7481ca93ffccd046aa658dbbed22caefbc09c66fa7cd247cbb90b8a5c609'
PYTHON='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def wait(predicate):
 deadline=time.monotonic()+15
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except OSError:pass
 time.sleep(.1)
 raise AssertionError('fixture did not become ready')

@contextmanager
def lab(configuration,files=None,backend=None):
 name='xbcatm-nginx-'+uuid.uuid4().hex[:10];tags=[name+':proxy',name+':backend'];containers=[name+'-proxy',name+'-backend']
 try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'files').mkdir();(root/'nginx.conf').write_text(configuration)
 for path,content in (files or {}).items():
 target=root/'files'/path;target.parent.mkdir(parents=True,exist_ok=True);target.write_text(content)
 docker('network','create',name)
 if backend is not None:
 (root/'backend.py').write_text(backend)
 (root/'backend.Dockerfile').write_text(f'FROM {PYTHON}\nCOPY backend.py /backend.py\nCMD ["python", "/backend.py"]\n')
 docker('build','--network','none','-f',str(root/'backend.Dockerfile'),'-t',tags[1],str(root))
 docker('run','-d','--name',containers[1],'--network',name,'--network-alias','backend',tags[1])
 wait(lambda:docker('exec',containers[1],'python','-c','import socket;s=socket.create_connection(("127.0.0.1",8081),timeout=.5);s.close()',check=False).returncode==0)
 (root/'Dockerfile').write_text(f'FROM {NGINX}\nCOPY nginx.conf /etc/nginx/nginx.conf\nCOPY files /srv\n')
 docker('build','--network','none','-t',tags[0],str(root))
 docker('run','--rm','--network',name,tags[0],'nginx','-t')
 docker('run','-d','--name',containers[0],'--network',name,'-p','127.0.0.1::8080',tags[0])
 details=json.loads(docker('inspect',containers[0]).stdout)[0]
 binding=details['NetworkSettings']['Ports']['8080/tcp'][0];assert binding['HostIp']=='127.0.0.1';port=int(binding['HostPort'])
 def ready():
 connection=socket.create_connection(('127.0.0.1',port),timeout=.5);connection.close();return True
 wait(ready)
 def request(path,method='GET',headers=None,body=None):
 connection=http.client.HTTPConnection('127.0.0.1',port,timeout=5)
 try:
 connection.request(method,path,body=body,headers=headers or {})
 response=connection.getresponse();return response.status,{k.lower():v for k,v in response.getheaders()},response.read()
 finally:connection.close()
 yield containers[0],request
 finally:
 docker('rm','-f',*containers,check=False);docker('image','rm',*tags,check=False);docker('network','rm',name,check=False)

def config(locations,extra=''):
 return 'worker_processes 1;\nerror_log /tmp/error.log notice;\nevents {}\nhttp {\naccess_log off;\n'+extra+'\nserver { listen 8080; root /srv;\n'+locations+'\n}\n}\n'

extra=r'''
log_format default_escape '{"agent":"$http_user_agent","upstream_time":"$upstream_response_time"}';
log_format raw escape=none '{"agent":"$http_user_agent"}';
log_format numeric escape=json '{"upstream_time":$upstream_response_time}';
log_format structured escape=json '{"agent":"$http_user_agent","upstream_time":"$upstream_response_time","status":$status,"request_id":"$request_id"}';
'''
locations=r'''
access_log /tmp/default.json default_escape;
access_log /tmp/raw.json raw;
access_log /tmp/numeric.json numeric;
access_log /tmp/structured.json structured;
location / { return 204; }
'''
with lab(config(locations,extra)) as (container,request):
 agent='reader "quoted" '+chr(92)+' path'+chr(9)+'tab'
 assert request('/local',headers={'User-Agent':agent})[0]==204
 assert request('/local',headers={'User-Agent':'plain'})[0]==204
 def lines(path):return docker('exec',container,'cat',path).stdout.splitlines()
 wait(lambda:len(lines('/tmp/structured.json'))==2)
 default=lines('/tmp/default.json');raw=lines('/tmp/raw.json');numeric=lines('/tmp/numeric.json');structured=list(map(json.loads,lines('/tmp/structured.json')))
 for line in [default[0],raw[0]]:
 try:json.loads(line)
 except json.JSONDecodeError:pass
 else:raise AssertionError('non-JSON escaping unexpectedly formed valid JSON')
 assert r'\x22' in default[0]
 assert json.loads(default[1])=={'agent':'plain','upstream_time':'-'}
 assert json.loads(raw[1])=={'agent':'plain'}
 for line in numeric:
 try:json.loads(line)
 except json.JSONDecodeError:pass
 else:raise AssertionError('unquoted missing timing unexpectedly formed valid JSON')
 assert [row['agent'] for row in structured]==[agent,'plain']
 assert all(row['status']==204 and row['upstream_time']=='' for row in structured)
 identifiers=[row['request_id'] for row in structured]
 assert len(set(identifiers))==2 and all(len(value)==32 and int(value,16)>=0 for value in identifiers)
 print('escape=json preserves quotes, backslashes and tabs as valid JSON strings; both default and no escaping break JSON on the same public header')
 print('default escaping writes a dash for the absent timing, while JSON escaping writes an empty string in the quoted field')
 print('an unavailable upstream timing is empty under JSON escaping, so emitting it unquoted still creates invalid JSON')
 print('structured records parse with a numeric status, explicit empty timing string and distinct nginx-generated request IDs')

The first request carries a User-Agent containing quotes, one backslash, and a tab. The second carries plain. Both receive a local 204 response without an upstream request. This deliberately combines challenging string content with an absent timing, allowing the experiment to test escaping and optional-value handling independently rather than relying only on ordinary alphanumeric headers.

The default-escaping record for the first request contains \x22 and fails JSON parsing. The no-escaping record fails too because its raw quote breaks the surrounding string. Their plain-header controls show that a format can appear correct under normal inputs while still failing for another valid header value. A few readable sample lines are therefore weak validation.

The default format’s plain record parses with a dash as its quoted missing timing. Under JSON escaping, the structured format instead produces an empty timing string. These actual observations agree with the version-specific logging code. The difference matters when a collector assumes all nginx missing values use one universal sentinel regardless of format options.

The deliberately unquoted timing format is invalid for both requests. JSON escaping cannot turn the absent variable into a numeric value, so the record lacks a value after the colon. Quoting the optional timing in the structured format keeps the record valid without pretending that a missing measurement equals zero seconds.

Both structured records parse successfully. Their decoded User-Agent values exactly match the supplied strings, status is the numeric value 204, and the optional timing is an empty string. The two nginx-generated request IDs are distinct 32-character hexadecimal strings. The test checks parsed meaning, not just whether a line happens to contain opening and closing braces.

Define what collectors should accept

Choose stable field names and types, and document missing values. Upstream measurements can contain multiple attempt values, so preserving their string form may be more accurate than forcing a scalar number. If a collector converts them into structured attempts, give that conversion an explicit grammar and error policy rather than silently coercing unusual records to zero.

Validate text encoding as well as JSON escaping at the collection boundary. This fixture covers ordinary text with special ASCII characters; it does not claim arbitrary invalid byte sequences become valid Unicode JSON. Keep parser failures observable so malformed records do not silently disappear during an incident.

Record only necessary request data, with appropriate retention and access controls. A correctly escaped field can still contain sensitive information or an untrusted identity claim. Test the complete shipping and querying path separately: local JSON validity establishes a useful contract, but does not prove that rotation, collection, delivery, and retention preserve every event.

Sources

nginx HTTP log module, log formats, escape modes, and request-completion records, checked 2026-09-12.
nginx 1.28.0 log implementation, different missing-variable behavior for default and JSON escaping, checked 2026-09-12.
nginx core variables, request identifiers and HTTP request variables, checked 2026-09-12.

Docker healthchecks report a state; recovery needs a consumer

Mon, 05 Jul 2027 00:00:00 +0000

A Docker HEALTHCHECK runs a configured command inside a container and records the result as health state. It adds an observation beyond whether the main process is running. The meaning of that observation depends on what the command tests: a process check, an HTTP response, or an application-specific readiness condition. Docker does not infer a universal readiness contract from the presence of the instruction.

When it breaks

A command that always exits zero produces reassuring health reports even when the application cannot serve useful work. A shell sequence can accidentally hide an earlier failure by ending with a successful command. Conversely, a probe that depends on an optional external service can mark a usable application unhealthy and encourage unnecessary recovery actions by whatever system consumes the result.

Standalone Docker restart policies react to container termination, not merely to an unhealthy health status. Adding restart=always does not make Docker restart a still-running application when its probe fails. Likewise, a healthcheck does not automatically remove a container from every proxy’s routing table. A controller or proxy must explicitly consume that state and implement the intended action.

Probe timing is another source of incorrect expectations. The retry threshold counts consecutive failures, while startup grace changes which early failures count. A successful probe during startup can end that initial grace behavior. Detection time depends on command duration, interval, timeout, and configuration; it is not simply a promise that one failed request instantly stops traffic.

Check it quickly

Run this script using host Python and Docker with the pinned base already available. The recorded environment used Docker 29.3.1 and Linux arm64 Python 3.11.16. It creates one temporary image and container, disables networking outside the container, and uses its own loopback HTTP server. The only state changed is a harmless readiness file inside the experiment:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,time,uuid

BASE='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],text=True,capture_output=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def inspect(container):return json.loads(docker('inspect',container).stdout)[0]
def wait_for(container,wanted):
 deadline=time.monotonic()+25
 while time.monotonic()<deadline:
 state=inspect(container)
 if state['State']['Health']['Status']==wanted:return state
 time.sleep(.1)
 raise AssertionError('health did not reach '+wanted)

name='xbcatm-health-'+uuid.uuid4().hex[:10];tag=name+':probe'
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory)
 (root/'server.py').write_text('from http.server import BaseHTTPRequestHandler,HTTPServer\nfrom pathlib import Path\nclass Handler(BaseHTTPRequestHandler):\n def do_GET(self):\n code=200 if Path("/ready").exists() else 503\n self.send_response(code);self.end_headers();self.wfile.write(str(code).encode())\n def log_message(self,*args):pass\nHTTPServer(("127.0.0.1",8080),Handler).serve_forever()\n')
 probe='import urllib.request;urllib.request.urlopen("http://127.0.0.1:8080/health",timeout=.5).read()'
 (root/'Dockerfile').write_text(f'FROM {BASE}\nCOPY server.py /server.py\nHEALTHCHECK --interval=1s --timeout=1s --retries=2 --start-period=5s CMD ["python", "-c", {json.dumps(probe)}]\nCMD ["python", "/server.py"]\n')
 docker('build','--network','none','-t',tag,str(root))
 began=time.monotonic()
 docker('run','-d','--name',name,'--network','none','--restart','always',tag)
 initial=inspect(name);assert initial['State']['Health']['Status']=='starting'
 first=wait_for(name,'unhealthy');assert first['State']['Running'] and first['RestartCount']==0
 assert first['State']['StartedAt']==initial['State']['StartedAt']
 assert first['State']['Health']['FailingStreak']>=2
 assert any(log['ExitCode']!=0 and '503' in log['Output'] for log in first['State']['Health']['Log'])
 time.sleep(max(0,12-(time.monotonic()-began)))
 stable=inspect(name);assert stable['State']['Running'] and stable['RestartCount']==0
 assert stable['State']['StartedAt']==initial['State']['StartedAt'] and stable['State']['Health']['Status']=='unhealthy'
 docker('exec',name,'python','-c','from pathlib import Path;Path("/ready").touch()')
 healthy=wait_for(name,'healthy');assert healthy['RestartCount']==0 and healthy['State']['Health']['FailingStreak']==0
 docker('exec',name,'python','-c','from pathlib import Path;Path("/ready").unlink()')
 unhealthy=wait_for(name,'unhealthy');assert unhealthy['RestartCount']==0 and unhealthy['State']['Running']
 assert unhealthy['State']['StartedAt']==initial['State']['StartedAt']
 print('starting -> unhealthy -> healthy -> unhealthy follows the real HTTP probe; failure logs contain HTTP 503')
 print('restart=always does not restart a still-running process solely because its health is unhealthy')
finally:
 docker('rm','-f',name,check=False);docker('image','rm',tag,check=False)

The server returns HTTP 503 while /ready is absent. Its Dockerfile healthcheck uses Python’s HTTP client, which raises on that status and exits unsuccessfully. The probe has a one-second interval and timeout, two retries, and a five-second start period. The container starts with restart=always, and initial inspection records health as starting.

After the counted failures, Docker reports unhealthy; the health log contains the real HTTP 503 error and the failing streak is at least two. The fixture retains the initial start timestamp, waits until at least twelve seconds after launch, and confirms the main process is still running with no restart. This avoids confusing initial startup behavior with recovery from an established unhealthy state.

Creating /ready makes the HTTP endpoint return 200. A later probe changes health to healthy and resets the failing streak to zero. The same container process remains running. That transition demonstrates actual application recovery as observed by the command, without requiring the controller to manufacture a restart simply to clear the status.

Removing the file makes the endpoint fail again, and health eventually returns to unhealthy. The start timestamp remains the same and RestartCount remains zero. The test verifies a full failure, recovery, and failure sequence, rather than observing one initial boot failure and assuming all later health behavior will be identical.

Probe exit status is the interface Docker consumes: zero means success, one indicates failure, and two is reserved by the Dockerfile contract. The Python client naturally returns a failing process status for the HTTP error. The script also inspects the diagnostic output, which helps distinguish a meaningful readiness failure from a missing probe executable or a malformed command.

Define the reaction separately

Choose a cheap, bounded check that represents the state its consumer needs. An internal HTTP check can verify application handling, but does not establish public DNS, TLS termination, proxy routing, or end-user reachability. Use external observations when those paths matter. Keep probes free of destructive actions and avoid logging credentials into retained health output.

For Compose startup dependencies, explicitly use a health-based condition when another service must wait for readiness. For runtime routing or remediation, verify the actual controller’s policy and test the action it performs. Health observation and action should be connected deliberately, with enough grace to avoid turning transient dependency trouble into restart storms.

During an incident, inspect main-process state, health state, probe logs, and restart history separately. A running unhealthy process, an exited process, and a repeatedly restarting process are different failures. Clear evidence for each state makes the recovery choice more precise.

Sources

Dockerfile HEALTHCHECK reference, probe exit codes, intervals, retries, startup grace, and health states, checked 2026-09-12.
Start containers automatically, restart policies responding to container termination, checked 2026-09-12.
Control startup order in Compose, explicit dependency conditions that consume health state, checked 2026-09-12.

systemd-sysusers creates missing service accounts

Tue, 08 Jun 2027 00:00:00 +0000

A service package often needs a dedicated account before it can own files or run without root. systemd-sysusers describes that account and its groups in a small configuration file. Its central operation is creating missing identities and establishing requested memberships. It is not a general account migration engine, a directory creator, or a guarantee that automatically allocated numeric IDs match on every host.

When it breaks

A u entry creates a missing system user and normally its matching primary group. A g entry creates a group, and m adds a supplementary membership. An ID of - requests automatic allocation. An r entry constrains the allocation pool, but another account already occupying a number changes what is available. The same pool therefore does not establish cross-machine identity equality.

Changing a declaration for an existing account does not automatically renumber that account or update its home directory and descriptive fields. This behavior makes repeated package setup useful, but it also means a configuration diff is not proof that live account records changed. A real UID migration needs a separate plan for existing ownership, running processes, shared storage, and rollback.

The home-directory column sets an account-database field; it does not create the directory. A service can consequently have a valid account and still fail because its state path is missing or unwritable. Use an appropriate managed service directory or a tmpfiles rule when a directory must be created with defined ownership and permissions.

Check it quickly

Run this Python script as an administrator in a disposable Linux environment containing systemd-sysusers. It does not need to alter the machine’s live account database. The recorded run used Debian 12 with systemd 252.39. Each test operates on files inside a separate temporary root selected with --root; shadow database contents are compared internally and never printed:

from pathlib import Path
from tempfile import TemporaryDirectory
import subprocess

definition='''r - 61000-61010
u field-worker - "Field worker" /var/lib/field-worker /usr/sbin/nologin
g field-readers -
m field-worker field-readers
'''
def prepare(root,occupied=None):
 (root/'etc').mkdir(parents=True)
 passwd='root:x:0:0:root:/root:/bin/sh\n';group='root:x:0:\n'
 if occupied is not None:
 passwd+=f'occupied:x:{occupied}:{occupied}:Fixture:/nonexistent:/usr/sbin/nologin\n'
 group+=f'occupied:x:{occupied}:\n'
 (root/'etc/passwd').write_text(passwd);(root/'etc/group').write_text(group)
 directory=root/'usr/lib/sysusers.d';directory.mkdir(parents=True)
 (directory/'field.conf').write_text(definition)
def apply(root):
 result=subprocess.run(['systemd-sysusers','--root='+str(root)],capture_output=True,text=True,timeout=10)
 assert result.returncode==0,result.stderr
def entries(root,name):
 return {line.split(':')[0]:line.split(':') for line in (root/'etc'/name).read_text().splitlines()}
with TemporaryDirectory(prefix='xbcatm-sysusers-') as directory:
 first=Path(directory)/'first';second=Path(directory)/'second';prepare(first);apply(first)
 worker=entries(first,'passwd')['field-worker'];groups=entries(first,'group')
 assert worker[3]==groups['field-worker'][2]
 assert 'field-worker' in groups['field-readers'][3].split(',')
 assert worker[5:]==['/var/lib/field-worker','/usr/sbin/nologin']
 assert not (first/'var/lib/field-worker').exists()
 before={name:(first/'etc'/name).read_bytes() for name in ('passwd','group','shadow','gshadow')}
 apply(first)
 assert before=={name:(first/'etc'/name).read_bytes() for name in before}
 print('user, primary group, and supplementary membership created; repeat leaves databases unchanged')
 print('home and nologin shell recorded; home directory was not created')
 prepare(second,int(worker[2]));apply(second)
 other=entries(second,'passwd')['field-worker']
 assert other[2]!=worker[2]
 print('same allocation range and definition choose different UIDs when existing accounts differ')
 changed=definition.replace('u field-worker - "Field worker" /var/lib/field-worker',
 'u field-worker 62000 "Changed description" /changed-home')
 (first/'usr/lib/sysusers.d/field.conf').write_text(changed);apply(first)
 assert entries(first,'passwd')['field-worker']==worker
 print('changing a declaration does not renumber or rewrite an existing account')
 assert not (first/'changed-home').exists()
print('only temporary alternate-root account databases were changed')

The first root begins with only its fixture root account. Applying the declaration creates the worker, its primary group, the readers group, and the worker’s supplementary membership. The script checks the numeric primary-group relationship and actual membership list. A successfully parsed configuration alone would not establish either relationship.

The account record contains the requested home path and a nologin shell, but that home directory does not exist. This directly separates identity declaration from filesystem provisioning. An account intended for service execution can still be used by an authorized service manager; a nologin shell is not a replacement for all permission and credential controls.

Applying the same declaration again leaves the passwd, group, shadow, and gshadow file contents unchanged. This verifies repeat behavior for the exact fixture instead of assuming that every imaginable declaration is harmless. Conflicting input and allocation exhaustion still need diagnosis when the command reports them.

The second root reserves the first worker’s numeric identity for another fixture account before applying the identical configuration. Its worker receives a different UID. Names and requested allocation range match, but existing database state differs. Copying numeric file ownership between these roots without an identity mapping would therefore be unsafe.

Finally, the first root’s declaration requests a new numeric UID, description, and home path for the existing worker. Reapplying leaves its complete account record unchanged and creates no new home. This is the expected boundary of creation-oriented setup, not evidence that the edited fields were successfully migrated. The temporary roots are removed when the exercise ends.

Make packaging and identity policy explicit

Packages normally place declarations under /usr/lib/sysusers.d, while administrator files can override matching filenames under /etc/sysusers.d. Filename ordering and conflicting names matter; declarations are not universally order-independent statements. Inspect the effective input when more than one package or local policy defines the same identity.

Writing a file does not execute sysusers by itself. Distribution boot units and package integration determine when the tool is invoked. Verify the relevant installation path and service dependency so a daemon does not start before its account or state directory exists. Keep account creation and directory provisioning coordinated without treating them as the same operation.

Use automatically allocated IDs when host-local identity is sufficient. If shared storage or a published interface requires a particular numeric identity, establish and verify that allocation policy explicitly, including collisions and existing accounts. Test the actual service’s access after installation. Stable names, stable numbers, and correct filesystem ownership are related requirements that each need evidence.

Sources

systemd 252 sysusers.d, user/group directives, allocation ranges, file precedence, and home-directory fields, checked 2026-09-12.
systemd 252 systemd-sysusers, account creation, repeat execution, and alternate-root processing, checked 2026-09-12.
systemd 252 tmpfiles.d, separate creation and ownership of service directories, checked 2026-09-12.

Autovacuum quietly prunes dead rows

Fri, 04 Jun 2027 00:00:00 +0000

An update creates a new PostgreSQL row version, and a delete makes an existing version invisible to later snapshots. Neither operation means every older transaction has finished using that version. Ordinary vacuum, including work started by autovacuum, must respect that visibility horizon before reclaiming storage. Scheduling more cleanup cannot override a reader’s legitimate snapshot.

When it breaks

A long-lived snapshot can keep old versions needed even after their deleting transaction commits. Vacuum may run successfully while reclaiming less than expected. Lowering an autovacuum threshold will not make those versions removable. Investigate old transactions and other horizon holders, such as an xmin retained by a replication slot, before assuming the background worker is simply too slow.

Reclaiming a tuple’s space usually makes it reusable inside the same relation. It does not necessarily reduce the file’s length. Ordinary vacuum can truncate empty pages at the end, but it does not move surviving rows out of the middle to compact every hole. A stable file size after cleanup can therefore be healthy reuse rather than evidence that nothing happened.

Check it quickly

Use two fresh psql -X -v ON_ERROR_STOP=1 sessions, A and B, on the same disposable PostgreSQL 18 database. This exercise temporarily disables autovacuum only on its fixture, then invokes ordinary vacuum manually to control the timing. It demonstrates the visibility rule shared with automatic vacuum; it does not claim to measure the automatic scheduler. B creates five thousand rows:

CREATE SCHEMA vacuum_demo;
CREATE TABLE vacuum_demo.items (id integer PRIMARY KEY,payload text)
 WITH (autovacuum_enabled=false);
INSERT INTO vacuum_demo.items
SELECT g,repeat(md5(g::text),10) FROM generate_series(1,5000) AS g;
SELECT pg_stat_force_next_flush();
VACUUM (ANALYZE) vacuum_demo.items;
CREATE TABLE vacuum_demo.baseline AS
SELECT pg_relation_size('vacuum_demo.items') AS heap_bytes;

The statistics-flush helper publishes pending counters at the next opportunity; here it keeps laboratory observations from lagging behind the commands. The baseline records heap-file bytes, excluding indexes and other forks. In A, establish a Repeatable Read snapshot and leave it open:

BEGIN ISOLATION LEVEL REPEATABLE READ;
SELECT count(*) FROM vacuum_demo.items;

A sees five thousand rows. In B, delete the first four thousand, publish the write counters, and vacuum:

DELETE FROM vacuum_demo.items WHERE id<=4000;
SELECT pg_stat_force_next_flush();
VACUUM (VERBOSE) vacuum_demo.items;
SELECT pg_stat_force_next_flush();
SELECT count(*) FROM vacuum_demo.items;
SELECT n_dead_tup FROM pg_stat_user_tables WHERE relid='vacuum_demo.items'::regclass;

B sees one thousand live rows, while this isolated run reports four thousand dead tuples still retained. The statistics view is an estimate, not an exact count of physical bytes. In A, read again before ending its transaction:

SELECT count(*) FROM vacuum_demo.items;
COMMIT;

A still sees all five thousand rows. This is the reason vacuum cannot simply erase the older versions: an existing snapshot can still read them. After A commits, B repeats cleanup:

VACUUM (VERBOSE) vacuum_demo.items;
SELECT pg_stat_force_next_flush();
SELECT n_dead_tup FROM pg_stat_user_tables WHERE relid='vacuum_demo.items'::regclass;
SELECT pg_relation_size('vacuum_demo.items')=heap_bytes AS same_heap_size
FROM vacuum_demo.baseline;

The dead-tuple estimate falls to zero, while the heap-size comparison remains true. Surviving rows occupy pages near the original end, so clearing earlier space does not shorten this file. No snapshot now needs the deleted rows; the unchanged size is not a reason to delete the database file or force a rewrite.

B inserts four thousand new rows and checks whether the reclaimed space is reused:

INSERT INTO vacuum_demo.items
SELECT g,repeat(md5(g::text),10) FROM generate_series(5001,9000) AS g;
SELECT count(*),min(id),max(id) FROM vacuum_demo.items;
SELECT pg_relation_size('vacuum_demo.items')=heap_bytes AS reused_existing_heap
FROM vacuum_demo.baseline;
DROP SCHEMA vacuum_demo CASCADE;

Expect five thousand rows with identifiers 4001 through 9000, and a true size comparison. In this fixture, replacement rows fit in existing heap space without growing the relation. Different row widths, concurrent activity, and free-space distribution can change that size result in a real workload. The example finally drops its schema, including the table-level autovacuum override.

Observe removal, reuse, and eligibility separately

n_dead_tup estimates obsolete tuples; last_autovacuum records an automatic run. Neither alone proves that a given amount of disk capacity was returned. Compare retained versions, relation growth, cleanup duration, and the workload’s latency. Distinguish manual vacuum timestamps from automatic ones, and account for delayed cumulative-statistics publication when observing short experiments.

Autovacuum also maintains visibility information and protects against transaction-ID wraparound. Insert-only tables can need vacuum even without an update/delete backlog. Disabling routine work is therefore not a safe long-term answer to maintenance load. Conversely, a quiet table with an old automatic-vacuum timestamp is not automatically broken. Check its eligibility, table overrides, worker availability, and blocking activity. Tune thresholds or capacity only after identifying which part prevents timely cleanup; retained snapshots and insufficient worker throughput call for different corrections.

Sources

PostgreSQL 18 routine vacuuming, version removal, space reuse, visibility maintenance, and freezing, checked 2026-09-10.
PostgreSQL 18 cumulative statistics, estimated tuple counts and publication timing, checked 2026-09-10.
PostgreSQL 18 statistics function implementation, the explicit statistics-flush helper used by the experiment, checked 2026-09-10.

Redis memory fills with fragmentation over time

Thu, 03 Jun 2027 00:00:00 +0000

A Redis process can keep a substantial resident footprint after application keys are deleted. That observation deserves explanation, but it does not by itself prove a leak or allocator fragmentation. The useful question is which bytes remain allocated, which pages remain resident, and whether that footprint threatens the actual capacity budget.

Separate the measurements

INFO memory exposes used_memory for Redis-accounted allocations and used_memory_rss for the resident process footprint observed by the operating system. Neither is simply the sum of application payload lengths. Keys, objects, metadata, clients, and other overhead consume memory too. MEMORY USAGE provides per-key allocation estimates, including overhead; it does not measure that key’s independent contribution to process RSS.

Mem_fragmentation_ratio broadly compares resident memory with Redis-accounted allocations. Code, shared libraries, stacks, and allocator retention also contribute to the difference. The ratio can rise when the denominator falls after deletion, even if RSS barely changes. Inspect absolute bytes alongside ratios: a small empty instance can have a high ratio without consuming a dangerous amount of RAM.

Used_memory_peak helps explain a footprint left after an earlier workload peak. Used_memory_dataset is another accounting view, not a promise that an empty keyspace leaves zero process allocations. Measurements are not all sampled at exactly the instant INFO runs. Do not require an exact equality between a rounded ratio and independently observed values during active writes.

Check the allocator and version

Allocator-specific metrics can help distinguish fragmentation from retained pages and non-heap overhead, but their meaning depends on the build. Redis 8.10.1’s source computes allocator fragmentation from small-bin counters with a Lua adjustment; blindly reconstructing every version’s metric as active bytes divided by allocated bytes is unsafe. Check the versioned implementation when a general documentation formula disagrees.

This exercise deliberately builds with libc. That build lacks jemalloc’s detailed accounting; Redis substitutes some allocator fields from other measurements, and the fragmentation counter remains zero. An allocator_frag_ratio of one is therefore not proof that this process has no fragmentation. Always record mem_allocator before interpreting the more specialized fields or choosing an intervention.

When it breaks

A rising RSS curve alongside a growing dataset calls for workload and capacity analysis, not an automatic fragmentation diagnosis. A falling dataset with stable RSS suggests retained footprint, but still does not identify every retained page. Likewise, one snapshot cannot rule out a leak. Compare the same workload over time, value-size distribution, peak usage, and relevant process limits.

MEMORY PURGE attempts allocator reclamation where implemented. Redis documents it as a benign no-op for allocators other than jemalloc, so a successful reply is not evidence of recovered memory. It also does not remove application keys. Active defragmentation is a different mechanism and requires supported builds; enabling it can add CPU and latency costs that need measurement.

Restarting merely to lower a ratio can introduce downtime, cold-cache load, or data loss when persistence and recovery are inadequate. Establish a measured capacity problem and a rehearsed recovery path before treating restart as remediation. Maxmemory is not an RSS ceiling, and reducing key count does not guarantee immediate page return to the operating system.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1 using libc, and redis-py 8.1.0. The commands build Redis core and install the client in a temporary virtual environment. Downloads require the network; nothing is installed system-wide and optional bundled modules are not built.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python". It creates a dedicated Redis process and private Unix socket under /tmp, disables TCP and persistence, and writes only its own 320 sample keys. Alternating values contain 16 or 64 KiB. Short pauses permit metric sampling, without assuming exact refresh timing. The process and temporary runtime directory are removed afterward.

from contextlib import contextmanager
import json
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance(policy="noeviction"):
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", policy,
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=True,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


with instance() as client:
 def snapshot(stage):
 time.sleep(0.2)
 info = client.info("memory")
 fields = ["used_memory", "used_memory_dataset", "used_memory_rss",
 "used_memory_peak", "mem_fragmentation_ratio", "mem_allocator",
 "allocator_frag_ratio", "allocator_frag_bytes"]
 print(stage, json.dumps({"keys": client.dbsize(),
 **{name: info[name] for name in fields}}, sort_keys=True))
 return info

 empty = snapshot("empty")
 for index in range(320):
 client.set(f"sample:{index}", "x" * (16384 if index % 2 == 0 else 65536))
 assert client.dbsize() == 320
 assert client.memory_usage("sample:0") >= 16384
 loaded = snapshot("loaded")
 assert loaded["used_memory"] > empty["used_memory"] + 10 * 1024 * 1024
 for index in range(1, 320, 2):
 assert client.delete(f"sample:{index}") == 1
 assert client.dbsize() == 160
 half = snapshot("large values deleted")
 assert half["used_memory"] < loaded["used_memory"] - 8 * 1024 * 1024
 for index in range(0, 320, 2):
 assert client.delete(f"sample:{index}") == 1
 assert client.dbsize() == 0
 assert client.memory_usage("sample:0") is None
 cleared = snapshot("all values deleted")
 assert cleared["used_memory"] < half["used_memory"]
 assert cleared["used_memory_peak"] >= loaded["used_memory"]
 assert client.memory_purge()
 snapshot("after MEMORY PURGE")
 print("PASS: logical allocations fell; no RSS or purge-effect threshold is assumed")

One run reported about 16.7 MiB of used_memory after loading. Deleting all keys reduced that to about 1.04 MiB, while RSS remained around 20.8 MiB. The overall ratio consequently rose to roughly twenty. MEMORY USAGE confirmed the named key no longer existed; logical allocations fell, but resident memory did not track them proportionally.

The allocator-specific fragmentation ratio stayed at one throughout this libc run, and MEMORY PURGE returned success without a meaningful RSS drop. Counts and logical-allocation relationships are asserted; exact RSS values are intentionally observations, not pass thresholds. This experiment demonstrates accounting differences and the no-op limitation, not jemalloc defragmentation effectiveness, Linux memory-pressure behavior, or a leak diagnosis.

Sources

Redis INFO, process, allocation, peak, and allocator memory fields, checked 2026-09-08.
Redis MEMORY USAGE, per-key allocation estimates and overhead, checked 2026-09-08.
Redis MEMORY PURGE, jemalloc support and other allocators’ no-op behavior, checked 2026-09-08.
Redis 8.10.1 object accounting, version-specific fragmentation calculation, checked 2026-09-08.
Redis 8.10.1 memory sampling, cached measurements and substituted allocator fields, checked 2026-09-08.
Redis 8.10.1 build instructions, core build and allocator selection, checked 2026-09-08.

Redis keys read well with clear naming

Tue, 01 Jun 2027 00:00:00 +0000

Redis keys are binary-safe names in a flat keyspace, not filesystem paths or schema declarations. Colons are a useful convention for separating ownership, environment, entity, and identifier. A name such as cart:prod:string:42 can explain intent, but Redis does not interpret those labels as a type check or access restriction.

Make the convention unambiguous

Agree on field order and identifier encoding across writers, readers, maintenance jobs, and migrations. Distinct prefixes separate names, but identical encoded names still collide. A convention helps people cooperate; it does not create an independently managed namespace.

A type label is documentation. SET can create a string under a key whose name contains hash, and HSET will then report WRONGTYPE. Use TYPE to inspect data, and keep serialization and type expectations in the application contract. Names cannot prove value types.

Prefer short, readable names, not cryptic abbreviations or entire documents embedded in keys. Names consume memory and travel in command arguments, but object overhead and payloads also matter. A naming change is not evidence of faster queries without measurement. Encode identifiers so separator characters, glob metacharacters, and cluster braces cannot acquire unintended meaning.

Match patterns without inventing deletion semantics

SCAN MATCH accepts a glob pattern. Start with cursor zero, reuse each returned cursor, and continue until zero returns again. COUNT is a work hint, not an exact result count or memory ceiling. A call may return no matches while the cursor remains nonzero. Each call still performs server work; incremental iteration does not mean zero blocking cost.

A full scan returns matching keys present throughout it, but it may return duplicates. Keys added or removed during the scan have weaker guarantees. Deduplicate when necessary or make processing safe to repeat. This is not a snapshot, and an empty page is not the end of the collection.

DEL has different semantics: it receives exact key names, not patterns. Sending an argument ending in an asterisk does not delete matching siblings. It only deletes a key literally containing that asterisk if one exists. Shell expansion is another layer, so quote arguments when using command-line tools; do not mistake shell behavior for Redis pattern matching.

When it breaks

A maintenance script that assumes DEL expands a prefix may remove nothing and falsely report cleanup. The opposite mistake is collecting a broad SCAN result and deleting every match without checking ownership. Review exact targets and the mutation policy. A key can be recreated between discovery and deletion, so a previously reviewed name alone may not identify the same logical object later.

Prefixes are not security boundaries. Restrict access with appropriate ACL key patterns and command permissions, or with separate deployment boundaries when required. Key-pattern permissions do not constrain commands that operate on an entire database without key arguments. A visible service prefix also does not reserve memory or isolate eviction.

Redis Cluster hash tags are an intentional exception to otherwise conventional naming. A valid brace-delimited hash tag selects the bytes used for slot calculation. Different service prefixes can therefore share a slot when their tags match. This can support related multi-key work, but putting every key under one constant tag can concentrate load instead of distributing it.

Check it quickly

Use a POSIX environment with a C compiler, make, curl, tar, and Python 3.10 or newer. This ran on macOS with Python 3.14.4, Redis 8.10.1, and redis-py 8.1.0. The setup compiles Redis core with libc and installs the client in a temporary virtual environment, not a system service.

redis_lab=$(mktemp -d)
curl --fail --location --output "$redis_lab/redis.tar.gz" https://download.redis.io/releases/redis-8.10.1.tar.gz
tar -xzf "$redis_lab/redis.tar.gz" -C "$redis_lab"
make -C "$redis_lab/redis-8.10.1" build redis -j2 MALLOC=libc
export REDIS_SERVER="$redis_lab/redis-8.10.1/src/redis-server"
python3 -m venv "$redis_lab/venv"
"$redis_lab/venv/bin/python" -m pip install redis==8.1.0

Run the Python block with "$redis_lab/venv/bin/python" in the same shell. It starts a private Unix-socket Redis under /tmp with TCP and persistence disabled, then stops only its own process and cleans its runtime directory. Downloads require network access. Existing Redis instances are not contacted.

from contextlib import contextmanager
import os
from pathlib import Path
import subprocess
from tempfile import TemporaryDirectory
import time

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry


@contextmanager
def instance():
 with TemporaryDirectory(prefix="redis-lab-", dir="/tmp") as directory:
 endpoint = str(Path(directory) / "redis.sock")
 process = subprocess.Popen([
 os.environ["REDIS_SERVER"], "--port", "0",
 "--unixsocket", endpoint, "--unixsocketperm", "700",
 "--save", "", "--appendonly", "no", "--dir", directory,
 "--maxmemory-policy", "noeviction",
 ], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
 client = redis.Redis(unix_socket_path=endpoint, decode_responses=False,
 socket_timeout=2, retry=Retry(NoBackoff(), 0))
 try:
 deadline = time.monotonic() + 5
 while True:
 assert process.poll() is None, "Redis exited during startup"
 try:
 if client.ping():
 break
 except redis.ConnectionError:
 if time.monotonic() >= deadline:
 raise
 time.sleep(0.02)
 yield client
 finally:
 client.close()
 if process.poll() is None:
 process.terminate()
 try:
 _, errors = process.communicate(timeout=5)
 except subprocess.TimeoutExpired:
 process.kill()
 process.communicate()
 raise
 assert process.returncode == 0, errors.decode()


with instance() as client:
 client.set("cart:prod:string:42", "cart-value")
 client.set("billing:prod:string:42", "billing-value")
 assert client.get("cart:prod:string:42") == b"cart-value"
 assert client.get("billing:prod:string:42") == b"billing-value"
 client.set("cart:prod:hash:42", "not-a-hash")
 assert client.type("cart:prod:hash:42") == b"string"
 try:
 client.hset("cart:prod:hash:42", "item", "value")
 except redis.ResponseError as error:
 assert "WRONGTYPE" in str(error)
 else:
 raise AssertionError("a type label unexpectedly enforced a schema")
 print("namespaces separate exact names; a hash label does not create a hash value")

 prefix = "cart:prod:string:"
 expected = {f"{prefix}{number}".encode() for number in range(24)}
 for key in expected:
 client.set(key, "value")
 expected.add(b"cart:prod:string:42")
 cursor, found, pages = 0, set(), 0
 while True:
 cursor, keys = client.scan(cursor=cursor, match=prefix + "*", count=3)
 assert all(key.startswith(prefix.encode()) for key in keys)
 found.update(keys)
 pages += 1
 assert pages < 1000, "unexpected nontermination in this bounded test"
 if cursor == 0:
 break
 assert found == expected
 print("SCAN reaches cursor zero and finds the stable fixture, with duplicates deduplicated")

 assert client.delete(prefix + "*") == 0
 assert all(client.exists(key) for key in expected)
 client.set(prefix + "*", "literal-star")
 assert client.delete(prefix + "*") == 1
 assert all(client.exists(key) for key in expected)
 print("DEL treats the asterisk literally; it does not expand a key pattern")

 from redis.crc import key_slot

 assert key_slot(b"cart:{customer-42}:items") == key_slot(b"billing:{customer-42}:total")
 assert key_slot(b"cart:{customer-42}:items") == key_slot(b"customer-42")
 print("matching hash tags select the same slot across different service prefixes")

The first case stores different values under cart and billing prefixes, then writes a string under a name labeled hash. TYPE reports string and HSET fails, proving that readable labels do not enforce a schema. These are separate exact names, not nested directories.

A stable fixture contains 25 matching string keys. The scan loop accumulates them until cursor zero, tolerating page size variation and deduplicating results. Its bounded test guard detects unexpected nontermination for this small fixture; it is not a universal production scan limit or evidence about concurrently changing keys.

Deleting the literal prefix-plus-asterisk first returns zero while all 25 keys remain. Creating exactly that literal name makes the next deletion return one, still leaving the siblings intact. No wildcard cleanup or shared data deletion is performed.

Finally, the pinned client’s key_slot helper confirms that two differently prefixed names with customer-42 in braces compute the same slot. This is a client-side calculation, not a live Redis Cluster test. Four summaries follow assertions; the exercise does not validate ACL enforcement, concurrent deletion safety, cluster routing, or memory savings.

Sources

Redis keys and values, binary-safe keys and naming conventions, checked 2026-09-08.
Redis SCAN, cursor guarantees, duplicates, and COUNT hints, checked 2026-09-08.
Redis DEL, deleting exact key arguments, checked 2026-09-08.
Redis TYPE, inspecting the actual value type, checked 2026-09-08.
Redis ACL documentation, key patterns and separate command permissions, checked 2026-09-08.
Redis Cluster specification, hash tags and slot placement, checked 2026-09-08.
redis-py 8.1.0 key-slot calculation, client-side hashing used in the exercise, checked 2026-09-08.
Redis 8.10.1 build instructions, core-only compilation and allocator selection, checked 2026-09-08.

Ops · XB Field Notes

Alertmanager grouping batches alerts while silences suppress notifications

A group key determines which details stay together

Waiting, changes and repetition use different timers

When it breaks

Check it quickly

Sources

Structured nginx logs need escaping and a value-type contract

When it breaks

Check it quickly

Define what collectors should accept

Sources

Docker healthchecks report a state; recovery needs a consumer

When it breaks

Check it quickly

Define the reaction separately

Sources

systemd-sysusers creates missing service accounts

When it breaks

Check it quickly

Make packaging and identity policy explicit

Further reading

Sources

Autovacuum quietly prunes dead rows

When it breaks

Check it quickly

Observe removal, reuse, and eligibility separately

Sources

Redis memory fills with fragmentation over time

Separate the measurements

Check the allocator and version

When it breaks

Check it quickly

Further reading

Sources

Redis keys read well with clear naming

Make the convention unambiguous

Match patterns without inventing deletion semantics

When it breaks

Check it quickly

Sources

Further reading