Logging · XB Field Notes

The systemd journal needs an explicit retention contract

Tue, 26 Oct 2027 00:00:00 +0000

The systemd journal stores messages with structured metadata in journal files. That makes service output searchable without first inventing a parser for every application’s text format. Keeping those messages for a useful period is a separate requirement. Storage location, size policy, rotation, and forwarding determine which evidence remains available when an incident is investigated later.

When it breaks

The default namespace normally uses Storage=auto: persistent storage is selected when /var/log/journal exists, otherwise runtime storage is used. Storage=volatile writes beneath /run/log/journal; this is not simply the kernel’s text ring buffer. Those runtime files normally disappear across a machine reboot, but restarting the journald service is a different event and need not erase them.

Storage=persistent prefers /var/log/journal, with runtime storage used during early boot or when persistent storage cannot be written. A flush operation transfers runtime data under the documented conditions. Inspect actual files and effective configuration instead of inferring location solely from the presence of an option in one configuration fragment.

journald has default size limits and rotation behavior. It does not grow without bounds merely because no administrator wrote a rotation job. However, SystemMaxUse is not an exact hard quota that guarantees the filesystem can never fill. Runtime storage has separate settings, free-space constraints also apply, other programs consume space, and only archived journal files can be deleted during vacuuming.

Check it quickly

Run this Python script as an administrator in a disposable Linux system managed by systemd. The recorded environment was Debian 12 with systemd 252.39. It creates a unique journal namespace, gives it an explicit persistent-storage configuration, emits two harmless messages through transient services, and removes only that namespace’s data afterward. The vacuum operations deliberately delete the experiment’s old message:

from pathlib import Path
import json,shutil,subprocess,uuid

name='xbcatm-journal-'+uuid.uuid4().hex[:8]
config=Path('/etc/systemd')/('journald@'+name+'.conf')
directory=Path('/var/log/journal')/(Path('/etc/machine-id').read_text().strip()+'.'+name)
assert not config.exists() and not directory.exists()
def run(*args):
 result=subprocess.run(args,text=True,capture_output=True,timeout=15)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout
def journal(*args):return run('journalctl','--namespace='+name,*args)
def emit(message):
 run('systemd-run','--quiet','--wait','--collect','--unit='+name+'-'+message,
 '--property=LogNamespace='+name,'--property=StandardOutput=journal',
 '--property=SyslogIdentifier='+name,'/usr/bin/printf',message+'\n')
 journal('--sync')
def messages():
 rows=journal('--no-pager','-o','json','SYSLOG_IDENTIFIER='+name)
 return [json.loads(line)['MESSAGE'] for line in rows.splitlines()]
config.write_text('[Journal]\nStorage=persistent\nSystemMaxUse=16M\nRuntimeMaxUse=8M\n')
try:
 emit('old');journal('--flush');journal('--sync')
 assert messages()==['old'] and list(directory.glob('*.journal'))
 run('systemctl','restart','systemd-journald@'+name+'.service')
 assert messages()==['old']
 print('persistent namespace: message survives journald service restart')
 journal('--vacuum-size=1')
 assert messages()==['old']
 print('vacuum cannot remove the active journal file to satisfy a one-byte target')
 journal('--rotate');emit('new')
 assert messages()==['old','new']
 assert list(directory.glob('*@*.journal'))
 journal('--vacuum-size=1')
 assert messages()==['new'] and not list(directory.glob('*@*.journal'))
 print('after rotation: archived old message removed; active new message remains')
 print('persistent files observed under /var/log/journal; no reboot durability claim')
finally:
 run('systemctl','stop','systemd-journald@'+name+'.service',
 'systemd-journald@'+name+'.socket','systemd-journald-varlink@'+name+'.socket')
 config.unlink(missing_ok=True)
 for path in (directory,Path('/run/log/journal')/directory.name,Path('/run/systemd')/('journal.'+name)):
 if path.exists():shutil.rmtree(path)

The first message is present in journal files beneath the namespace’s /var/log/journal directory. After restarting that namespace’s journald service, the message is still queryable. This verifies the selected storage path and survival across a daemon restart. It does not simulate a power failure, prove storage-device durability, or test a complete machine reboot.

Vacuuming to a one-byte target initially leaves the message visible because its file is still active. The requested size is too small, but vacuum is not permitted to delete that active file just to meet the number. This provides a concrete counterexample to treating a vacuum target as a guaranteed final disk-usage figure.

The script then rotates the journal and writes a new message. Rotation closes the earlier file as an archive and opens a replacement. Both messages are readable before the second vacuum. Afterward, the archived old message is gone while the new message in the active file remains. Rotation and deletion are separate operations with different effects on evidence.

All journal commands select the unique namespace, so they do not vacuum the system’s default journal. Namespace service and socket instances are stopped before their test directories are removed. A normal query without the namespace selector would look in the default namespace and could miss these records entirely, even though they were successfully stored.

Size the evidence window

Choose retention from the incident-response window and observed message volume. A size limit cannot promise a fixed number of days when traffic changes. A retention-age limit likewise does not reserve enough bytes to preserve every message until that age. Monitor actual disk use and the earliest available records, especially after an unusually noisy failure.

Log collection can also lose messages before retention becomes relevant. Rate limits, application output routing, permissions, and unavailable storage affect what reaches the journal. Preserve the distinction between “no matching stored record” and “the event never occurred.” Avoid putting credentials into messages merely because local journal access is restricted.

If records must survive loss of the machine, provide and verify an independent collection path. Forwarding configuration alone is not evidence that a remote destination received or retained the message. Check delivery, access control, and the remote retention window using an identifiable harmless event. Local searchable logs and durable external evidence answer related but different operational needs.

Sources

systemd 252 journald.conf, storage selection, size defaults, free-space limits, and retention, checked 2026-09-12.
systemd 252 journalctl, namespace selection, rotation, synchronization, and archive vacuuming, checked 2026-09-12.
systemd 252 systemd-journald.service, journal namespaces, transport, and persistent file locations, checked 2026-09-12.

Query the journal with explicit field and priority rules

Fri, 30 Jul 2027 00:00:00 +0000

A useful journal query selects the records that answer a specific incident question. Unit identity, invocation, time, priority, and application fields provide different dimensions. Combining them correctly matters more than finding a familiar word somewhere in the machine’s history. journald can attach process metadata, but it does not automatically collect every file an application writes or infer the business meaning of each message.

When it breaks

Field matches have a particular Boolean grammar. Different fields are combined with AND. Multiple matches for the same field are alternatives combined with OR. A separate + joins groups with OR. Consequently, adding another value for a field can broaden a result rather than narrow it. Treat the query as a predicate and check its result against known records before using it in an incident script.

-p err includes error and more severe priorities. It does not mean only priority three. Use an exact PRIORITY=3 field match when that is the intended condition. Application text saying “error” is not itself a priority assignment; stdout collection uses configured priority rules unless the application supplies a supported priority representation.

A successful command with no returned records is not proof that the service never ran. The selected boot, time window, namespace, reader permissions, retention, or output routing might explain the absence. Conversely, querying an entire unit history can find a previous successful startup and mistakenly attribute it to the current process. Separate unit identity from one specific invocation.

Check it quickly

Run this Python script as an administrator in a disposable Linux system whose PID 1 is systemd. The recorded environment was Debian 12 with systemd 252.39. It installs a uniquely named temporary service and activates it twice. Each worker submits four short structured datagrams to the native journal socket, with two application components and several explicit priorities:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid

name='xbcatm-query-'+uuid.uuid4().hex[:8];unit=name+'.service'
path=Path('/run/systemd/system')/unit
assert not path.exists()
def run(*args):
 result=subprocess.run(args,text=True,capture_output=True,timeout=10)
 assert result.returncode==0,(args,result.stderr)
 return result.stdout.strip()
def query(*matches):
 output=run('journalctl','--no-pager','-o','json','-u',unit,*matches)
 return [json.loads(line) for line in output.splitlines() if line]
def labels(rows):return sorted(row['MESSAGE'] for row in rows)
with TemporaryDirectory(prefix=name+'-',dir='/run') as directory:
 root=Path(directory);worker=root/'worker.py'
 worker.write_text(r'''from pathlib import Path
import socket
phase=Path(__file__).with_name('phase').read_text()
with socket.socket(socket.AF_UNIX,socket.SOCK_DGRAM) as log:
 log.connect('/run/systemd/journal/socket')
 for component,priority in [('api',2),('api',3),('api',6),('db',3)]:
 fields=[f'MESSAGE={phase}-{component}-{priority}',f'PRIORITY={priority}',
 f'LAB_RUN={phase}',f'LAB_COMPONENT={component}',
 '_SYSTEMD_UNIT=forged.service']
 log.sendall(('\n'.join(fields)+'\n').encode())
''')
 path.write_text(f'[Service]\nType=oneshot\nExecStart=/usr/bin/python3 {worker}\n')
 try:
 run('systemd-analyze','verify',str(path));run('systemctl','daemon-reload')
 for phase in ('first','second'):
 (root/'phase').write_text(phase);run('systemctl','start',unit)
 run('journalctl','--sync')
 first=query('LAB_RUN=first')
 assert len(first)==4 and all(row['_SYSTEMD_UNIT']==unit for row in first)
 second=query('LAB_RUN=second');assert len(second)==4
 invocation=first[0]['_SYSTEMD_INVOCATION_ID']
 assert {row['_SYSTEMD_INVOCATION_ID'] for row in first}=={invocation}
 assert {row['_SYSTEMD_INVOCATION_ID'] for row in second}!={invocation}
 assert labels(query('LAB_RUN=first','LAB_COMPONENT=api','-p','err'))==['first-api-2','first-api-3']
 print('different fields AND; -p err includes critical priority 2 and error priority 3')
 assert labels(query('LAB_RUN=first','LAB_COMPONENT=api','PRIORITY=3'))==['first-api-3']
 assert labels(query('LAB_RUN=first','LAB_COMPONENT=api','LAB_COMPONENT=db','PRIORITY=3'))==['first-api-3','first-db-3']
 print('exact PRIORITY=3 differs from threshold; repeated component fields OR')
 union=query('LAB_RUN=first','LAB_COMPONENT=db','+','LAB_RUN=second','LAB_COMPONENT=db')
 assert labels(union)==['first-db-3','second-db-3']
 assert labels(query('_SYSTEMD_INVOCATION_ID='+invocation,'LAB_COMPONENT=db'))==['first-db-3']
 print('explicit + joins match groups; invocation filter separates repeated starts')
 assert query('LAB_RUN=absent')==[]
 print('empty query still succeeds; underscore unit field comes from journal metadata')
 finally:
 run('systemctl','stop',unit)
 if run('systemctl','show',unit,'-p','ActiveState','--value')=='failed':run('systemctl','reset-failed',unit)
 path.unlink();run('systemctl','daemon-reload')

The first activation produces an API critical message, an API error, an API informational message, and a database error. The second activation produces the same categories with a different application run label. Filtering by the first run, API component, and -p err returns two records: priorities two and three. Switching to exact priority three returns only the API error.

Repeating the component field with API and database values returns both components’ error records for the first run. It does not demand that one record simultaneously have both values. The explicit + example then selects the database record from either run. The assertions check complete expected message lists, so an accidentally broad filter cannot pass by merely containing one desired phrase.

The service’s two activations have different journal invocation identifiers. Filtering with the first identifier selects only that activation’s database message, even though the unit name is unchanged. This is useful during crash loops and repeated manual starts: the unit groups a service’s history, while an invocation identifies one execution lifecycle.

The worker also attempts to submit a forged underscore-prefixed unit field. Returned records carry the actual service unit instead. Those trusted fields are supplied by the journal from process context rather than accepted as ordinary application labels. In contrast, the custom component and run fields are application claims; the experiment controls them, but a production parser should not treat their names as an independent authentication mechanism.

Finally, a match for an absent run returns an empty list while the query command succeeds. The script explicitly checks this case. It removes the temporary service and worker files afterward; the eight harmless journal entries remain subject to the host’s normal retention policy, rather than deleting unrelated system history to erase a small fixture.

Preserve context when exporting results

Use structured output when another program consumes the query. The sample parses JSON and its deliberately small textual fields. General journal JSON can represent repeated values or binary data differently, so define the consumer’s accepted schema rather than assuming every arbitrary field is always one simple string. Keep timestamps and attribution fields when exporting evidence for later comparison.

Record the relevant boot or invocation and make time-zone assumptions explicit for time-window queries. Reader access and selected namespace are part of the observation, not incidental details. Before concluding that an application event is missing, verify that a known harmless event from the same route is visible. A bounded, reproducible query should make both its selection rule and its limits clear.

Sources

systemd 252 journalctl, field-match Boolean rules, priority filters, and JSON output, checked 2026-09-12.
systemd 252 systemd.journal-fields, trusted metadata, application fields, and invocation identifiers, checked 2026-09-12.
systemd 252 systemd-journald.service, native logging transport, namespace separation, and collection scope, checked 2026-09-12.

Structured nginx logs need escaping and a value-type contract

Thu, 15 Jul 2027 00:00:00 +0000

A log_format that resembles JSON is not necessarily a reliable structured event format. nginx substitutes variable values into the text, and those values can contain quotes, backslashes, control characters, or no data at all. Correct string escaping and correct value types are separate requirements. Testing emitted records with a real parser catches failures that a visually convincing configuration or successful nginx -t cannot establish.

When it breaks

Raw interpolation of a User-Agent into a quoted JSON field fails when the header contains a quote. Disabling escaping does not make the field safer; it allows the request’s characters to alter the surrounding record syntax. nginx’s default log escaping has a different problem for JSON: it can use hexadecimal \xXX sequences, which are not valid JSON string escapes.

Adding escape=json handles JSON string escaping, but it does not automatically add quotes, supply null, or convert a variable into a number. An upstream timing can be unavailable for a locally generated response. Emitting that optional value without a defined representation can leave the field syntactically incomplete, even though every character that was present was escaped correctly.

Missing-value representation also depends on the escape path in the tested version. nginx 1.28.0’s default escaping writes a dash for a variable it cannot find. Its JSON escaping writes no variable text, so a surrounding pair of quotes becomes an empty string. A parser must distinguish that case from a real variable value that itself contains a dash or a list of upstream attempts.

Check it quickly

Run the script with host Python, Docker, and the pinned nginx image. The recorded environment used nginx 1.28.0 on Linux arm64 and Docker 29.3.1. It builds a temporary configuration with four log formats, validates syntax, makes two real HTTP requests, parses the resulting files, and removes its own container, image tags, and network. All header values are public test strings:

from contextlib import contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import http.client,json,socket,subprocess,time,uuid

NGINX='nginx@sha256:552e7481ca93ffccd046aa658dbbed22caefbc09c66fa7cd247cbb90b8a5c609'
PYTHON='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def wait(predicate):
 deadline=time.monotonic()+15
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except OSError:pass
 time.sleep(.1)
 raise AssertionError('fixture did not become ready')

@contextmanager
def lab(configuration,files=None,backend=None):
 name='xbcatm-nginx-'+uuid.uuid4().hex[:10];tags=[name+':proxy',name+':backend'];containers=[name+'-proxy',name+'-backend']
 try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'files').mkdir();(root/'nginx.conf').write_text(configuration)
 for path,content in (files or {}).items():
 target=root/'files'/path;target.parent.mkdir(parents=True,exist_ok=True);target.write_text(content)
 docker('network','create',name)
 if backend is not None:
 (root/'backend.py').write_text(backend)
 (root/'backend.Dockerfile').write_text(f'FROM {PYTHON}\nCOPY backend.py /backend.py\nCMD ["python", "/backend.py"]\n')
 docker('build','--network','none','-f',str(root/'backend.Dockerfile'),'-t',tags[1],str(root))
 docker('run','-d','--name',containers[1],'--network',name,'--network-alias','backend',tags[1])
 wait(lambda:docker('exec',containers[1],'python','-c','import socket;s=socket.create_connection(("127.0.0.1",8081),timeout=.5);s.close()',check=False).returncode==0)
 (root/'Dockerfile').write_text(f'FROM {NGINX}\nCOPY nginx.conf /etc/nginx/nginx.conf\nCOPY files /srv\n')
 docker('build','--network','none','-t',tags[0],str(root))
 docker('run','--rm','--network',name,tags[0],'nginx','-t')
 docker('run','-d','--name',containers[0],'--network',name,'-p','127.0.0.1::8080',tags[0])
 details=json.loads(docker('inspect',containers[0]).stdout)[0]
 binding=details['NetworkSettings']['Ports']['8080/tcp'][0];assert binding['HostIp']=='127.0.0.1';port=int(binding['HostPort'])
 def ready():
 connection=socket.create_connection(('127.0.0.1',port),timeout=.5);connection.close();return True
 wait(ready)
 def request(path,method='GET',headers=None,body=None):
 connection=http.client.HTTPConnection('127.0.0.1',port,timeout=5)
 try:
 connection.request(method,path,body=body,headers=headers or {})
 response=connection.getresponse();return response.status,{k.lower():v for k,v in response.getheaders()},response.read()
 finally:connection.close()
 yield containers[0],request
 finally:
 docker('rm','-f',*containers,check=False);docker('image','rm',*tags,check=False);docker('network','rm',name,check=False)

def config(locations,extra=''):
 return 'worker_processes 1;\nerror_log /tmp/error.log notice;\nevents {}\nhttp {\naccess_log off;\n'+extra+'\nserver { listen 8080; root /srv;\n'+locations+'\n}\n}\n'

extra=r'''
log_format default_escape '{"agent":"$http_user_agent","upstream_time":"$upstream_response_time"}';
log_format raw escape=none '{"agent":"$http_user_agent"}';
log_format numeric escape=json '{"upstream_time":$upstream_response_time}';
log_format structured escape=json '{"agent":"$http_user_agent","upstream_time":"$upstream_response_time","status":$status,"request_id":"$request_id"}';
'''
locations=r'''
access_log /tmp/default.json default_escape;
access_log /tmp/raw.json raw;
access_log /tmp/numeric.json numeric;
access_log /tmp/structured.json structured;
location / { return 204; }
'''
with lab(config(locations,extra)) as (container,request):
 agent='reader "quoted" '+chr(92)+' path'+chr(9)+'tab'
 assert request('/local',headers={'User-Agent':agent})[0]==204
 assert request('/local',headers={'User-Agent':'plain'})[0]==204
 def lines(path):return docker('exec',container,'cat',path).stdout.splitlines()
 wait(lambda:len(lines('/tmp/structured.json'))==2)
 default=lines('/tmp/default.json');raw=lines('/tmp/raw.json');numeric=lines('/tmp/numeric.json');structured=list(map(json.loads,lines('/tmp/structured.json')))
 for line in [default[0],raw[0]]:
 try:json.loads(line)
 except json.JSONDecodeError:pass
 else:raise AssertionError('non-JSON escaping unexpectedly formed valid JSON')
 assert r'\x22' in default[0]
 assert json.loads(default[1])=={'agent':'plain','upstream_time':'-'}
 assert json.loads(raw[1])=={'agent':'plain'}
 for line in numeric:
 try:json.loads(line)
 except json.JSONDecodeError:pass
 else:raise AssertionError('unquoted missing timing unexpectedly formed valid JSON')
 assert [row['agent'] for row in structured]==[agent,'plain']
 assert all(row['status']==204 and row['upstream_time']=='' for row in structured)
 identifiers=[row['request_id'] for row in structured]
 assert len(set(identifiers))==2 and all(len(value)==32 and int(value,16)>=0 for value in identifiers)
 print('escape=json preserves quotes, backslashes and tabs as valid JSON strings; both default and no escaping break JSON on the same public header')
 print('default escaping writes a dash for the absent timing, while JSON escaping writes an empty string in the quoted field')
 print('an unavailable upstream timing is empty under JSON escaping, so emitting it unquoted still creates invalid JSON')
 print('structured records parse with a numeric status, explicit empty timing string and distinct nginx-generated request IDs')

The first request carries a User-Agent containing quotes, one backslash, and a tab. The second carries plain. Both receive a local 204 response without an upstream request. This deliberately combines challenging string content with an absent timing, allowing the experiment to test escaping and optional-value handling independently rather than relying only on ordinary alphanumeric headers.

The default-escaping record for the first request contains \x22 and fails JSON parsing. The no-escaping record fails too because its raw quote breaks the surrounding string. Their plain-header controls show that a format can appear correct under normal inputs while still failing for another valid header value. A few readable sample lines are therefore weak validation.

The default format’s plain record parses with a dash as its quoted missing timing. Under JSON escaping, the structured format instead produces an empty timing string. These actual observations agree with the version-specific logging code. The difference matters when a collector assumes all nginx missing values use one universal sentinel regardless of format options.

The deliberately unquoted timing format is invalid for both requests. JSON escaping cannot turn the absent variable into a numeric value, so the record lacks a value after the colon. Quoting the optional timing in the structured format keeps the record valid without pretending that a missing measurement equals zero seconds.

Both structured records parse successfully. Their decoded User-Agent values exactly match the supplied strings, status is the numeric value 204, and the optional timing is an empty string. The two nginx-generated request IDs are distinct 32-character hexadecimal strings. The test checks parsed meaning, not just whether a line happens to contain opening and closing braces.

Define what collectors should accept

Choose stable field names and types, and document missing values. Upstream measurements can contain multiple attempt values, so preserving their string form may be more accurate than forcing a scalar number. If a collector converts them into structured attempts, give that conversion an explicit grammar and error policy rather than silently coercing unusual records to zero.

Validate text encoding as well as JSON escaping at the collection boundary. This fixture covers ordinary text with special ASCII characters; it does not claim arbitrary invalid byte sequences become valid Unicode JSON. Keep parser failures observable so malformed records do not silently disappear during an incident.

Record only necessary request data, with appropriate retention and access controls. A correctly escaped field can still contain sensitive information or an untrusted identity claim. Test the complete shipping and querying path separately: local JSON validity establishes a useful contract, but does not prove that rotation, collection, delivery, and retention preserve every event.

Sources

nginx HTTP log module, log formats, escape modes, and request-completion records, checked 2026-09-12.
nginx 1.28.0 log implementation, different missing-variable behavior for default and JSON escaping, checked 2026-09-12.
nginx core variables, request identifiers and HTTP request variables, checked 2026-09-12.

Structured logs make their query contract explicit

Tue, 09 Mar 2027 00:00:00 +0000

Structured logging assigns stable meaning to fields instead of asking each query to infer everything from message text. A record might contain an event name, request identifier, numeric duration, severity, and a human-readable message. Free text remains useful and can also be searched or parsed. The practical gain is that routine questions can use a documented schema whose meaning survives changes to explanatory wording.

That schema needs more than curly braces. A duration stored as a number supports numeric comparisons, while a duration stored as a string may be indexed or compared differently. Milliseconds and seconds must not share an ambiguous field name. An event name should describe what happened consistently across versions. A schema version can identify deliberate format changes, but it does not automatically transform old records or keep every consumer compatible.

Put context on the correct record

A request identifier is useful only if it belongs to the operation that emitted the record. Concurrent requests make a process-global mutable variable unsuitable for that purpose. Python context variables can hold task-local state across asynchronous work, provided values are set and reset at the intended scope. This does not mean every thread or custom execution mechanism inherits context in the same way; verify the actual boundary.

Capture context while it is available. This example formats each record synchronously in the emitting task, so its formatter can read the current context variable. A queue-based logging pipeline needs a different placement: attach the required fields before handing the record to another thread or process. Reading the listener’s current request context would associate the record with the listener, rather than reliably recovering the original request.

When it breaks

Hand-concatenating JSON can turn quotes or line breaks from a message into invalid records. A line-oriented collector may then split one application event into several apparent events. Use a JSON serializer and one complete JSON object per physical line. The message can retain a logical newline inside its string value while the serialized line contains an escaped representation. Rendering that message later should still keep its record boundary clear.

Structured output does not automatically create indexes, reduce query bills, or guarantee delivery. Unbounded fields, accidental type changes, missing identifiers, and sensitive values can make a JSON stream difficult or inappropriate to use. Select fields according to real queries, keep the schema stable, and verify the collector’s parser and retention rules. Searchable structure is a contract between producers and consumers, not a property established by the file extension.

Check it quickly

Save this as structured-logging.py and run it with Python; the executed version was 3.14.4. It uses only standard-library modules. Two asynchronous requests each log a start event, wait at a shared gate, then log a completion event containing a deliberately troublesome message. The durations are fixed synthetic values for checking JSON types, not measurements of request performance. Output is captured in memory for assertions.

import asyncio
import io
import json
import logging
from collections import Counter
from contextvars import ContextVar

request_id = ContextVar("request_id", default=None)
stream = io.StringIO()


class JsonFormatter(logging.Formatter):
 def format(self, record):
 return json.dumps({"schema_version": 1, "event": record.event,
 "request_id": request_id.get(), "worker": record.worker,
 "duration_ms": record.duration_ms,
 "level": record.levelname, "message": record.getMessage()},
 ensure_ascii=False, separators=(",", ":"))


logger = logging.getLogger("structured-log-lab")
logger.setLevel(logging.INFO)
logger.propagate = False
handler = logging.StreamHandler(stream)
handler.setFormatter(JsonFormatter())
logger.addHandler(handler)


async def request(name, ready, release):
 token = request_id.set(name)
 try:
 logger.info("starting %s", name, extra={"event": "request.started",
 "worker": name, "duration_ms": 0})
 ready.set()
 await release.wait()
 await asyncio.sleep(0)
 logger.info('untrusted line\n{"event":"forged"}',
 extra={"event": "request.completed", "worker": name, "duration_ms": 4})
 finally:
 request_id.reset(token)


async def main():
 ready = [asyncio.Event(), asyncio.Event()]
 release = asyncio.Event()
 tasks = [asyncio.create_task(request(name, signal, release))
 for name, signal in zip(("req-a", "req-b"), ready)]
 await asyncio.gather(*(signal.wait() for signal in ready))
 release.set()
 await asyncio.gather(*tasks)
 assert request_id.get() is None


try:
 asyncio.run(main())
finally:
 logger.removeHandler(handler)
 handler.close()
lines = stream.getvalue().splitlines()
records = [json.loads(line) for line in lines]
assert len(lines) == 4
assert Counter(record["request_id"] for record in records) == {"req-a": 2, "req-b": 2}
assert all(record["request_id"] == record["worker"] for record in records)
assert all(type(record["duration_ms"]) is int for record in records)
completed = [record for record in records if record["event"] == "request.completed"]
assert len(completed) == 2 and all("\n" in record["message"] for record in completed)
assert not any(record["event"] == "forged" for record in records)
print(json.dumps({"physical_lines": len(lines), "parsed_records": len(records),
 "requests": 2, "context_mixing": False,
 "numeric_duration_preserved": True, "forged_records": 0}, indent=2))

The assertions require four physical lines and four parsed records, with two records for each request identifier. Each record’s identifier must match the worker that emitted it, even though the requests overlap. Duration values remain JSON numbers. The embedded newline survives inside each decoded completion message, while no additional record named forged appears. Resetting each token also leaves the enclosing context without a request identifier.

This verifies task isolation, synchronous formatting, and line-safe serialization. It does not exercise a background queue, collector retry, storage indexing, or secret redaction. When integrating it, check the complete path with representative fields and the actual concurrency model. Inspect what the backend stores after parsing, and test the queries that motivated those fields. A successful producer-side serialization check is one useful boundary test, while a dependable logging system also needs evidence that the intended records reach their consumers with their meaning intact.

Sources

Python logging cookbook, contextual records and logging across concurrent work, checked 2026-09-13.
Python contextvars, task-local context and token reset behavior, checked 2026-09-13.
Python json, serialization, escaped characters, and JSON value types, checked 2026-09-13.
JSON Lines, one complete JSON value per physical line, checked 2026-09-13.

An access log separates client outcomes from upstream attempts

Mon, 01 Mar 2027 00:00:00 +0000

An nginx access log records request information when logging runs at the end of processing. The selected fields can distinguish what the client requested, which URI ultimately handled it, what response was sent, and what happened during upstream attempts. These are different observations. A useful format keeps them separate instead of assuming that one status, duration, or path explains the whole request by itself.

When it breaks

The final client status can conceal an earlier upstream failure. If nginx retries another server successfully, the client may receive 200 while an attempt list records a failed connection first. Conversely, a locally generated response has no upstream operation to time. Treating every request as exactly one backend response loses the information needed to explain retries and local handling.

Original and current URI fields also differ after internal redirects. $request retains the request line, and $request_uri retains the original URI and query. $uri can reflect a later normalized or rewritten path. An investigator who logs only the final URI may not know which public address triggered it, while logging only the original path can hide the final handler’s context.

Timing fields are not interchangeable latency components that can always be subtracted into a precise diagnosis. Request time includes processing through the client-facing completion point used by nginx logging. Upstream timing fields describe their own phases and can contain several attempt values. Their resolution, missing values, and boundaries matter; a zero-looking millisecond value does not establish that no work occurred.

Check it quickly

Run the script with host Python, Docker, and the pinned nginx and Python images. The recorded environment used nginx 1.28.0 and Python 3.11.16 on Linux arm64 with Docker 29.3.1. It configures a real backend plus an unreachable port inside the proxy container, validates nginx configuration, publishes only a host-loopback proxy port, and removes its own resources after inspecting the logs:

from contextlib import contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import http.client,json,socket,subprocess,time,uuid

NGINX='nginx@sha256:552e7481ca93ffccd046aa658dbbed22caefbc09c66fa7cd247cbb90b8a5c609'
PYTHON='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def wait(predicate):
 deadline=time.monotonic()+15
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except OSError:pass
 time.sleep(.1)
 raise AssertionError('fixture did not become ready')

@contextmanager
def lab(configuration,files=None,backend=None):
 name='xbcatm-nginx-'+uuid.uuid4().hex[:10];tags=[name+':proxy',name+':backend'];containers=[name+'-proxy',name+'-backend']
 try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'files').mkdir();(root/'nginx.conf').write_text(configuration)
 for path,content in (files or {}).items():
 target=root/'files'/path;target.parent.mkdir(parents=True,exist_ok=True);target.write_text(content)
 docker('network','create',name)
 if backend is not None:
 (root/'backend.py').write_text(backend)
 (root/'backend.Dockerfile').write_text(f'FROM {PYTHON}\nCOPY backend.py /backend.py\nCMD ["python", "/backend.py"]\n')
 docker('build','--network','none','-f',str(root/'backend.Dockerfile'),'-t',tags[1],str(root))
 docker('run','-d','--name',containers[1],'--network',name,'--network-alias','backend',tags[1])
 wait(lambda:docker('exec',containers[1],'python','-c','import socket;s=socket.create_connection(("127.0.0.1",8081),timeout=.5);s.close()',check=False).returncode==0)
 (root/'Dockerfile').write_text(f'FROM {NGINX}\nCOPY nginx.conf /etc/nginx/nginx.conf\nCOPY files /srv\n')
 docker('build','--network','none','-t',tags[0],str(root))
 docker('run','--rm','--network',name,tags[0],'nginx','-t')
 docker('run','-d','--name',containers[0],'--network',name,'-p','127.0.0.1::8080',tags[0])
 details=json.loads(docker('inspect',containers[0]).stdout)[0]
 binding=details['NetworkSettings']['Ports']['8080/tcp'][0];assert binding['HostIp']=='127.0.0.1';port=int(binding['HostPort'])
 def ready():
 connection=socket.create_connection(('127.0.0.1',port),timeout=.5);connection.close();return True
 wait(ready)
 def request(path,method='GET',headers=None,body=None):
 connection=http.client.HTTPConnection('127.0.0.1',port,timeout=5)
 try:
 connection.request(method,path,body=body,headers=headers or {})
 response=connection.getresponse();return response.status,{k.lower():v for k,v in response.getheaders()},response.read()
 finally:connection.close()
 yield containers[0],request
 finally:
 docker('rm','-f',*containers,check=False);docker('image','rm',*tags,check=False);docker('network','rm',name,check=False)

def config(locations,extra=''):
 return 'worker_processes 1;\nerror_log /tmp/error.log notice;\nevents {}\nhttp {\naccess_log off;\n'+extra+'\nserver { listen 8080; root /srv;\n'+locations+'\n}\n}\n'

backend=r'''from http.server import BaseHTTPRequestHandler,HTTPServer
import json
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 body=self.rfile.read(int(self.headers.get('Content-Length','0')))
 document={'path':self.path,'method':self.command,'body':body.decode(),'headers':{key:self.headers.get(key) for key in ['Host','X-Forwarded-For','X-Real-IP','X-Forwarded-Proto','X-Forwarded-Host']}}
 data=json.dumps(document).encode()
 self.send_response(200);self.send_header('Content-Type','application/json');self.send_header('Content-Length',str(len(data)));self.send_header('X-Backend-Info','fixture-node');self.end_headers();self.wfile.write(data)
 do_POST=do_GET
 def log_message(self,*args):pass
HTTPServer(('0.0.0.0',8081),Handler).serve_forever()
'''

fields={'request':'$request','request_uri':'$request_uri','uri':'$uri','status':'$status','upstream_status':'$upstream_status','upstream_addr':'$upstream_addr','request_time':'$request_time','upstream_connect_time':'$upstream_connect_time','upstream_header_time':'$upstream_header_time','upstream_response_time':'$upstream_response_time','body_bytes_sent':'$body_bytes_sent','bytes_sent':'$bytes_sent'}
extra="log_format details escape=json '"+json.dumps(fields,separators=(',',':'))+"';\nupstream service_pool { server 127.0.0.1:1 max_fails=1; server backend:8081 backup; }"
locations=r'''
access_log /tmp/access.json details;
location = /local { return 204; }
location = /rewrite { rewrite ^ /normalized last; }
location = /normalized { return 200 "normalized"; }
location /proxy {
 proxy_next_upstream error timeout;
 proxy_next_upstream_tries 2;
 proxy_pass http://service_pool;
}
'''
with lab(config(locations,extra),backend=backend) as (container,request):
 assert request('/local')[0]==204
 assert request('/rewrite?x=1')[2]==b'normalized'
 status,headers,body=request('/proxy?x=1');assert status==200
 def lines():return docker('exec',container,'cat','/tmp/access.json').stdout.splitlines()
 wait(lambda:len(lines())>=3)
 records={record['request_uri']:record for record in map(json.loads,lines())}
 print(json.dumps(records,indent=2),flush=True)
 local=records['/local'];assert local['status']=='204' and local['upstream_status']=='' and local['upstream_response_time']==''
 assert local['body_bytes_sent']=='0' and int(local['bytes_sent'])>0
 changed=records['/rewrite?x=1'];assert changed['request']=='GET /rewrite?x=1 HTTP/1.1' and changed['uri']=='/normalized'
 proxy=records['/proxy?x=1'];assert proxy['status']=='200' and proxy['upstream_status']=='502, 200'
 assert len(proxy['upstream_addr'].split(', '))==2 and int(proxy['body_bytes_sent'])==len(body)
 assert float(proxy['request_time'])>=0 and len(proxy['upstream_response_time'].split(', '))==2
 print('a local 204 has no upstream and no body bytes, but still sends response-header bytes')
 print('the final uri reflects the internal redirect while request and request_uri retain the original request')
 print('one client-visible 200 records upstream attempts 502, 200 and separate attempt-address/timing lists')

The /local request returns 204 without contacting an upstream. Its log has zero body bytes but a positive bytes_sent value because response headers were sent. In this JSON-escaped format, absent upstream fields are empty strings. This is different from a measured zero-second upstream operation and from the dash used by the tested version’s default escaping path.

The request /rewrite?x=1 internally becomes /normalized. The log retains GET /rewrite?x=1 HTTP/1.1 as the request line and the original URI with its query, while uri records /normalized. This verifies the difference at the actual log-write point, rather than inferring it only from the rewrite declaration or the response body.

The proxy request first tries loopback port one, where the disposable proxy container has no listener. nginx records that failed attempt with status 502 and then reaches the configured backup server. The final client response is 200, while upstream_status is the string 502, 200. The first value is nginx’s recorded failure outcome, not evidence that a backend actually returned an HTTP 502 response.

The same event contains two upstream addresses and two response-time entries. Connect and header timings can contain a dash for an attempt that never reached those phases, even though the overall variable exists. That differs from an entirely absent upstream variable. Preserve these relationships when parsing instead of replacing every unusual value with an indistinguishable numeric zero.

The logged body byte count equals the actual response-body length read by the client in this uncompressed fixture. bytes_sent includes more HTTP response bytes, but neither field is presented as a complete accounting of Ethernet, TCP, or TLS overhead. The example uses the counters to distinguish body content from header traffic, not to claim a physical-network billing measurement.

The script waits until the three events are present before parsing them. Logging occurs after request handling, and configured buffering can delay visibility further. A successful HTTP response does not guarantee that a remote log collector has already received or retained the corresponding event.

Keep diagnostic meaning in the schema

Choose fields that answer concrete operational questions: original target, final route, client status, upstream attempt outcomes, and relevant durations. Keep attempt lists intact until a parser with the documented grammar separates them. Multiple upstream groups or internal transitions can introduce additional structure beyond the simple two-attempt case shown here.

Avoid treating client-controlled forwarding headers as verified caller identity. Also consider whether query strings or other request fields contain sensitive values before retaining them. Richer logs are useful only when the information has a clear purpose and appropriate access and retention controls.

Test format changes by producing representative local responses, retries, rewrites, and failures, then parsing actual records. Validate rotation and collection separately. A trustworthy access log connects each field to its measured event and preserves missing or multi-attempt information instead of flattening it into a misleading summary.

Sources

nginx HTTP log module, request-completion timing, byte counters, and log-format fields, checked 2026-09-12.
nginx upstream variables, per-attempt addresses, statuses, and timing lists, checked 2026-09-12.
nginx core request variables, original request text and current normalized URI, checked 2026-09-12.
nginx 1.28.0 log implementation, JSON-escaped missing-variable representation, checked 2026-09-12.

Timestamps need explicit time semantics

Sun, 29 Nov 2026 00:00:00 +0000

A wall-clock label such as 2026-11-01 01:30 does not identify one instant by itself. Its interpretation depends on a timezone and sometimes on which occurrence was intended during a clock transition. Conversely, an explicitly defined Unix timestamp already carries epoch and unit semantics without displaying a zone name. Decide which kind of value your system is storing before choosing a format.

An offset is not a timezone rulebook

An offset-bearing timestamp identifies how its civil fields relate to UTC. The strings 12:00Z and 20:00+08:00 on the same date can represent the same instant. A named IANA zone additionally describes historical and future civil-time rules, including daylight-saving changes. An offset of minus seven hours is not a substitute for America/Los_Angeles when scheduling next month’s local appointment.

For events that already happened, retain an unambiguous instant and render it in the reader’s desired zone. A recurring instruction such as nine each business morning needs the local-time intent and applicable zone rules too. Political changes can alter future offsets, so record the relevant timezone data version when reproducibility matters. Converting today’s offset into a permanent schedule silently changes that meaning.

UTC formatting does not prove clock accuracy or causality. Two machines can both print a Z suffix while their clocks disagree. Timestamps with limited precision can also tie. Keep event identifiers and causal context where ordering matters, and use an appropriate monotonic clock for local elapsed intervals. Timezone conversion cannot repair a clock that measured the wrong instant.

Numeric interfaces must document seconds versus milliseconds and the epoch they use. A large integer is not self-describing, and interpreting its unit incorrectly cannot be corrected later by choosing another display timezone.

When it breaks

Sorting arbitrary timestamp strings can disagree with chronological order. RFC 3339 describes conditions under which text sorting works, including consistent timezone representation and fractional precision. Normalize parsed instants before sorting heterogeneous offsets. A visually later local hour can correspond to an earlier UTC instant, so comparing characters is not a replacement for interpreting the timestamp.

The autumn clock transition can repeat a local hour. Python’s ZoneInfo uses fold to distinguish its two occurrences. The spring transition can instead make a local time nonexistent. Attaching a zone object does not automatically reject such a missing time. An application accepting local appointments must define how to reject or resolve these cases rather than silently inventing the user’s intent.

Arithmetic needs equal care. Operations on datetime objects sharing a tzinfo can follow wall-time semantics and ignore the distinction implied by fold. Convert both values to UTC when you intend to compare elapsed instants. Keep that choice explicit: the interval between two appointments measured by local clock labels may differ from actual elapsed seconds across a transition.

Check it quickly

Save the following as timezone.py and run python3 timezone.py with Python 3.11 or newer and IANA timezone data available. The recorded run used Python 3.11.2 and Debian tzdata 2026b. The script uses fixed examples and does not change the machine’s clock or timezone.

from datetime import datetime, timezone
from zoneinfo import ZoneInfo
import json


def instant(text):
 value = datetime.fromisoformat(text)
 if value.tzinfo is None:
 raise ValueError('an instant requires an explicit offset')
 return value.astimezone(timezone.utc)


same = ['2026-09-13T12:00:00Z', '2026-09-13T20:00:00+08:00']
assert instant(same[0]) == instant(same[1])
values = ['2026-01-01T10:00:00+02:00', '2026-01-01T09:30:00+00:00']
assert sorted(values) != sorted(values, key=instant)
try:
 instant('2026-09-13T12:00:00')
except ValueError as error:
 rejection = str(error)
else:
 raise AssertionError('ambiguous instant accepted')

zone = ZoneInfo('America/Los_Angeles')
first = datetime(2026, 11, 1, 1, 30, tzinfo=zone, fold=0)
second = first.replace(fold=1)
first_utc, second_utc = first.astimezone(timezone.utc), second.astimezone(timezone.utc)
assert (second_utc - first_utc).total_seconds() == 3600
missing = datetime(2026, 3, 8, 2, 30)
round_trips = []
for fold in (0, 1):
 candidate = missing.replace(tzinfo=zone, fold=fold)
 returned = candidate.astimezone(timezone.utc).astimezone(zone)
 assert returned.replace(tzinfo=None) != missing
 round_trips.append(returned.isoformat())
print(json.dumps({'same_instant': True, 'lexical_order': sorted(values),
 'instant_order': sorted(values, key=instant), 'naive_rejected': rejection,
 'repeated_wall_time': [first.isoformat(), second.isoformat()],
 'actual_separation_seconds': 3600, 'nonexistent_time_round_trips': round_trips}))

The first assertion shows that two offset representations identify the same instant. Next, lexical sorting puts 09:30+00:00 before 10:00+02:00, while chronological sorting correctly reverses them. The helper rejects a timezone-free value instead of borrowing the host’s current zone. It is a deliberately small instant helper, not a complete RFC 3339 validator.

For Los Angeles on November 1, 2026, the two 01:30 values carry offsets of minus seven and minus eight hours. Converting them to UTC reveals 3,600 elapsed seconds. For March 8 at 02:30, neither fold survives a UTC round trip with the original local fields: the returned times are 03:30 and 01:30. All assertions passed. These examples verify specific transition rules in the installed database; they do not promise that every jurisdiction retains today’s rules forever. Preserve instant meaning, local scheduling intent, and parsing policy as separate parts of the data contract.

Sources

RFC 3339: Date and Time on the Internet, offset-bearing timestamps and conditions for string sorting, checked 2026-09-13.
Python: datetime, aware values, UTC conversion and datetime arithmetic, checked 2026-09-13.
Python: zoneinfo, IANA rules, ambiguous transitions and fold, checked 2026-09-13.

Logrotate applies a rotation policy when it runs

Sat, 21 Nov 2026 00:00:00 +0000

Logrotate evaluates configuration when invoked, rotates eligible logs, and applies the selected retention and compression actions. A timer or another scheduler determines when that evaluation happens. The tool does not continuously cap file size, and renaming a log does not automatically redirect an application’s open descriptor. A working setup combines evaluation frequency, file policy, and the writer’s actual reopening behavior.

When it breaks

A size threshold can be exceeded between invocations, particularly during bursts or when the scheduler stops running. rotate limits retained generations, not elapsed history or all possible temporary disk use. Multiplying the threshold by the generation count therefore is not a hard capacity guarantee. The active file, growth between checks, and compression or copying work also consume space.

create specifies the replacement file’s mode and ownership; it does not signal the application. A postrotate hook can invoke a supported reopening mechanism, but HUP is not universally a reload request. delaycompress postpones compression by a rotation cycle; it does not fix a writer that never stops using an old inode.

Script failures can occur after file operations have already succeeded. A nonzero logrotate result is not a rollback of every rename or creation. Repeating forced rotation without inspecting the actual files and writer state can rotate again or discard needed generations. Diagnose the partial outcome instead of assuming that failure means nothing changed.

Check it quickly

Save this as logrotate-cli.py and run python3 logrotate-cli.py as root in a disposable Linux container with Python, logrotate, and gzip installed. The recorded run used Linux 6.12.76, Python 3.11.2, logrotate 3.21.0, and ext4. It uses a private configuration, private state files, and one logger child whose HUP handler explicitly reopens its own temporary log:

from pathlib import Path
from tempfile import TemporaryDirectory
import gzip,os,stat,subprocess,sys

worker=r'''
import os,signal,sys
path=sys.argv[1];fd=os.open(path,os.O_WRONLY|os.O_CREAT|os.O_APPEND,0o600)
def reopen(signum,frame):
 global fd
 os.close(fd);fd=os.open(path,os.O_WRONLY|os.O_APPEND)
 print('reopened',flush=True)
signal.signal(signal.SIGHUP,reopen)
try:
 print('ready',flush=True)
 for line in sys.stdin:
 if line=='QUIT\n':break
 os.write(fd,line.encode());print('wrote',flush=True)
finally:os.close(fd)
'''
assert os.geteuid()==0,'use a disposable container for the root-owned fixture'
with TemporaryDirectory(prefix='xbcatm-logrotate-') as directory:
 root=Path(directory);active=root/'app.log';configuration=root/'logrotate.conf';state=root/'state'
 process=subprocess.Popen([sys.executable,'-c',worker,str(active)],stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 try:
 assert process.stdout.readline().strip()=='ready'
 configuration.write_text(f'''"{active}" {{
 size 64
 rotate 2
 compress
 delaycompress
 missingok
 notifempty
 create 0600 root root
 postrotate
 /bin/kill -HUP {process.pid}
 endscript
}}
''')
 def append(text):
 process.stdin.write(text+'\n');process.stdin.flush();assert process.stdout.readline().strip()=='wrote'
 def rotate(*options):return subprocess.run(['logrotate',*options,'--state',str(state),str(configuration)],capture_output=True,text=True,timeout=5)
 append('short');initial=active.read_bytes()
 result=rotate('--debug');assert result.returncode==0,result.stderr
 assert active.read_bytes()==initial and not state.exists() and not (root/'app.log.1').exists()
 result=rotate();assert result.returncode==0,result.stderr
 assert state.exists() and not (root/'app.log.1').exists()
 print('debug changes neither logs nor state; a normal below-threshold run records state without rotating')
 first=initial+b'first-'+b'a'*80+b'\n';second=b'second-'+b'b'*80+b'\n';third=b'third-'+b'c'*80+b'\n'
 for number,payload in enumerate([first,second,third],1):
 append((payload[len(initial):] if number==1 else payload).decode().rstrip('\n'))
 result=rotate();assert result.returncode==0,result.stderr
 assert process.stdout.readline().strip()=='reopened'
 assert active.read_bytes()==b'' and stat.S_IMODE(active.stat().st_mode)==0o600
 assert (root/'app.log.1').read_bytes()==payload
 if number>=2:
 with gzip.open(root/'app.log.2.gz','rb') as handle:assert handle.read()==(first if number==2 else second)
 assert not (root/'app.log.3.gz').exists()
 append('new-active');assert active.read_bytes()==b'new-active\n'
 print('three actual rotations reopen the writer, delay compression one generation, retain two archives, and preserve the expected bytes')
 failed=root/'failed.log';failed.write_bytes(b'failure-fixture')
 bad=root/'failed.conf';bad.write_text(f'''"{failed}" {{
 size 1
 rotate 1
 create 0600 root root
 postrotate
 /bin/false
 endscript
}}
''')
 result=subprocess.run(['logrotate','--state',str(root/'failed.state'),str(bad)],capture_output=True,text=True,timeout=5)
 assert result.returncode!=0
 assert (root/'failed.log.1').read_bytes()==b'failure-fixture' and failed.read_bytes()==b''
 print('a failed postrotate command returns failure after rename/create side effects have already happened')
 output,error=process.communicate('QUIT\n',timeout=5);assert process.returncode==0,error
 finally:
 if process.poll() is None:process.kill();process.wait()

The configuration selects a 64-byte size threshold, two generations, compression with one-cycle delay, and creation of a root-owned mode-0600 replacement. Its hook signals only the controlled child’s known PID. The logger acknowledges writes and reopening, letting the controller verify the handoff instead of inferring it from a successful signal command.

With a short log, debug mode changes neither log files nor the private state file. A normal invocation then creates state but does not rotate the below-threshold log. These checks distinguish configuration inspection from mutation and demonstrate that a successful invocation can legitimately perform no rotation.

The script next writes enough data to cross the threshold and invokes logrotate three times, adding a known payload before each invocation. Each rotation creates an empty mode-0600 active file, and the child acknowledges reopening it. The newest archive remains uncompressed as generation one, while the preceding generation is available as a gzip-compressed generation two.

The fixture decompresses the older archive and compares exact payload bytes. After the third rotation, generation two contains the second payload and generation one contains the third; no third compressed generation remains. A final record lands in the current active file. This verifies retention, compression, and writer placement together, not merely the appearance of expected filenames.

A second private configuration deliberately runs a failing postrotate command. Logrotate returns failure, but the old content is already in generation one and the new active file exists empty. This controlled failure exposes side effects that remain after the error. The fixture then stops and reaps its logger and removes all temporary files.

Operate the policy as a complete workflow

Use a state file consistent with the intended rotation job, and account for its locking when scheduling concurrent invocations. Separate state files are not a shared history. Debug mode is useful for reviewing decisions, while force mode changes eligibility and should not be mistaken for proof that the ordinary schedule will rotate at the intended time.

Choose directives according to the desired trigger. Size and time policies have specific interactions and ordering rules; combining familiar words does not automatically produce an AND condition. Verify the effective policy against representative small, large, empty, and missing files, and check the application identity can write the newly created file.

Monitor whether scheduled evaluations and reopening actually succeed, alongside filesystem capacity and the retained history window. If copytruncate is necessary, explicitly accept and test its copy-to-truncate loss window. Use independent log shipping or backups when required by the retention contract, and verify that representative old records remain readable after compression, rotation, and recovery.

Sources

logrotate 3.21.0 manual, size, rotate, create, compression, state, debugging, and script failures, checked 2026-09-12.
rename(2), why renaming a pathname does not redirect a live writer, checked 2026-09-12.
signal(7), application-defined HUP handling versus default termination, checked 2026-09-12.

Metrics and logs need an explicit population

Fri, 16 Oct 2026 00:00:00 +0000

Metrics and logs are different representations of observed behavior, not competing definitions of truth. A metric usually summarizes measurements under a set of dimensions. A log record describes an event and can carry detailed context. Both can answer aggregate questions if their coverage and semantics support the calculation. Neither becomes authoritative merely because its dashboard looks cleaner or its records contain more fields.

For a checkout service, a completed-request counter can classify every terminal outcome as success or error. A few bounded dimensions, such as route and outcome, make it practical to compare traffic and failure rates over time. A diagnostic log can then explain a particular failed request using its correlation identifier, error category, and dependency response. These serve related questions, but their denominators must still be established separately.

Decide what the measurement includes

A cumulative counter is not itself a requests-per-second measurement. A live rate calculation needs observations across time and handling for resets. A complete synthetic batch, however, can be counted directly because its beginning, end, and membership are known. Do not silently replace a windowed production rate with a ratio of unrelated snapshots, or compare a sampled event stream against a counter covering every request.

Logs are not limited to manual reading. Consistently structured, complete terminal events can be aggregated into the same request totals as a counter. The tradeoff includes event volume, indexing, query latency, retention, and operational complexity. Metrics are often economical for repeated aggregate questions, but a metric with a new label value for every request can be expensive. The representation alone does not settle cost.

When it breaks

Consider a hundred completed requests with four failures. An incident-focused log selection retains five requests, including two failures. The complete population has a four-percent failure fraction, while the selected logs have a forty-percent fraction. Both calculations are arithmetically correct. Presenting the second as the service-wide failure rate is wrong because the five records were not a complete or representative population.

Counting retry failures as terminal request failures creates another mismatch: a request may fail an attempt and later succeed. Missing logs, duplicated deliveries, differing time windows, and incompatible route definitions can also make apparently related charts disagree. Define one terminal outcome per request for the reliability measure, then keep attempt-level evidence for diagnosis. Never repair the discrepancy merely by changing the chart label or averaging the two percentages.

Check it quickly

Save this as metrics-logs.py and run it with Python 3.14.4 and Docker available. The script uses the prom/prometheus:v3.5.1 image to run the real promtool check metrics validator. It creates a temporary container without network access or host mounts and removes it afterward. The workload is synthetic and complete by construction; no production request stream is scraped or sampled here.

import json
import subprocess
import uuid
from collections import Counter

# A complete synthetic population, with exactly one terminal outcome per request.
requests = [{"request_id": f"r{i}", "route": "/checkout",
 "outcome": "error" if i in {7, 18, 53, 94} else "success"}
 for i in range(1, 101)]
counter = Counter(request["outcome"] for request in requests)
selected_logs = [request for i, request in enumerate(requests, 1)
 if i in {1, 7, 18, 33, 55}]
logged_errors = sum(record["outcome"] == "error" for record in selected_logs)
text = "# HELP demo_requests_total Completed synthetic requests.\n# TYPE demo_requests_total counter\n"
for outcome, count in sorted(counter.items()):
 text += f'demo_requests_total{{route="/checkout",outcome="{outcome}"}} {count}\n'
name = "xbcatm-metric-format-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "-i", "--name", name, "--network", "none",
 "--entrypoint", "/bin/promtool", "prom/prometheus:v3.5.1", "check", "metrics"],
 input=text, text=True, capture_output=True, timeout=30,
 )
 assert result.returncode == 0, result.stdout + result.stderr
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
population_rate = counter["error"] / len(requests)
selected_rate = logged_errors / len(selected_logs)
assert population_rate == 0.04 and selected_rate == 0.4
assert "request_id" not in text and len(counter) == 2
# Full event coverage would recover the same aggregate; these five logs cannot.
assert sum(r["outcome"] == "error" for r in requests) == counter["error"]
print(json.dumps({"requests": len(requests), "errors": counter["error"],
 "population_error_rate": population_rate, "exposed_metric_series": len(counter),
 "selected_logs": len(selected_logs), "selected_log_errors": logged_errors,
 "selected_log_error_rate": selected_rate,
 "exposition_validated_by": "promtool 3.5.1"}, indent=2))

The result should report one hundred requests, four errors, and a population fraction of 0.04. Five selected log records contain two errors, producing 0.4. The exposition has two counter series distinguished by outcome, and its labels contain no request identifier. Promtool validates the exported format; the Python assertions establish the arithmetic and the deliberate difference in populations. This is not a live scrape, counter-reset test, or storage-cost benchmark.

For real investigations, use bounded metric dimensions to locate the affected route, outcome, or deployment. Follow a correlation identifier into logs or traces when individual context is needed. Keep that identifier out of ordinary metric labels unless a justified cardinality budget supports it; exemplars can provide a separate link when the stack supports them. Finally, document event completeness and selection rules next to derived rates. A chart becomes trustworthy when its unit, population, and time basis are clear enough for another person to reproduce its meaning.

Sources

OpenTelemetry metrics, measurements and aggregation over a population, checked 2026-09-13.
OpenTelemetry logs, event records and their relationship to other telemetry, checked 2026-09-13.
Prometheus metric and label naming, bounded labels and the cost of high-cardinality dimensions, checked 2026-09-13.

Give a log query a question and a denominator

Sun, 06 Sep 2026 00:00:00 +0000

A useful logging design starts with a question precise enough to become a query. “Show errors” is different from “How many checkout requests finally failed between ten and eleven UTC?” The first locates diagnostic events. The second needs a population of completed requests, a definition of failure, and a time rule. Collecting more text does not supply these missing semantics automatically.

For a request-outcome question, define a terminal event emitted once per logical request. Keep separate attempt events when retries matter, and give events stable identities so delivery duplicates can be recognized. A request identifier joins its events; an event identifier distinguishes one event from another. Reusing the request identifier as every event’s deduplication key would discard legitimate attempts and the final outcome together.

Make the query contract visible

Choose whether the window uses event occurrence time or arrival time. These differ when a collector is delayed. A half-open interval includes the start and excludes the end, allowing adjacent windows to meet without double-counting their boundary. Normalize timestamps to an unambiguous time basis before comparing them. A local clock string without its offset is insufficient evidence for a UTC incident window.

Define which terminal outcomes count as failures. Server errors, client cancellations, and rejected input need not belong to the same reliability measure. Also state whether every relevant request emits a terminal event and whether the collection path can lose records. A query can be perfectly correct for stored events while giving an incomplete account of the service. Retention and selection policies belong beside the interpretation, not just in collector configuration.

When it breaks

A request that fails an attempt and then succeeds produces a real error event without being a failed completed request. Counting every error line inflates the final failure count. Delivery duplication can inflate it again. Deduplication must use an immutable event identity; two different payloads claiming the same identity are a data-integrity problem, not evidence that either can safely be ignored.

An empty window presents a different trap. Zero recorded failures divided by zero recorded requests is not a measured zero-percent failure rate. Preserve the absence of a denominator. Otherwise a broken collector can make a reliability chart appear healthy. When an incident requires one request’s explanation, retrieve its ordered events and dependency details after establishing the aggregate; a count alone cannot explain which retry recovered or why another request failed.

Check it quickly

Save this script as log-questions.py and run it with Python. It was executed on Python 3.14.4 and uses the standard SQLite module with an in-memory database. The eight supplied delivery records contain one exact duplicate. Event times are constructed around a fixed UTC hour, including one event just before the window and one exactly at its end. No live collector or external database is involved.

import json
import sqlite3
from datetime import datetime


def timestamp(value):
 parsed = datetime.fromisoformat(value)
 assert parsed.tzinfo is not None
 return int(parsed.timestamp())


start = timestamp("2026-09-01T10:00:00+00:00")
end = timestamp("2026-09-01T11:00:00+00:00")
rows = [
 ("e0", "r0", "request.completed", start - 1, 503),
 ("e1", "r1", "attempt.failed", start + 600, 503),
 ("e2", "r1", "request.completed", start + 660, 200),
 ("e3", "r2", "attempt.failed", start + 900, 503),
 ("e4", "r2", "request.completed", start + 960, 503),
 ("e4", "r2", "request.completed", start + 960, 503),
 ("e5", "r3", "request.completed", start + 1080, 200),
 ("e6", "r4", "request.completed", end, 503),
]
db = sqlite3.connect(":memory:")
db.execute("""CREATE TABLE events (
 event_id TEXT PRIMARY KEY, request_id TEXT,
 event_type TEXT, occurred_at INTEGER, status INTEGER)""")
db.executemany("""INSERT INTO events VALUES (?, ?, ?, ?, ?)
 ON CONFLICT(event_id) DO NOTHING""", rows)
stored = db.execute("SELECT count(*) FROM events").fetchone()[0]
assert stored == 7
error_events = db.execute("""SELECT count(*) FROM events
 WHERE occurred_at >= ? AND occurred_at < ? AND status >= 500""",
 (start, end)).fetchone()[0]
query = """SELECT count(*), coalesce(sum(status >= 500), 0)
 FROM events WHERE event_type = 'request.completed'
 AND occurred_at >= ? AND occurred_at < ?"""
total, failed = db.execute(query, (start, end)).fetchone()
completed = db.execute("""SELECT request_id, status FROM events
 WHERE event_type = 'request.completed' AND occurred_at >= ? AND occurred_at < ?
 ORDER BY request_id""", (start, end)).fetchall()
assert completed == [("r1", 200), ("r2", 503), ("r3", 200)]
assert (error_events, total, failed) == (3, 3, 1)
empty_total, empty_failed = db.execute(query, (end + 1, end + 3600)).fetchone()
empty_rate = empty_failed / empty_total if empty_total else None
assert (empty_total, empty_failed, empty_rate) == (0, 0, None)
print(json.dumps({"received_events": len(rows), "deduplicated_events": stored,
 "error_events_in_window": error_events, "completed_requests": total,
 "failed_requests": failed, "terminal_failure_rate": failed / total,
 "empty_window_rate": empty_rate}, indent=2))
db.close()

Seven unique events remain after deduplication. Within the window, the broad status query finds three error events: a failed attempt for request one, a failed attempt for request two, and request two’s failed completion. The terminal query instead finds three completed requests, of which one failed. Request one recovered successfully, request three succeeded immediately, and the boundary events are excluded. The final failure fraction is therefore one third.

The second query targets an empty later interval. Its count is zero, its failure sum is normalized to zero, and the derived rate remains null because there is no denominator. These assertions establish the intended query semantics for complete, controlled records. For production use, verify event emission and delivery coverage, inspect conflicting identities, and compare terminal totals with independent request counters. The useful outcome is a question another operator can reproduce with the same definitions, rather than an unexplained number attached to a large pile of logs.

Sources

OpenTelemetry logs, event records and correlation with other telemetry, checked 2026-09-13.
SQLite aggregate functions, count, sum, and empty-input behavior used by the query, checked 2026-09-13.

Log rotation needs writer handoff and a retention budget

Fri, 10 Jul 2026 00:00:00 +0000

Log rotation divides a continuing stream into files that can be retained, searched, compressed, and eventually removed. A useful policy defines both when to rotate and which history to keep. It must also account for how the application holds its output file open. Renaming a pathname and successfully redirecting future log records are separate steps in that operational contract.

When it breaks

A writer usually keeps an open descriptor rather than resolving the log pathname for every record. Renaming the file and creating a new file at the old name does not move that descriptor. The active pathname can look empty while the archive keeps growing. Monitoring only the new name may falsely suggest that the application stopped logging.

Removing an archive name also does not close a writer that still references the inode. The file can remain open, consume resources, and accept further writes without a normal directory entry. A cleanup job that reports successful deletion therefore has not necessarily recovered the expected space or redirected the application’s logging.

Copy-and-truncate avoids requiring the writer to reopen, but copying and truncating are separate operations. A record written after the copy and before truncation can disappear from both the archive and active file. A quiet period can reduce the chance of that interleaving; it does not turn the method into an atomic, lossless handoff.

Check it quickly

Save this as rotation.py and run python3 rotation.py in a disposable Linux environment. The recorded run used Linux 6.12.76, Python 3.11.2, and ext4. The fixture uses only temporary files and explicitly controls the order of writes. Its copytruncate case is a deterministic interleaving of the underlying operations, not a claim that a race happened accidentally:

from pathlib import Path
from tempfile import TemporaryDirectory
import os,shutil

with TemporaryDirectory(prefix='xbcatm-rotation-') as directory:
 root=Path(directory);active=root/'app.log';archive=root/'app.log.1'
 with active.open('a+b',buffering=0) as writer:
 writer.write(b'before\n');identity=os.fstat(writer.fileno()).st_ino
 active.rename(archive);active.touch();writer.write(b'after-rename\n')
 assert active.read_bytes()==b'' and archive.read_bytes()==b'before\nafter-rename\n'
 assert archive.stat().st_ino==identity and active.stat().st_ino!=identity
 print('renaming and creating a replacement path leaves an existing writer appending to the archive inode')
 archive.unlink();assert os.fstat(writer.fileno()).st_nlink==0
 writer.write(b'after-unlink\n');writer.seek(0)
 assert b'after-unlink\n' in writer.read() and not archive.exists()
 print('deleting the archive name does not close the writer; its unlinked file remains open and writable')
 with active.open('ab',buffering=0) as reopened:reopened.write(b'new-active\n')
 assert active.read_bytes()==b'new-active\n'
 print('closing and reopening the active path switches subsequent writes to the replacement inode')
 source=root/'copytruncate.log';copied=root/'copytruncate.log.1'
 with source.open('a+b',buffering=0) as writer:
 writer.write(b'before-copy\n');shutil.copyfile(source,copied)
 writer.write(b'between-copy-and-truncate\n')
 os.ftruncate(writer.fileno(),0);writer.write(b'after-truncate\n')
 assert copied.read_bytes()==b'before-copy\n' and source.read_bytes()==b'after-truncate\n'
 assert b'between-copy-and-truncate' not in copied.read_bytes()+source.read_bytes()
 print('deterministic copy-then-truncate interleaving loses the write placed in that gap')

The first writer appends before, then the controller renames the log and creates a replacement active pathname. The same open writer appends after-rename. Both records appear in the archive, and the replacement remains empty. Inode checks confirm that the writer retained the original object while the active pathname now identifies a different file.

Next, the archive is unlinked while that writer remains open. Its link count becomes zero, and another appended record is still readable through the retained reference. The archive pathname no longer exists. These observations establish why pathname deletion and the end of a writer’s file lifetime must be investigated separately during a disk-space incident.

Closing the old writer and reopening the active pathname makes a subsequent record land in the replacement file. This is the essential handoff a service’s documented reopen mechanism must perform. Sending a conventionally named signal is useful only if that application actually implements the required behavior and its result is observed.

The second case copies a log containing before-copy, writes another record into the original, then truncates the original and appends after-truncate. The archive contains only the first record; the active file contains only the last. The middle record is absent from both. This demonstrates the precise loss window described for copytruncate.

The script closes all references and removes its temporary files. It checks runtime visibility and data placement, not crash durability or a remote backup system. Those are additional properties that cannot be inferred from a successful rename, a completed compression, or an archive directory containing the expected number of names.

Budget time, growth, and retrieval

A size threshold is evaluated when the rotation tool runs; it is not a continuously enforced maximum file size. For example, a ten-MiB threshold checked every sixty seconds can permit roughly seventy MiB before the next check if writing is bounded at one MiB per second. Delayed checks or an unbounded producer invalidate that illustrative growth bound.

Account for the active file, retained generations, temporary compression or copying space, and still-open old files. Keeping four generations does not automatically preserve four days: frequent rotations can consume those generations much sooner. Choose both a capacity budget and a history window that supports the incidents the team actually needs to investigate.

Verify retrieval as well as deletion. Confirm that representative records can be found across rotated and compressed files and that the oldest required interval remains available. If history must survive host or disk loss, maintain an independently stored copy and test recovery. Local rotation organizes and limits local history; its success alone does not prove that the history remains recoverable after the machine is gone.

Sources

rename(2), name replacement without redirecting existing open descriptions, checked 2026-09-12.
unlink(2), unlinked files retained by open references, checked 2026-09-12.
logrotate 3.21.0 manual, retention directives, size evaluation, and copytruncate loss window, checked 2026-09-12.

Logging · XB Field Notes

The systemd journal needs an explicit retention contract

When it breaks

Check it quickly

Size the evidence window

Further reading

Sources

Query the journal with explicit field and priority rules

When it breaks

Check it quickly

Preserve context when exporting results

Further reading

Sources

Structured nginx logs need escaping and a value-type contract

When it breaks

Check it quickly

Define what collectors should accept

Sources

Structured logs make their query contract explicit

Put context on the correct record

When it breaks

Check it quickly

Further reading

Sources

An access log separates client outcomes from upstream attempts

When it breaks

Check it quickly

Keep diagnostic meaning in the schema

Sources

Timestamps need explicit time semantics

An offset is not a timezone rulebook

When it breaks

Check it quickly

Sources

Logrotate applies a rotation policy when it runs

When it breaks

Check it quickly

Operate the policy as a complete workflow

Sources

Metrics and logs need an explicit population

Decide what the measurement includes

When it breaks

Check it quickly

Further reading

Sources

Give a log query a question and a denominator

Make the query contract visible

When it breaks

Check it quickly

Further reading

Sources

Log rotation needs writer handoff and a retention budget

When it breaks

Check it quickly

Budget time, growth, and retrieval

Further reading

Sources