Observability · XB Field Notes

Baggage carries context across an explicit boundary

Mon, 27 Mar 2028 00:00:00 +0000

OpenTelemetry baggage is a context-associated collection of string key-value pairs. It lets information known near a request’s entry point travel to downstream operations: a deployment region or a rollout cohort, for example. Unlike a span attribute, baggage is intended for propagation. Unlike trace context, it describes application-selected information rather than the identifiers and flags connecting spans. These mechanisms can travel together without being the same data structure.

Propagation still needs a working boundary. An outgoing client injects context into a carrier such as HTTP headers; the receiver extracts that carrier into its local context. Instrumentation can perform these steps automatically when configured, while manually instrumented code must do them explicitly. Merely creating baggage in one process does not deliver it to an unrelated process, a background task, or an external service.

Choose what crosses and what gets recorded

Baggage entries do not automatically become span, log, or metric attributes. A downstream operation must read the value and copy it deliberately, or use a configured processor implementing that policy. This distinction matters when a team expects a searchable attribute but has only enabled propagation. It also prevents the opposite mistake: copying every incoming key into telemetry without considering its meaning, size, or cardinality.

Context operations return a new context rather than changing every existing reference. Keep the returned context and pass or attach it at the intended boundary. When removing entries for an outgoing request, preserve the trace context that should accompany that request. Treat an outbound allowlist and a telemetry-attribute allowlist as explicit decisions; the values appropriate for one receiver need not be suitable for every later destination.

When it breaks

Baggage has no built-in proof that a value was issued by a trusted component. A caller can supply a header claiming an expensive account tier or a privileged role. Parsing that header successfully establishes its representation, not its authority. Authentication and authorization must use their own trusted inputs. Small, plausible-looking values deserve the same trust analysis as obviously suspicious ones.

Unbounded baggage also increases request-header sizes and can spread sensitive data to unintended recipients, including third-party APIs reached by automatic instrumentation. Limit names and values, review the destinations that receive them, and use propagators for wire encoding. A successful round trip between two copies of one SDK is narrower evidence than interoperability across every language and proxy in a production path. Test those actual boundaries when they matter.

Check it quickly

Create a virtual environment, install opentelemetry-sdk==1.35.0, save the script as baggage.py, and execute it with that environment’s Python. This exercise ran on Python 3.14.4. It uses the real W3C trace-context and baggage propagators with an in-memory span exporter. It does not start an HTTP server: the dictionary is the carrier that an HTTP client would subsequently send.

import json

from opentelemetry import baggage, trace
from opentelemetry.context import Context
from opentelemetry.baggage.propagation import W3CBaggagePropagator
from opentelemetry.propagators.composite import CompositePropagator
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

base = Context()
raw = baggage.set_baggage("region", "eu-west", context=base)
raw = baggage.set_baggage("release", "canary/2", context=raw)
raw = baggage.set_baggage("internal_note", "synthetic-only", context=raw)
assert not baggage.get_all(base)
propagator = CompositePropagator([TraceContextTextMapPropagator(), W3CBaggagePropagator()])
provider = TracerProvider()
exporter = InMemorySpanExporter()
provider.add_span_processor(SimpleSpanProcessor(exporter))
tracer = provider.get_tracer("baggage-lab")
with tracer.start_as_current_span("parent", context=Context()) as parent:
 outgoing = trace.set_span_in_context(parent, baggage.clear(context=raw))
 for key in ("region", "release"):
 outgoing = baggage.set_baggage(key, baggage.get_baggage(key, raw), context=outgoing)
 carrier = {}
 propagator.inject(carrier, context=outgoing)
 received = propagator.extract(carrier, context=Context())
 assert set(baggage.get_all(received)) == {"region", "release"}
 assert baggage.get_baggage("region", received) == "eu-west"
 assert baggage.get_baggage("internal_note", raw) == "synthetic-only"
 assert trace.get_current_span(received).get_span_context().trace_id == parent.get_span_context().trace_id
 with tracer.start_as_current_span("inherited", context=received):
 pass
 with tracer.start_as_current_span("copied", context=received) as span:
 span.set_attribute("deployment.region", baggage.get_baggage("region", received))
assert provider.force_flush()
spans = {span.name: span for span in exporter.get_finished_spans()}
assert not spans["inherited"].attributes
assert spans["copied"].attributes == {"deployment.region": "eu-west"}
independent = propagator.extract({"traceparent": "invalid", "baggage": "region=eu-west"},
 context=Context())
assert not trace.get_current_span(independent).get_span_context().is_valid
assert baggage.get_baggage("region", independent) == "eu-west"
untrusted = propagator.extract({"baggage": "tier=gold"}, context=Context())
assert baggage.get_baggage("tier", untrusted) == "gold"
print(json.dumps({"wire_fields": sorted(carrier),
 "wire_baggage": carrier["baggage"],
 "automatic_span_attributes": dict(spans["inherited"].attributes),
 "explicit_span_attributes": dict(spans["copied"].attributes),
 "baggage_survives_invalid_traceparent": True,
 "caller_supplied_tier_was_parsed": True}, indent=2))
provider.shutdown()

The original context contains region, release, and a synthetic internal note. The outgoing context keeps only the first two entries while retaining the parent span. The extracted region is eu-west, and the release value canary/2 round-trips through the encoded header. Assertions also confirm that constructing the filtered context did not remove the note from the original context or change the empty base context.

Two child spans demonstrate the recording boundary. inherited receives baggage but has no attributes; copied contains only the explicitly assigned deployment.region. A malformed traceparent is tested separately: its trace context is invalid, while the independently encoded baggage still extracts successfully. Finally, a caller-supplied tier=gold is accepted as data without any identity check. These results establish serialization, filtering, and explicit recording behavior. Before relying on the same policy across services, verify the real client instrumentation, receiver extraction, outbound filtering, and the attribute mapping used by the destination backend.

Sources

OpenTelemetry baggage, baggage is separate from telemetry attributes and lacks built-in integrity checks, checked 2026-09-13.
W3C Baggage, HTTP representation, value encoding, and propagation limits, checked 2026-09-13.
W3C Trace Context, trace identity travels separately from application baggage, checked 2026-09-13.

Annotations explain an alert without changing its identity

Tue, 22 Feb 2028 00:00:00 +0000

Alert labels identify which condition is being tracked and provide fields for routing and grouping. Annotations explain that condition: a current value, a concise summary, or a runbook link. Putting changing prose in labels can split one ongoing problem into many identities. Putting routing fields only in annotations can leave a route unable to recognize the alert’s intended owner.

Separate identity from the current explanation

A queue alert might keep alertname, service, and instance as stable labels while its summary changes from six waiting items to eight. Alertmanager can update the annotation for the same alert fingerprint rather than adding another alert. A fingerprint is derived from identity labels; it is not a hash of the rendered notification text.

This distinction also matters before Alertmanager receives the alert. Prometheus tracks its pending duration for an alert label set. A label containing a changing current value can keep creating new identities and prevent the intended waiting period from completing. Put that changing value in an annotation unless it deliberately defines a different alert condition. Review templates and rule labels together when diagnosing repeated pending alerts.

Notification templates receive several scopes of data. GroupLabels contains the labels chosen for grouping. CommonLabels contains label pairs common to all alerts in the notification. CommonAnnotations similarly contains annotations shared by the group. The Alerts list retains the labels and annotations of individual members, allowing a notification to explain their different states.

If two replicas have summaries describing different queue sizes, there is no shared summary to put in CommonAnnotations. A template that prints only that field can omit the very information an operator needs. Use common context for the heading, then iterate over individual alerts for changing summaries. A shared runbook link can remain common even when the individual values differ.

Do not assume a grouping label such as instance exists in every group. Grouping by service and alert name intentionally combines instances. A template should reflect the selected grouping contract and handle an absent common field deliberately. Representative fixtures need multiple alerts, unequal annotations, and the relevant firing or resolved states, not just one idealized alert where every field happens to be common.

When it breaks

A dynamic label creates a new fingerprint for every value change, increasing alert churn and potentially restarting pending state. A route matching team cannot use a team annotation in place of a label. An overly broad notification heading can also describe one instance while listing alerts from several services if grouping and template assumptions drift apart.

A template that parses successfully can still render an empty or misleading message for actual group data. Check the rendered text, not only syntax. Conversely, rendering successfully does not prove the integration accepts the message or that a notification was delivered. Keep credentials and sensitive request details out of annotations because they can be copied to every receiver selected by routing.

Check it quickly

Save this as annotations.py and run python3 annotations.py with Docker available. It uses Alertmanager 0.28.1, downloading the image if needed. The isolated server has a receiver with no delivery integrations. Two synthetic alerts are submitted through its actual API; one annotation is then updated without changing any labels.

import http.client
import io
import json
import subprocess
import time
import uuid

name = "xbcatm-monitoring-" + uuid.uuid4().hex[:12]


class ResponseSocket:
 def __init__(self, data):
 self.data = data

 def makefile(self, mode):
 return io.BytesIO(self.data)


def request(method, path, body="", content_type="application/json"):
 body = body.encode()
 headers = (f"{method} {path} HTTP/1.1\r\nHost: 127.0.0.1:{port}\r\n"
 f"Connection: close\r\nContent-Type: {content_type}\r\n"
 f"Content-Length: {len(body)}\r\n\r\n").encode()
 result = subprocess.run(
 ["docker", "exec", "-i", name, "/bin/nc", "-w", "3", "127.0.0.1", str(port)],
 input=headers + body, capture_output=True, timeout=5,
 )
 if result.returncode:
 raise RuntimeError(result.stderr.decode())
 response = http.client.HTTPResponse(ResponseSocket(result.stdout))
 response.begin()
 return response.status, response.read().decode()


def get_json(path):
 status, body = request("GET", path)
 assert status == 200, (status, body)
 return json.loads(body)


def eventually(check):
 for attempt in range(150):
 try:
 value = check()
 if value:
 return value
 except (RuntimeError, AssertionError, http.client.HTTPException):
 pass
 time.sleep(0.1)
 raise AssertionError("Expected local server state did not appear")


port = 9093
script = r"""set -eu
cat > /tmp/alertmanager.yml <<'ALERT_CONFIG'
route:
 receiver: preview
 group_by: [alertname, service]
 group_wait: 1h
 group_interval: 5m
 repeat_interval: 4h
receivers:
 - name: preview
ALERT_CONFIG
exec /bin/alertmanager --config.file=/tmp/alertmanager.yml --storage.path=/tmp/data \
 --web.listen-address=127.0.0.1:9093 --cluster.listen-address=
"""
try:
 subprocess.run(
 ["docker", "run", "-d", "--rm", "--name", name, "--network", "none",
 "--entrypoint", "/bin/sh", "prom/alertmanager:v0.28.1", "-c", script],
 check=True, capture_output=True, text=True, timeout=20,
 )
 eventually(lambda: request("GET", "/-/ready")[0] == 200)
 alerts = [{"labels": {"alertname": "QueueHigh", "service": "checkout", "instance": instance},
 "annotations": {"summary": "queue=" + str(depth), "runbook": "https://example.invalid/queue"}}
 for instance, depth in [("api-a", 6), ("api-b", 9)]]
 assert request("POST", "/api/v2/alerts", json.dumps(alerts))[0] == 200
 before = eventually(lambda: (a if len(a := get_json("/api/v2/alerts")) == 2 else False))
 fingerprints = {a["labels"]["instance"]: a["fingerprint"] for a in before}
 alerts[0]["annotations"]["summary"] = "queue=8"
 assert request("POST", "/api/v2/alerts", json.dumps([alerts[0]]))[0] == 200
 def updated_annotations():
 items = get_json("/api/v2/alerts")
 changed = any(x["labels"]["instance"] == "api-a" and
 x["annotations"]["summary"] == "queue=8" for x in items)
 return items if changed else None
 updated = eventually(updated_annotations)
 assert len(updated) == 2
 assert {a["labels"]["instance"]: a["fingerprint"] for a in updated} == fingerprints
 # Supply the documented notification Data shape to the real template renderer.
 data = {"receiver": "preview", "status": "firing",
 "groupLabels": {"alertname": "QueueHigh", "service": "checkout"},
 "commonLabels": {"alertname": "QueueHigh", "service": "checkout"},
 "commonAnnotations": {"runbook": "https://example.invalid/queue"},
 "alerts": [{"status": "firing", "labels": a["labels"], "annotations": a["annotations"]}
 for a in sorted(updated, key=lambda a: a["labels"]["instance"])]}
 template = '{{define "preview"}}{{.GroupLabels.service}}|common-summary={{with .CommonAnnotations.summary}}{{.}}{{else}}none{{end}}{{range .Alerts}}\n{{.Labels.instance}}:{{.Annotations.summary}}{{end}}{{end}}'
 for path, content in [("/tmp/preview.tmpl", template), ("/tmp/data.json", json.dumps(data))]:
 subprocess.run(["docker", "exec", "-i", name, "/bin/sh", "-c", "cat > " + path],
 input=content, text=True, capture_output=True, check=True, timeout=5)
 rendered = subprocess.run(["docker", "exec", name, "/bin/amtool", "template", "render",
 "--template.glob=/tmp/preview.tmpl", "--template.data=/tmp/data.json",
 '--template.text={{ template "preview" . }}'],
 text=True, capture_output=True, check=True, timeout=5).stdout.strip()
 assert rendered == "checkout|common-summary=none\napi-a:queue=8\napi-b:queue=9", rendered
 print(json.dumps({"alert_count": 2, "fingerprints_unchanged": True, "rendered": rendered}, indent=2))
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

The result confirms that two alerts remain and their fingerprints are unchanged after the update. The program then supplies the documented notification data structure to the real amtool template renderer. Its common annotations intentionally include the shared runbook but exclude the unequal summaries. This explicit template input tests rendering, not the server’s notification assembly or delivery pipeline.

The rendered heading says common-summary=none, followed by api-a:queue=8 and api-b:queue=9. Assertions verify the updated value and the separate per-alert details. All HTTP traffic remains on container loopback, with no host mounts or published ports, and the container is removed on exit. A production notification still needs validation using its actual route, template data, and receiver integration.

Sources

Alertmanager: notification template reference, group labels, common annotations and individual alert data, checked 2026-09-13.
Prometheus: alerting rules, labels identify alerts while annotations carry explanatory content, checked 2026-09-13.
Alertmanager 0.28.1: README, local template rendering with amtool, checked 2026-09-13.

pg_stat_statements finds the queries that cost the most

Sun, 12 Dec 2027 00:00:00 +0000

Because pg_stat_statements aggregates every execution of one query shape, it turns a vague “the page is slow” into a ranked table. The setup is a config line and a restart; the payoff is knowing which of your query shapes actually owns the clock.

The one decision that matters is which column you sort by. Sorting by mean time reproduces what the slow log already told you; sorting by total time is what finds the query that has been quietly consuming a quarter of your database.

Normalize, then aggregate

The extension does not store your SQL verbatim. Constants are folded into parameter symbols, so WHERE tenant_id = 42 and WHERE tenant_id = 8712 collapse into one entry, displayed as the text of the first query that produced that queryid. The numbering rule is positional — symbols start at the next number after the highest $n already present, which is why a PL/pgSQL statement like SELECT i + 1 INTO j shows up as SELECT i + $2. Lists behave the same way: two queries differing only in the number of IN elements get squashed into IN ($1 /*, ... */) with a call count of two.

The effect is dramatic: a modelled workload of 1,420,052 executions, each with different literals, collapsed into 6 rows — 236,675 raw statements per row. Two caveats come with it: queryid is a hash of the post-parse-analysis tree, so identical text can land in different entries when search_path differs, unrelated queries can collide into one entry, and the value is not promised to be stable across major versions. Treat queryid as a better identifier than the text, not as a permanent key.

Total time, not mean time

The columns that matter are calls, total_exec_time, mean_exec_time, rows, and the block counters. min/max/stddev_exec_time tell you whether a query is uniformly slow or occasionally long, which points at bad plans versus lock contention respectively.

Ranking by mean answers a different question from ranking by total, and the gap is large. In the same workload the query with the worst mean (9,689 ms, 12 calls) is 22.4% of measured time; the query with the worst total is 25.9% but averages 0.749 ms per call. The worst-by-total query never appears in a slow log at any sane log_min_duration_statement setting, and the three worst by total together are 65.5% of all measured time. A 0.099 ms lookup called 900,000 times ranks sixth by mean and third by total, at 89,217 ms.

Counters accumulate, so compare windows

Counters run from the moment statistics started, so a query that used to be slow never fades out of the top of the table on its own. The fix is either a scoped reset — pg_stat_statements_reset(userid, dbid, queryid, minmax_only) takes all four, so you can clear one query, one database, or just the min/max fields — or a delta between two snapshots.

The block counters deserve their own reading. shared_blks_hit and shared_blks_read are always tracked; track_io_timing adds the *_blk_read_time and *_blk_write_time columns, not the counts. Hit ratio is a diagnostic, not a cost: the query with the best hit ratio in the workload (98.89%) was also the largest block consumer at 81,000,000 blocks, while the query with the worst ratio (62.22%) moved 37x fewer.

When it breaks

The first failure is that the extension is simply not there. It must be listed in shared_preload_libraries because it needs shared memory, which means adding or removing it requires a restart, compute_query_id must be auto or on, and the views only exist in databases where CREATE EXTENSION pg_stat_statements was run. Miss any of those and every query against it fails, so the monitoring you thought you had has been empty for months.

The second failure is a reset that destroys the comparison. Reset the counters mid-window and every delta comes out negative: in the fixture, 6 of 6 shapes reported negative time. The arithmetic is fine; the window is not. pg_stat_statements_info.stats_reset exists precisely for this — check it before trusting any subtraction, and if the reset was intentional, start the window after it rather than trying to reconcile across it.

Check it quickly

Run both rankings and see how little they agree:

psql -c "SELECT calls, round(total_exec_time) AS total_ms, round(mean_exec_time,3) AS mean_ms, rows FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 10;"
psql -c "SELECT calls, round(mean_exec_time,2) AS mean_ms, round(total_exec_time) AS total_ms FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;"
psql -c "SELECT stats_reset FROM pg_stat_statements_info;"
psql -c "SELECT sum(calls) AS calls, round(sum(total_exec_time)/1000) AS total_s, round(sum(shared_blks_read)*8.0/1024/1024,1) AS read_gib FROM pg_stat_statements;"
psql -c "SELECT pg_stat_statements_reset(0, 0, queryid) FROM pg_stat_statements WHERE query LIKE 'SELECT%';"

The first two lists should disagree, and the first is the one to work from. stats_reset is the timestamp that decides whether a delta window is valid. The totals line is the sanity check on magnitude — if total_s is larger than the wall-clock age of your statistics, you are reading across a restart. The last command resets selectively; run it deliberately, because it is exactly what makes the next comparison wrong.

Sources

PostgreSQL: pg_stat_statements, normalization, the statistics columns, and pg_stat_statements_reset, checked 2026-09-16.
PostgreSQL: cumulative statistics system, where stats_reset and the counters live, checked 2026-09-16.
PostgreSQL: statistics monitoring configuration, track_io_timing and the block-time columns it enables, checked 2026-09-16.

Read procfs as documented views of a changing process

Fri, 29 Oct 2027 00:00:00 +0000

Linux procfs exposes process and kernel information through filesystem interfaces. Many entries generate their contents when read instead of storing ordinary disk-file bytes. Each entry has a documented meaning, permission model, and format. Reading several entries does not freeze the process or create a transactionally consistent snapshot, so useful observations must be tied to the right process instance and interpretation.

When it breaks

Ordinary file assumptions can fail immediately. A proc entry can report size zero yet return useful content when read. A per-process directory can disappear as the process exits, and a numerical PID can eventually identify another instance. Treating a failed read as proof of a broken installation, or assuming a PID forever names one workload, can misdiagnose normal lifecycle changes.

Formats require entry-specific parsing. status contains labeled fields, while stat uses a positional layout with a parenthesized command name that can contain spaces and closing parentheses. Splitting the whole stat line on whitespace can shift the expected fields. cmdline and environ use NUL separators, so ordinary line or space splitting loses argument and value boundaries.

The views also have semantic limits. environ describes the environment region established at program execution, not a guaranteed live dictionary of every later environment change. Command-line contents can be changed by the process. A descriptor link ending in (deleted) does not mean the open object has vanished. These entries are evidence about specific state, not interchangeable proofs of identity or access.

Check it quickly

Save this as procview.py and run python3 procview.py in a disposable Linux environment. The recorded run used Linux 6.12.76 and Python 3.11.2. The script creates one controlled child with a small, explicitly supplied environment, reads only its own and that child’s proc entries, and performs no writes to kernel tuning interfaces:

from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,subprocess,sys

worker=r'''
from pathlib import Path
import ctypes,json,os,sys
assert ctypes.CDLL(None).prctl(15,b'worker ) sample',0,0,0)==0
os.environ['XB_PROC_TEST']='changed-after-start'
path=Path(sys.argv[1]);path.write_bytes(b'fixture-data')
with path.open('rb') as handle:
 info=os.fstat(handle.fileno());path.unlink()
 print(json.dumps({'fd':handle.fileno(),'device':info.st_dev,'inode':info.st_ino,'current_environment':os.environ['XB_PROC_TEST']}),flush=True)
 assert sys.stdin.readline()=='done\n'
'''
with TemporaryDirectory(prefix='xbcatm-proc-') as directory:
 path=Path(directory)/'data'
 process=subprocess.Popen([sys.executable,'-c',worker,str(path),'argument with spaces'],stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True,env={'PATH':'/usr/bin:/bin','XB_PROC_TEST':'fixture-value'})
 try:
 metadata=json.loads(process.stdout.readline());root=Path('/proc')/str(process.pid)
 status_path=root/'status';status_text=status_path.read_text()
 assert status_path.stat().st_size==0 and len(status_text)>0
 status=dict(line.split(':',1) for line in status_text.splitlines() if ':' in line)
 assert int(status['Pid'])==process.pid and int(status['PPid'])==os.getpid()
 raw=(root/'stat').read_text();pid_text,rest=raw.split(' (',1);comm,fields=rest.rsplit(') ',1)
 fields=fields.split();assert int(pid_text)==process.pid and comm=='worker ) sample'
 assert int(fields[1])==os.getpid() and int(fields[19])>0
 assert raw.split()[2] not in ('R','S','D','T','Z')
 print('proc files can have size zero but readable data; stat comm can contain spaces and a closing parenthesis')
 args=(root/'cmdline').read_bytes().rstrip(b'\0').split(b'\0')
 environment=(root/'environ').read_bytes().rstrip(b'\0').split(b'\0')
 assert args[-1]==b'argument with spaces' and b'XB_PROC_TEST=fixture-value' in environment
 assert metadata['current_environment']=='changed-after-start'
 assert os.readlink('/proc/self')==str(os.getpid())
 print('cmdline and environ use NUL separators; environ retains the initial value here; self resolves for the reader')
 entry=root/'fd'/str(metadata['fd']);target=os.readlink(entry);actual=entry.stat()
 assert target==str(path)+' (deleted)' and not path.exists()
 assert (actual.st_dev,actual.st_ino)==(metadata['device'],metadata['inode'])
 assert entry.read_bytes()==b'fixture-data'
 print('a deleted pathname remains inspectable through an open descriptor subject to access checks')
 output,error=process.communicate('done\n',timeout=5)
 assert process.returncode==0,error
 assert not root.exists()
 print('after the controlled child exits and is reaped, its proc directory disappears')
 finally:
 if process.poll() is None:process.kill();process.wait()

The child sets its short process name to worker ) sample, opens a temporary file, and unlinks its pathname while keeping the descriptor open. It then reports only fixture metadata and waits. This gives the controller a known live process whose name deliberately exercises the stat-format edge case.

The controller finds that status has reported size zero but nonempty readable contents. It checks labeled PID and parent PID fields. For stat, it separates the PID and parenthesized name before interpreting the remaining positional fields, confirming the parent PID and a positive start-time field. Naive whitespace splitting would mistake part of the name for a later field.

The final command-line argument contains spaces and remains one argument after NUL-based parsing. The initial environment contains XB_PROC_TEST=fixture-value, even though the child has changed its application environment value after startup. Both observations use harmless fixture data. They demonstrate why whitespace parsing and assuming a live environment map can produce incorrect conclusions.

Reading /proc/self in the controller resolves to the controller’s PID. Reading the child’s descriptor link returns the removed pathname with the deleted suffix, but following that proc descriptor entry still exposes the expected device/inode pair and fixture bytes under the experiment’s permissions. The missing ordinary pathname and the surviving open object are distinct facts.

After receiving the controller’s message, the child closes its handle and exits. The controller waits for completion and confirms that this child’s proc directory disappears. The example establishes a controlled before-and-after lifecycle observation; it does not promise atomic consistency among arbitrary proc reads while a busy application changes its state.

Match the view to the question

Use the relevant entry’s documented parser and units. A stat field number should be interpreted only after handling the command-name field correctly; a status label should not be assigned an assumed line number. If correlating samples, include process start time and boot context as appropriate so a reused numeric PID is not silently treated as the same instance.

Access and namespace context matter. Proc mount options, credentials, and ptrace-related checks can restrict visibility. /proc/net reflects the relevant network namespace, and a container’s visible proc data is not automatically its configured resource budget. Confirm which namespace and resource interface answer the operational question before labeling an observation as host-wide or container-specific.

Some proc entries are writable controls, but their scope differs. /proc/sys exposes tunables, while /proc/PID/oom_score_adj adjusts a particular process’s OOM preference; writing integers to status is not the interface for that task. Keep observation separate from a deliberate configuration change, and inspect the actual target and documented permissions before applying any write.

Sources

proc(5), procfs structure, per-process visibility, and mount options, checked 2026-09-12.
proc_pid_stat(5), field layout, parenthesized command name, and start time, checked 2026-09-12.
proc_pid_environ(5), initial environment region and NUL-separated entries, checked 2026-09-12.
proc_pid_fd(5), descriptor links, referenced objects, and permission checks, checked 2026-09-12.
proc_pid_cmdline(5), argument separators and process-controlled command-line contents, checked 2026-09-12.

Trace sampling reduces volume without fixing a cost ceiling

Sat, 23 Oct 2027 00:00:00 +0000

Trace sampling selects which recorded work reaches the tracing pipeline. It can reduce export traffic and storage, but a configured percentage is not a strict ceiling on bytes, processor time, or money. Requests produce different numbers of spans, attributes vary in size, and traffic itself changes. A one-percent policy applied to a sudden tenfold traffic increase can still produce roughly ten times as much sampled data.

Head sampling makes a decision near the start of a trace. A trace-identifier ratio sampler is useful when identifiers have the randomness its algorithm requires. Its probability describes selection across suitable traffic; it does not promise exactly half of every sixteen requests, or equal coverage of every route. Small populations and rare failures can be poorly represented even when the overall percentage looks reasonable.

Keep a decision consistent across boundaries

Downstream instrumentation should follow the accepted parent decision when that is the intended policy. In the Python SDK, ParentBased delegates root decisions to its configured root sampler and uses separate policies for sampled and unsampled parents. Wrapping TraceIdRatioBased(0.5) therefore differs from treating every downstream span as an independent new coin flip. Context propagation and the receiving sampler configuration must agree.

An unsampled trace still involves application work, context handling, and potentially instrumentation overhead. Dropping export does not make the business request free. Conversely, preserving a parent decision does not guarantee that every child will be visible: a library might be uninstrumented, a service might lose context, or an exporter might fail. Sampling consistency addresses one source of missing data, not the whole delivery path.

When it breaks

A request can look ordinary at its start and fail much later. A head sampler that already dropped it cannot recover the missing spans merely because application code subsequently sets an error status. Tail sampling can inspect more of the completed trace before deciding, but requires buffering, routing related spans together, and managing delayed or missing data. It also cannot reconstruct spans discarded upstream before they reached it.

Another failure is measuring request reliability directly from a biased trace collection. Policies that deliberately retain slow requests or errors change the population being observed. Keep request outcome metrics with a clear denominator, and describe the trace selection policy alongside investigations. For cost control, observe actual exported spans and bytes, queue behavior, collector resources, and retention; a sampling percentage alone is not evidence that a spending limit holds.

Check it quickly

In a Python virtual environment, install opentelemetry-sdk==1.35.0. Save this as trace-sampling.py and run it using that environment; Python 3.14.4 was used here. The custom identifier generator is exclusively a deterministic test fixture. Its sixteen identifiers deliberately straddle this SDK version’s ratio threshold. Do not copy that sequential identifier scheme into production instrumentation, where the sampler relies on appropriate randomness.

import json
from collections import Counter

from opentelemetry.context import Context
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.sdk.trace.id_generator import IdGenerator
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased
from opentelemetry.trace import Status, StatusCode
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator


class TestIds(IdGenerator):
 # Controlled test inputs only; production IDs need suitable randomness.
 def __init__(self):
 self.trace_index = 0
 self.span_index = 0

 def generate_trace_id(self):
 index = self.trace_index
 self.trace_index += 1
 return ((index + 1) << 64) | (index * (1 << 60) + 1)

 def generate_span_id(self):
 self.span_index += 1
 return self.span_index


exporter = InMemorySpanExporter()
provider = TracerProvider(sampler=ParentBased(TraceIdRatioBased(0.5)),
 id_generator=TestIds())
provider.add_span_processor(SimpleSpanProcessor(exporter))
tracer = provider.get_tracer("sampling-lab")
executed_children = 0
late_error_recording = None
for index in range(16):
 with tracer.start_as_current_span(f"root-{index}", context=Context()) as root:
 for child in range(2):
 with tracer.start_as_current_span(f"child-{index}-{child}"):
 executed_children += 1
 if index == 15:
 late_error_recording = root.is_recording()
 root.set_status(Status(StatusCode.ERROR, "error discovered after start"))
assert provider.force_flush()
spans = exporter.get_finished_spans()
counts = Counter(span.context.trace_id for span in spans)
assert len(counts) == 8 and set(counts.values()) == {3} and len(spans) == 24
assert {s.name for s in spans if s.parent is None} == {f"root-{i}" for i in range(8)}
assert executed_children == 32 and late_error_recording is False
assert not any(span.name == "root-15" for span in spans)

exporter.clear()
propagator = TraceContextTextMapPropagator()
for flag, label in (("01", "sampled"), ("00", "unsampled")):
 # High trace-ID bits do not override an accepted remote parent's flag.
 header = f"00-1111111111111111ffffffffffffffff-2222222222222222-{flag}"
 context = propagator.extract({"traceparent": header}, context=Context())
 with tracer.start_as_current_span(label, context=context):
 with tracer.start_as_current_span(label + "-child"):
 pass
assert provider.force_flush()
remote_spans = exporter.get_finished_spans()
assert {span.name for span in remote_spans} == {"sampled", "sampled-child"}
assert next(span for span in remote_spans if span.name == "sampled").parent.is_remote
print(json.dumps({"controlled_roots": 16, "exported_traces": len(counts),
 "exported_spans": len(spans), "executed_child_operations": executed_children,
 "late_error_was_recording": late_error_recording,
 "remote_parent_exported_spans": len(remote_spans)}, indent=2))
provider.shutdown()

The sixteen root operations each start two children. Exactly eight traces, containing twenty-four spans, are exported because of the chosen test inputs; all thirty-two child operations still execute. Root fifteen is deliberately outside the retained set. Its later error status does not cause it to become recording or appear in the exporter. These assertions test behavior at known inputs rather than estimating a sampling percentage statistically.

The second phase extracts two remote parent headers with the same high trace identifier and different sampled flags. Only the sampled parent’s child tree is exported, producing two spans despite that identifier being above the root ratio threshold. The unsampled parent produces none. No collector or tail sampler runs in this experiment. When applying the policy to a real service, test accepted remote context separately from new roots, then validate the resulting volume and diagnostic coverage against the workload that actually matters.

Sources

OpenTelemetry sampling, head and tail decisions and the risk of losing rare failures, checked 2026-09-13.
OpenTelemetry Python sampling reference, ParentBased and TraceIdRatioBased sampler behavior, checked 2026-09-13.
W3C Trace Context, propagated sampling flags and remote parent context, checked 2026-09-13.

Prometheus pulls metrics and records scrape outcomes

Mon, 30 Aug 2027 00:00:00 +0000

Prometheus collects metrics by requesting an HTTP endpoint on each configured target. The target exposes observations; the collector chooses when to scrape, parses the response, and stores samples. This arrangement makes collection failures observable from the collector’s perspective. It does not guarantee that the endpoint is reachable, that every expected target was discovered, or that the application’s user journeys work.

A job contains independently scraped instances

A job groups targets with a shared purpose, such as several API replicas. Each target is an instance with its own scrape outcome. The usual job and instance labels identify that context, although relabeling can change the final labels. Service discovery supplies candidates; scrape configuration determines their endpoint, protocol, interval, timeout, and authentication. Discovery and collection therefore have separate failure modes.

The collector must be able to connect to the target. An exporter needs an inbound path from authorized collectors, even though it does not initiate metric uploads to them. Network policy, TLS identity, credentials, and routing remain deployment responsibilities. A pull design changes who initiates the connection; it does not remove network boundaries or make arbitrary exporters safe to expose publicly.

Read the generated metric precisely

For a configured instance, Prometheus generates up, with value one for a successful scrape and zero for a failed scrape. It is a metric, not an up label. Scrape duration and sample counts provide additional collection evidence. An exporter can return a valid response while the database behind it is unavailable, or while an important business metric is missing. Such a scrape can still succeed.

Use application outcomes and exporter-specific dependency metrics to answer those further questions. Conversely, up=0 describes a collection failure, which might be caused by the monitoring path while users remain unaffected. Keep that distinction in alert wording so responders investigate the correct surface instead of assuming every collection problem is a customer outage.

When it breaks

A connection refusal, timeout, certificate error, or invalid exposition can prevent a scrape from succeeding. Inspect the target’s last scrape error before changing the interval. A slower interval does not repair a wrong port, and a larger timeout can conceal a slow exporter without removing its cost. Review emitted series and response size when collection becomes expensive. Keep scrape timeouts within the configured interval, and evaluate exporter load before shortening the schedule. Faster polling can multiply database queries performed by an exporter without making those queries cheaper.

A target removed from discovery is different from a configured target returning zero. Its absence can eventually remove the corresponding series from instant queries. Therefore, a fleet-wide sum of up cannot prove that every expected instance still exists. Compare discovery inventory with the intended deployment when completeness matters. Likewise, a successful /metrics request is not a substitute for testing a checkout or login operation.

Check it quickly

Save this as pull.py and run python3 pull.py with Docker available. It uses prom/prometheus:v3.5.1; an uncached image requires an initial registry download. One disposable container scrapes its own loopback endpoint and an unused loopback port. No host port is published, and the container has no external network access.

import json
import subprocess
import uuid

# Requires Docker and the public Prometheus 3.5.1 image.
# All HTTP traffic stays on loopback inside one disposable container.
script = r"""set -eu
cat > /tmp/prom.yml <<'CONFIG'
global:
 scrape_interval: 1s
 scrape_timeout: 500ms
scrape_configs:
 - job_name: self
 static_configs:
 - targets: ['127.0.0.1:9090']
 - job_name: refused
 static_configs:
 - targets: ['127.0.0.1:1']
CONFIG
/bin/prometheus --config.file=/tmp/prom.yml --storage.tsdb.path=/tmp/data \
 --web.listen-address=127.0.0.1:9090 > /tmp/prom.log 2>&1 &
for attempt in $(seq 1 30); do
 if wget -qO /tmp/ready.json 'http://127.0.0.1:9090/api/v1/query?query=count%28up%29%20%3D%3D%202%20and%20sum%28up%29%20%3D%3D%201'; then
 if grep -q '"status":"success"' /tmp/ready.json && ! grep -q '"result":\[\]' /tmp/ready.json; then
 wget -qO- 'http://127.0.0.1:9090/api/v1/query?query=up'
 exit 0
 fi
 fi
 sleep 1
done
cat /tmp/prom.log >&2
exit 1
"""
name = "xbcatm-prom-live-" + uuid.uuid4().hex[:12]
try:
 result = subprocess.run(
 ["docker", "run", "--rm", "--name", name, "--network", "none", "-i",
 "--entrypoint", "/bin/sh", "prom/prometheus:v3.5.1", "-s"],
 input=script, text=True, capture_output=True, timeout=45,
 )
finally:
 subprocess.run(["docker", "rm", "-f", name],
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
if result.returncode:
 raise RuntimeError(result.stderr or result.stdout)
payload = json.loads(result.stdout)
assert payload["status"] == "success"
values = {item["metric"]["job"]: float(item["value"][1])
 for item in payload["data"]["result"]}
assert values == {"self": 1.0, "refused": 0.0}, values
print(json.dumps({"up_by_job": values, "scrape_reachability_only": True}, indent=2))

The result must contain self: 1.0 and refused: 0.0. Assertions check actual query results after both targets have been evaluated; the startup loop avoids interpreting an initially empty database as a failure. The final cleanup removes only the uniquely named test container. This verifies collection outcomes, not business availability or discovery completeness.

Turn the outcome into a useful investigation

In a real incident, identify the failing instance, read its scrape error, and test connectivity from the collector’s network context. Compare the response with the expected exporter format and check its dependency status. Keep secrets out of copied diagnostic output. Once collection recovers, confirm both up and the specific business series used by dashboards. Restoring one does not automatically establish the other.

Sources

Prometheus: jobs and instances, generated scrape metrics and target identity, checked 2026-09-13.
Prometheus: configuration, scrape intervals, timeouts and target configuration, checked 2026-09-13.
Prometheus: HTTP API, instant query response format, checked 2026-09-13.

Dashboards summarize; they do not magnify

Sun, 18 Jul 2027 00:00:00 +0000

A useful dashboard helps a reader decide what to inspect next. It reduces a large collection of measurements into a small set of questions: are users succeeding, where is the impact, and is capacity becoming constrained? Reduction must preserve important differences. A green global percentage can conceal a failing region, and a blank panel can mean missing telemetry rather than a quiet, healthy service.

Give the page a reading order

Start with the service outcome and observation coverage, then show traffic, latency, errors, and relevant saturation. Provide drilldowns for regions, operations, or dependencies when those distinctions change the next action. Put deployment annotations and links to diagnostic material near the relevant views. Correlated lines can suggest where to look, but their timing alone does not prove that a deployment or dependency caused the incident.

Each panel should answer a clear question, not necessarily contain exactly one metric. Comparing queue depth with its configured limit can be useful when both share a unit. Mixing unrelated units on a complicated dual-axis chart makes apparent relationships easy to invent. State the unit, aggregation window, population, and threshold meaning. A reader should not need to remember whether an axis represents milliseconds, seconds, fractions, or percentages.

Preserve weighting and coverage

Compute a service error fraction from the total bad events divided by total eligible events over matching windows. An unweighted mean of regional percentages gives a quiet region the same influence as a busy one. That is a different statistic, not a harmless display choice. Even a correctly weighted result should have a useful breakdown when small populations matter to the service commitment.

Freshness is another dimension of scope. Display the last successful observation or an explicit coverage signal, and define when stale data becomes unknown. A service-specific freshness threshold is separate from the storage system’s staleness rules. Filling gaps with zero can turn an unavailable exporter into an apparently perfect error rate or an empty queue. Missing data should remain visible, with a next step to investigate collection or the service itself.

When it breaks

Automatically chosen axis ranges can make a tiny variation look like a major incident, while an overly broad fixed range can conceal a meaningful threshold crossing. Select scales according to the operational question and keep comparable panels consistent. An average latency also hides a slow minority; use appropriate distributions and counts without averaging precomputed percentiles across instances.

An attractive dashboard can still become a collection of unrelated snapshots. Different time zones, filters, refresh intervals, and aggregation windows prevent a coherent comparison. Shared variables should have explicit defaults and visible scope, especially for production environments. Test the page with a regional failure, sparse traffic, and absent measurements. If those situations all appear green, the page is compressing away information needed for decisions.

Check it quickly

Save this as dashboard.py and run Python 3. It computes panel-ready values from synthetic counts for one completed sixty-second interval. Two regions have fresh observations; a third has missing values and an older observation time. The ninety-second freshness threshold is an example dashboard policy, not a claim about Prometheus defaults. No Grafana installation is required for this aggregation exercise.

import json
from fractions import Fraction

# Counts cover the same completed 60-second interval.
rows = [
 {"region": "west", "requests": 9800, "errors": 0, "age_seconds": 10},
 {"region": "east", "requests": 200, "errors": 100, "age_seconds": 10},
 {"region": "south", "requests": None, "errors": None, "age_seconds": 240},
]
fresh = [r for r in rows if r["age_seconds"] <= 90
 and r["requests"] is not None and r["errors"] is not None]
missing = [r["region"] for r in rows if r not in fresh]
requests = sum(r["requests"] for r in fresh)
errors = sum(r["errors"] for r in fresh)
regional = {r["region"]: Fraction(r["errors"], r["requests"]) for r in fresh}
known_ratio = Fraction(errors, requests)
unweighted = sum(regional.values()) / len(regional)

assert known_ratio == Fraction(1, 100)
assert regional["east"] == Fraction(1, 2) and unweighted == Fraction(1, 4)
assert missing == ["south"]
print(json.dumps({
 "coverage": "incomplete",
 "fresh_requests_per_second": round(requests / 60, 3),
 "fresh_error_percent": float(known_ratio * 100),
 "regional_error_percent": {k: float(v * 100) for k, v in regional.items()},
 "wrong_unweighted_percent": float(unweighted * 100),
 "unknown_regions": missing,
}, indent=2))

Fresh observations contain 10,000 requests and 100 errors, giving 166.667 requests per second and a weighted 1% error fraction. The small eastern region is nevertheless failing half its requests. Averaging the two regional percentages would incorrectly label the combined request fraction as 25%. Show the 1% overview with its regional breakdown instead of choosing one statistic to hide the other.

The output also declares coverage incomplete and lists the southern region as unknown. Neither its requests nor its errors are invented as zero. This fixture checks the arithmetic and missing-data meaning behind the panels; it does not validate an actual dashboard query or screen. Carry the same definitions into the rendered dashboard and review it against known incident and telemetry-loss cases.

Sources

Grafana: Dashboard Best Practices, goals, focused questions, and a coherent reading order, checked 2026-09-13.
Google SRE: Monitoring Distributed Systems, service-oriented dashboards and signals, checked 2026-09-13.
Prometheus: Metric and Label Naming, units and meaningful aggregation, checked 2026-09-13.

Structured logs make their query contract explicit

Tue, 09 Mar 2027 00:00:00 +0000

Structured logging assigns stable meaning to fields instead of asking each query to infer everything from message text. A record might contain an event name, request identifier, numeric duration, severity, and a human-readable message. Free text remains useful and can also be searched or parsed. The practical gain is that routine questions can use a documented schema whose meaning survives changes to explanatory wording.

That schema needs more than curly braces. A duration stored as a number supports numeric comparisons, while a duration stored as a string may be indexed or compared differently. Milliseconds and seconds must not share an ambiguous field name. An event name should describe what happened consistently across versions. A schema version can identify deliberate format changes, but it does not automatically transform old records or keep every consumer compatible.

Put context on the correct record

A request identifier is useful only if it belongs to the operation that emitted the record. Concurrent requests make a process-global mutable variable unsuitable for that purpose. Python context variables can hold task-local state across asynchronous work, provided values are set and reset at the intended scope. This does not mean every thread or custom execution mechanism inherits context in the same way; verify the actual boundary.

Capture context while it is available. This example formats each record synchronously in the emitting task, so its formatter can read the current context variable. A queue-based logging pipeline needs a different placement: attach the required fields before handing the record to another thread or process. Reading the listener’s current request context would associate the record with the listener, rather than reliably recovering the original request.

When it breaks

Hand-concatenating JSON can turn quotes or line breaks from a message into invalid records. A line-oriented collector may then split one application event into several apparent events. Use a JSON serializer and one complete JSON object per physical line. The message can retain a logical newline inside its string value while the serialized line contains an escaped representation. Rendering that message later should still keep its record boundary clear.

Structured output does not automatically create indexes, reduce query bills, or guarantee delivery. Unbounded fields, accidental type changes, missing identifiers, and sensitive values can make a JSON stream difficult or inappropriate to use. Select fields according to real queries, keep the schema stable, and verify the collector’s parser and retention rules. Searchable structure is a contract between producers and consumers, not a property established by the file extension.

Check it quickly

Save this as structured-logging.py and run it with Python; the executed version was 3.14.4. It uses only standard-library modules. Two asynchronous requests each log a start event, wait at a shared gate, then log a completion event containing a deliberately troublesome message. The durations are fixed synthetic values for checking JSON types, not measurements of request performance. Output is captured in memory for assertions.

import asyncio
import io
import json
import logging
from collections import Counter
from contextvars import ContextVar

request_id = ContextVar("request_id", default=None)
stream = io.StringIO()


class JsonFormatter(logging.Formatter):
 def format(self, record):
 return json.dumps({"schema_version": 1, "event": record.event,
 "request_id": request_id.get(), "worker": record.worker,
 "duration_ms": record.duration_ms,
 "level": record.levelname, "message": record.getMessage()},
 ensure_ascii=False, separators=(",", ":"))


logger = logging.getLogger("structured-log-lab")
logger.setLevel(logging.INFO)
logger.propagate = False
handler = logging.StreamHandler(stream)
handler.setFormatter(JsonFormatter())
logger.addHandler(handler)


async def request(name, ready, release):
 token = request_id.set(name)
 try:
 logger.info("starting %s", name, extra={"event": "request.started",
 "worker": name, "duration_ms": 0})
 ready.set()
 await release.wait()
 await asyncio.sleep(0)
 logger.info('untrusted line\n{"event":"forged"}',
 extra={"event": "request.completed", "worker": name, "duration_ms": 4})
 finally:
 request_id.reset(token)


async def main():
 ready = [asyncio.Event(), asyncio.Event()]
 release = asyncio.Event()
 tasks = [asyncio.create_task(request(name, signal, release))
 for name, signal in zip(("req-a", "req-b"), ready)]
 await asyncio.gather(*(signal.wait() for signal in ready))
 release.set()
 await asyncio.gather(*tasks)
 assert request_id.get() is None


try:
 asyncio.run(main())
finally:
 logger.removeHandler(handler)
 handler.close()
lines = stream.getvalue().splitlines()
records = [json.loads(line) for line in lines]
assert len(lines) == 4
assert Counter(record["request_id"] for record in records) == {"req-a": 2, "req-b": 2}
assert all(record["request_id"] == record["worker"] for record in records)
assert all(type(record["duration_ms"]) is int for record in records)
completed = [record for record in records if record["event"] == "request.completed"]
assert len(completed) == 2 and all("\n" in record["message"] for record in completed)
assert not any(record["event"] == "forged" for record in records)
print(json.dumps({"physical_lines": len(lines), "parsed_records": len(records),
 "requests": 2, "context_mixing": False,
 "numeric_duration_preserved": True, "forged_records": 0}, indent=2))

The assertions require four physical lines and four parsed records, with two records for each request identifier. Each record’s identifier must match the worker that emitted it, even though the requests overlap. Duration values remain JSON numbers. The embedded newline survives inside each decoded completion message, while no additional record named forged appears. Resetting each token also leaves the enclosing context without a request identifier.

This verifies task isolation, synchronous formatting, and line-safe serialization. It does not exercise a background queue, collector retry, storage indexing, or secret redaction. When integrating it, check the complete path with representative fields and the actual concurrency model. Inspect what the backend stores after parsing, and test the queries that motivated those fields. A successful producer-side serialization check is one useful boundary test, while a dependable logging system also needs evidence that the intended records reach their consumers with their meaning intact.

Sources

Python logging cookbook, contextual records and logging across concurrent work, checked 2026-09-13.
Python contextvars, task-local context and token reset behavior, checked 2026-09-13.
Python json, serialization, escaped characters, and JSON value types, checked 2026-09-13.
JSON Lines, one complete JSON value per physical line, checked 2026-09-13.

An access log separates client outcomes from upstream attempts

Mon, 01 Mar 2027 00:00:00 +0000

An nginx access log records request information when logging runs at the end of processing. The selected fields can distinguish what the client requested, which URI ultimately handled it, what response was sent, and what happened during upstream attempts. These are different observations. A useful format keeps them separate instead of assuming that one status, duration, or path explains the whole request by itself.

When it breaks

The final client status can conceal an earlier upstream failure. If nginx retries another server successfully, the client may receive 200 while an attempt list records a failed connection first. Conversely, a locally generated response has no upstream operation to time. Treating every request as exactly one backend response loses the information needed to explain retries and local handling.

Original and current URI fields also differ after internal redirects. $request retains the request line, and $request_uri retains the original URI and query. $uri can reflect a later normalized or rewritten path. An investigator who logs only the final URI may not know which public address triggered it, while logging only the original path can hide the final handler’s context.

Timing fields are not interchangeable latency components that can always be subtracted into a precise diagnosis. Request time includes processing through the client-facing completion point used by nginx logging. Upstream timing fields describe their own phases and can contain several attempt values. Their resolution, missing values, and boundaries matter; a zero-looking millisecond value does not establish that no work occurred.

Check it quickly

Run the script with host Python, Docker, and the pinned nginx and Python images. The recorded environment used nginx 1.28.0 and Python 3.11.16 on Linux arm64 with Docker 29.3.1. It configures a real backend plus an unreachable port inside the proxy container, validates nginx configuration, publishes only a host-loopback proxy port, and removes its own resources after inspecting the logs:

from contextlib import contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import http.client,json,socket,subprocess,time,uuid

NGINX='nginx@sha256:552e7481ca93ffccd046aa658dbbed22caefbc09c66fa7cd247cbb90b8a5c609'
PYTHON='python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84'
def docker(*args,check=True):
 result=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=120)
 if check:assert result.returncode==0,result.stderr
 return result
def wait(predicate):
 deadline=time.monotonic()+15
 while time.monotonic()<deadline:
 try:
 if predicate():return
 except OSError:pass
 time.sleep(.1)
 raise AssertionError('fixture did not become ready')

@contextmanager
def lab(configuration,files=None,backend=None):
 name='xbcatm-nginx-'+uuid.uuid4().hex[:10];tags=[name+':proxy',name+':backend'];containers=[name+'-proxy',name+'-backend']
 try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'files').mkdir();(root/'nginx.conf').write_text(configuration)
 for path,content in (files or {}).items():
 target=root/'files'/path;target.parent.mkdir(parents=True,exist_ok=True);target.write_text(content)
 docker('network','create',name)
 if backend is not None:
 (root/'backend.py').write_text(backend)
 (root/'backend.Dockerfile').write_text(f'FROM {PYTHON}\nCOPY backend.py /backend.py\nCMD ["python", "/backend.py"]\n')
 docker('build','--network','none','-f',str(root/'backend.Dockerfile'),'-t',tags[1],str(root))
 docker('run','-d','--name',containers[1],'--network',name,'--network-alias','backend',tags[1])
 wait(lambda:docker('exec',containers[1],'python','-c','import socket;s=socket.create_connection(("127.0.0.1",8081),timeout=.5);s.close()',check=False).returncode==0)
 (root/'Dockerfile').write_text(f'FROM {NGINX}\nCOPY nginx.conf /etc/nginx/nginx.conf\nCOPY files /srv\n')
 docker('build','--network','none','-t',tags[0],str(root))
 docker('run','--rm','--network',name,tags[0],'nginx','-t')
 docker('run','-d','--name',containers[0],'--network',name,'-p','127.0.0.1::8080',tags[0])
 details=json.loads(docker('inspect',containers[0]).stdout)[0]
 binding=details['NetworkSettings']['Ports']['8080/tcp'][0];assert binding['HostIp']=='127.0.0.1';port=int(binding['HostPort'])
 def ready():
 connection=socket.create_connection(('127.0.0.1',port),timeout=.5);connection.close();return True
 wait(ready)
 def request(path,method='GET',headers=None,body=None):
 connection=http.client.HTTPConnection('127.0.0.1',port,timeout=5)
 try:
 connection.request(method,path,body=body,headers=headers or {})
 response=connection.getresponse();return response.status,{k.lower():v for k,v in response.getheaders()},response.read()
 finally:connection.close()
 yield containers[0],request
 finally:
 docker('rm','-f',*containers,check=False);docker('image','rm',*tags,check=False);docker('network','rm',name,check=False)

def config(locations,extra=''):
 return 'worker_processes 1;\nerror_log /tmp/error.log notice;\nevents {}\nhttp {\naccess_log off;\n'+extra+'\nserver { listen 8080; root /srv;\n'+locations+'\n}\n}\n'

backend=r'''from http.server import BaseHTTPRequestHandler,HTTPServer
import json
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 body=self.rfile.read(int(self.headers.get('Content-Length','0')))
 document={'path':self.path,'method':self.command,'body':body.decode(),'headers':{key:self.headers.get(key) for key in ['Host','X-Forwarded-For','X-Real-IP','X-Forwarded-Proto','X-Forwarded-Host']}}
 data=json.dumps(document).encode()
 self.send_response(200);self.send_header('Content-Type','application/json');self.send_header('Content-Length',str(len(data)));self.send_header('X-Backend-Info','fixture-node');self.end_headers();self.wfile.write(data)
 do_POST=do_GET
 def log_message(self,*args):pass
HTTPServer(('0.0.0.0',8081),Handler).serve_forever()
'''

fields={'request':'$request','request_uri':'$request_uri','uri':'$uri','status':'$status','upstream_status':'$upstream_status','upstream_addr':'$upstream_addr','request_time':'$request_time','upstream_connect_time':'$upstream_connect_time','upstream_header_time':'$upstream_header_time','upstream_response_time':'$upstream_response_time','body_bytes_sent':'$body_bytes_sent','bytes_sent':'$bytes_sent'}
extra="log_format details escape=json '"+json.dumps(fields,separators=(',',':'))+"';\nupstream service_pool { server 127.0.0.1:1 max_fails=1; server backend:8081 backup; }"
locations=r'''
access_log /tmp/access.json details;
location = /local { return 204; }
location = /rewrite { rewrite ^ /normalized last; }
location = /normalized { return 200 "normalized"; }
location /proxy {
 proxy_next_upstream error timeout;
 proxy_next_upstream_tries 2;
 proxy_pass http://service_pool;
}
'''
with lab(config(locations,extra),backend=backend) as (container,request):
 assert request('/local')[0]==204
 assert request('/rewrite?x=1')[2]==b'normalized'
 status,headers,body=request('/proxy?x=1');assert status==200
 def lines():return docker('exec',container,'cat','/tmp/access.json').stdout.splitlines()
 wait(lambda:len(lines())>=3)
 records={record['request_uri']:record for record in map(json.loads,lines())}
 print(json.dumps(records,indent=2),flush=True)
 local=records['/local'];assert local['status']=='204' and local['upstream_status']=='' and local['upstream_response_time']==''
 assert local['body_bytes_sent']=='0' and int(local['bytes_sent'])>0
 changed=records['/rewrite?x=1'];assert changed['request']=='GET /rewrite?x=1 HTTP/1.1' and changed['uri']=='/normalized'
 proxy=records['/proxy?x=1'];assert proxy['status']=='200' and proxy['upstream_status']=='502, 200'
 assert len(proxy['upstream_addr'].split(', '))==2 and int(proxy['body_bytes_sent'])==len(body)
 assert float(proxy['request_time'])>=0 and len(proxy['upstream_response_time'].split(', '))==2
 print('a local 204 has no upstream and no body bytes, but still sends response-header bytes')
 print('the final uri reflects the internal redirect while request and request_uri retain the original request')
 print('one client-visible 200 records upstream attempts 502, 200 and separate attempt-address/timing lists')

The /local request returns 204 without contacting an upstream. Its log has zero body bytes but a positive bytes_sent value because response headers were sent. In this JSON-escaped format, absent upstream fields are empty strings. This is different from a measured zero-second upstream operation and from the dash used by the tested version’s default escaping path.

The request /rewrite?x=1 internally becomes /normalized. The log retains GET /rewrite?x=1 HTTP/1.1 as the request line and the original URI with its query, while uri records /normalized. This verifies the difference at the actual log-write point, rather than inferring it only from the rewrite declaration or the response body.

The proxy request first tries loopback port one, where the disposable proxy container has no listener. nginx records that failed attempt with status 502 and then reaches the configured backup server. The final client response is 200, while upstream_status is the string 502, 200. The first value is nginx’s recorded failure outcome, not evidence that a backend actually returned an HTTP 502 response.

The same event contains two upstream addresses and two response-time entries. Connect and header timings can contain a dash for an attempt that never reached those phases, even though the overall variable exists. That differs from an entirely absent upstream variable. Preserve these relationships when parsing instead of replacing every unusual value with an indistinguishable numeric zero.

The logged body byte count equals the actual response-body length read by the client in this uncompressed fixture. bytes_sent includes more HTTP response bytes, but neither field is presented as a complete accounting of Ethernet, TCP, or TLS overhead. The example uses the counters to distinguish body content from header traffic, not to claim a physical-network billing measurement.

The script waits until the three events are present before parsing them. Logging occurs after request handling, and configured buffering can delay visibility further. A successful HTTP response does not guarantee that a remote log collector has already received or retained the corresponding event.

Keep diagnostic meaning in the schema

Choose fields that answer concrete operational questions: original target, final route, client status, upstream attempt outcomes, and relevant durations. Keep attempt lists intact until a parser with the documented grammar separates them. Multiple upstream groups or internal transitions can introduce additional structure beyond the simple two-attempt case shown here.

Avoid treating client-controlled forwarding headers as verified caller identity. Also consider whether query strings or other request fields contain sensitive values before retaining them. Richer logs are useful only when the information has a clear purpose and appropriate access and retention controls.

Test format changes by producing representative local responses, retries, rewrites, and failures, then parsing actual records. Validate rotation and collection separately. A trustworthy access log connects each field to its measured event and preserves missing or multi-attempt information instead of flattening it into a misleading summary.

Sources

nginx HTTP log module, request-completion timing, byte counters, and log-format fields, checked 2026-09-12.
nginx upstream variables, per-attempt addresses, statuses, and timing lists, checked 2026-09-12.
nginx core request variables, original request text and current normalized URI, checked 2026-09-12.
nginx 1.28.0 log implementation, JSON-escaped missing-variable representation, checked 2026-09-12.

Traces connect the recorded parts of a request

Tue, 15 Dec 2026 00:00:00 +0000

A distributed trace connects recorded operations through a shared trace identifier and explicit relationships. A span represents one operation, with start and end timestamps, attributes, events, and status. This makes a trace useful when a checkout request crosses an application, an inventory service, and a payment service. The resulting picture depends on which operations were instrumented, retained, and exported; it is not an automatic inventory of every action the request performed.

The trace identifier groups related spans. A span identifier distinguishes one operation within that trace, while the parent identifier connects a child to its immediate predecessor. Across HTTP, W3C traceparent carries the trace identifier, the sending span identifier, and trace flags. The receiving instrumentation extracts that context and creates a new span. Copying a request identifier into unrelated log messages does not create these span relationships. For work with several causal inputs, span links can express additional relationships without assigning multiple parents to one span.

Read relationships before adding durations

A server span can contain database spans and several overlapping outbound requests. Adding all those durations counts overlapping time repeatedly. Even subtracting the sum of child durations from the parent can produce nonsense when siblings run concurrently. Read the timeline and identify intervals on the critical path. Cross-machine clock differences and missing instrumentation also limit how precisely apparent gaps can be interpreted.

Span status needs its own interpretation. An HTTP response code, a recorded exception event, and an OpenTelemetry error status are separate pieces of data. Instrumentation may connect them according to its conventions; manually recording an exception does not justify assuming that every relevant status field changed. Service resource attributes identify the emitter, whereas an individual span describes an operation performed there.

When it breaks

A proxy can drop propagation headers, or a service can fail to extract them. The downstream work then appears as a separate trace even though the business request reached it successfully. A shared trace identifier with the wrong parent is another failure: related work is grouped together, but its causal position is wrong. Test both identifiers at a boundary instead of checking only that some spans arrived.

An absent span has several possible causes, including sampling, missing instrumentation, asynchronous context loss, and export failure. It does not prove that the operation never ran. Likewise, a green trace view does not establish that every downstream dependency succeeded. Correlate it with request outcome metrics and relevant logs, and check collector or exporter health before interpreting an incomplete picture as complete evidence.

Check it quickly

Create a Python virtual environment and install opentelemetry-sdk==1.35.0, which installs the matching API dependency. Save this as trace-flow.py and run it with that environment’s Python. It was executed with Python 3.14.4. The script starts a loopback HTTP server, uses separate caller and worker tracer providers, and exports to memory. It closes the server after the requests; no collector is required.

import json
from http.server import BaseHTTPRequestHandler, HTTPServer
from threading import Thread
from urllib.error import HTTPError
from urllib.request import Request, urlopen

from opentelemetry.context import Context
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.trace import SpanKind, Status, StatusCode
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

exporter = InMemorySpanExporter()
providers = [TracerProvider(resource=Resource({"service.name": name}))
 for name in ("caller", "worker")]
for provider in providers:
 provider.add_span_processor(SimpleSpanProcessor(exporter))
caller, worker = [p.get_tracer("trace-lab") for p in providers]
propagator = TraceContextTextMapPropagator()


class Handler(BaseHTTPRequestHandler):
 def log_message(self, *args):
 pass

 def do_GET(self):
 carrier = {key.lower(): value for key, value in self.headers.items()}
 parent = propagator.extract(carrier, context=Context())
 with worker.start_as_current_span("handle" + self.path,
 context=parent, kind=SpanKind.SERVER) as span:
 status = 200
 if self.path == "/fail":
 try:
 raise ValueError("synthetic inventory failure")
 except ValueError as error:
 span.record_exception(error)
 span.set_status(Status(StatusCode.ERROR, str(error)))
 status = 503
 body = b"lab response\n"
 self.send_response(status)
 self.send_header("Content-Length", str(len(body)))
 self.end_headers()
 self.wfile.write(body)


server = HTTPServer(("127.0.0.1", 0), Handler)
thread = Thread(target=server.serve_forever, daemon=True)
thread.start()
base = f"http://127.0.0.1:{server.server_port}"
try:
 with caller.start_as_current_span("request", context=Context()):
 with caller.start_as_current_span("outbound", kind=SpanKind.CLIENT):
 headers = {}
 propagator.inject(headers)
 with urlopen(Request(base + "/ok", headers=headers), timeout=5) as response:
 assert response.status == 200
 response.read()
 try:
 with urlopen(base + "/fail", timeout=5) as response:
 raise AssertionError("Expected HTTP 503")
 except HTTPError as response:
 with response:
 assert response.code == 503
 response.read()
finally:
 server.shutdown()
 thread.join(timeout=5)
 server.server_close()

for provider in providers:
 assert provider.force_flush()
spans = {span.name: span for span in exporter.get_finished_spans()}
assert len(spans) == 4
root, client, received, orphan = [spans[name]
 for name in ("request", "outbound", "handle/ok", "handle/fail")]
assert root.parent is None
assert root.context.trace_id == client.context.trace_id == received.context.trace_id
assert client.parent.span_id == root.context.span_id
assert received.parent.span_id == client.context.span_id and received.parent.is_remote
assert orphan.parent is None and orphan.context.trace_id != root.context.trace_id
assert orphan.status.status_code == StatusCode.ERROR
assert [event.name for event in orphan.events] == ["exception"]
assert received.resource.attributes["service.name"] == "worker"
assert client.resource.attributes["service.name"] == "caller"
print(json.dumps({"exported_spans": len(spans), "linked_spans": 3,
 "unpropagated_request_started_new_trace": True,
 "failed_span_status": orphan.status.status_code.name,
 "exception_events": len(orphan.events)}, indent=2))
for provider in providers:
 provider.shutdown()

The first request creates three linked spans: request, outbound, and handle/ok. The assertions verify one trace identifier, two correct parent links, a remote parent on the server span, and different service names on the two providers. The second request deliberately sends no propagation header. It returns HTTP 503, creates a separate root span, and explicitly records both an exception event and an ERROR status. Four spans are exported altogether.

These are two providers in one Python process communicating over real local HTTP. The exercise verifies propagation and recording behavior, not deployment-wide clock synchronization, collector reliability, or concurrent critical-path analysis. In a deployed service, repeat the boundary checks through the actual ingress and client libraries, then confirm export health and sampling configuration. A useful trace is one whose recorded relationships and limitations are understood before its timeline is used to explain an incident.

Sources

OpenTelemetry traces, span structure, relationships, and trace identity, checked 2026-09-13.
W3C Trace Context, traceparent fields and propagation across process boundaries, checked 2026-09-13.
OpenTelemetry Python instrumentation, manual spans, processors, and explicit exception recording, checked 2026-09-13.

Put the unit in the metric name

Fri, 30 Oct 2026 00:00:00 +0000

A metric named request_duration leaves readers guessing whether 250 means seconds, milliseconds, or something else. A Prometheus-facing name such as http_request_duration_seconds carries that contract into queries, recording rules, and alerts. The contract still needs a correct implementation. Appending _seconds to measurements collected in milliseconds changes the spelling, not the magnitude, and can create a thousandfold monitoring error.

Convert at a defined boundary

Choose one quantity and one unit for each metric family. Prometheus recommends base units such as seconds and bytes, with a unit suffix where applicable. A duration and an absolute timestamp can both use seconds, but they are different quantities: last_success_timestamp_seconds should not be combined with request_duration_seconds. Describe the measurement and boundary in its help text so that a useful name has an equally clear definition.

If a library reports milliseconds, convert once before observation or export. Preserve sufficient precision instead of truncating a short duration to zero. Configure the dashboard to display the base unit appropriately; a formatter may show 0.250 seconds as 250 milliseconds without changing stored samples. Do not add another conversion in a recording rule because the panel happens to use milliseconds on its axis.

Unit and type answer different questions

An accumulating byte count conventionally ends in _bytes_total; a current memory usage gauge ends in _bytes. Both measure bytes, but only the former represents accumulated work and is suitable for a counter rate. Counts of requests conventionally use _total without inventing a physical unit. Histogram families add their own component suffixes, such as _seconds_bucket, _seconds_sum, and _seconds_count, to the duration family name.

A ratio commonly represents a fraction between zero and one; percentages use a different numeric scale. State which one a panel expects. A counter may reset when its process restarts, and Prometheus counter-rate functions account for decreases within the observed series. An additional _created series is not required for that behavior. Apply rate calculations before aggregating independent counters so that their individual resets remain visible.

When it breaks

A correctly named metric can still contain incorrect values. Mixing seconds and milliseconds under the same identity contaminates history, so changing only a label or help string cannot repair old samples. Treat a unit migration as a contract change: update the producer, queries, alerts, and dashboards together, and keep historical comparisons explicit. A new identity may be appropriate when the old series has an incompatible meaning.

The suffix also cannot enforce counter behavior. A gauge named _total remains a bad input to a counter-rate interpretation if it moves down during ordinary operation. Conversely, dividing an already calculated per-second rate by the interval again invents a different quantity. These are dimensional and semantic errors, not display problems. Other telemetry conventions may carry units in metadata rather than names; follow the documented exporter mapping instead of renaming every upstream instrument blindly.

Check it quickly

Save the following as units.py and run Python 3. It converts three synthetic latency measurements and emits a small Prometheus text exposition with help and type declarations. Decimal arithmetic makes the millisecond-to-second conversion explicit. The comparison demonstrates why a renamed but unconverted producer cannot share the same latency threshold.

from decimal import Decimal

latency_ms = [Decimal('120'), Decimal('250'), Decimal('630')]
latency_seconds = [value / Decimal('1000') for value in latency_ms]
threshold_seconds = Decimal('0.300')
correct_good = sum(value <= threshold_seconds for value in latency_seconds)
rename_only_good = sum(value <= threshold_seconds for value in latency_ms)
transferred_bytes = 1024 * 1024
used_ratio = Decimal('25') / Decimal('100')

assert correct_good == 2 and rename_only_good == 0
assert sum(latency_seconds) == Decimal('1.000')
assert used_ratio == Decimal('0.25')
print(f"# within 0.300s: converted={correct_good}/3, rename_only={rename_only_good}/3")
print('# HELP demo_last_request_duration_seconds Duration of the most recent completed request.')
print('# TYPE demo_last_request_duration_seconds gauge')
print(f'demo_last_request_duration_seconds {latency_seconds[-1]:.3f}')
print('# HELP demo_transferred_bytes_total Bytes transferred since this process started.')
print('# TYPE demo_transferred_bytes_total counter')
print(f'demo_transferred_bytes_total {transferred_bytes}')
print('# HELP demo_capacity_used_ratio Fraction of configured capacity currently used.')
print('# TYPE demo_capacity_used_ratio gauge')
print(f'demo_capacity_used_ratio {used_ratio}')

The first comment reports two of three observations within 0.300 seconds after conversion, compared with zero of three when millisecond values are merely treated as seconds. The exported last-request gauge is 0.630 seconds, the byte counter is 1,048,576, and the capacity ratio is 0.25, corresponding to 25% in a percentage display.

The complete output passed promtool check metrics using Prometheus 3.5.1. That parser check establishes valid exposition and naming lint for this sample; it cannot infer whether real instrumentation measured the promised quantity. Verify a known input through collection, export, query, and display before trusting a renamed production metric.

Sources

Prometheus: Metric and Label Naming, base units, suffixes, and conventions at the Prometheus boundary, checked 2026-09-13.
Prometheus: Data Model, metric identity and labels, checked 2026-09-13.
Prometheus: Query Functions, counter rates and reset handling, checked 2026-09-13.

Readable evidence before dashboards

Tue, 18 Aug 2026 00:00:00 +0000

Monitoring dashboards are convenient, but convenience is not the same as evidence. This essay argues that for a small static service, the most auditable form of “is the site healthy?” is a set of plain-text facts an operator can read, re-run, and paste into a report — not an aggregated panel.

The distinction matters because the two answer different questions. A dashboard is a lens over many samples, tuned to alert you to trends. A probe is a single, reproducible observation of the public edge. When the job is to prove a release or diagnose a specific request, the probe is the higher-fidelity artifact.

The dashboard’s aggregation is the risk

A dashboard turns thousands of raw observations into a series of aggregated panels: average latency, error rate, uptime percentage. Aggregation is where the audit trail thins. An “uptime 99.98%” figure does not tell you which request failed, which header was wrong, or whether today’s release is the one being served. It is a summary that erases the evidence that produced it unless you keep the raw log elsewhere.

For a confidence property like “the edge serves the tree I staged”, a dashboard is the wrong tool. That property is answered by a single deterministic observation, and a terminal command can answer it exactly once.

A probe you can read

The site ships a small read-only helper, downloads/check-http-headers.sh, that prints the served response in plain text:

HTTP/2 200
content-type: text/html; charset=utf-8
content-security-policy: default-src 'self'; …; script-src 'none'; …

A human reads this line and knows three things at once: the status is 200, the type is HTML, and the no-script policy is actually being served. Compare that to a panel that says “edge OK” — the panel requires trust in whatever computed it; the header line is itself the evidence. The certificate is judged the same way, by reading notAfter as a date rather than a color in a panel.

Replace signal loss with a checkable statement

The mechanism the site favors instead of a dashboard is a contract enforced during build and a set of deterministic commands for the release window. Roughly:

Offline checks validate the artifact (every internal link, every SVG, the no-script boundary, the checksums).
A single read-only probe against the public edge confirms status, headers, and certificate in one invocation.
A byte manifest ties the served tree to the staged tree so “agrees with what we staged” is a hash, not a feeling.

Each of these is a sentence an operator can paste into an issue or a changelog and a future auditor can re-run. None of them requires a panel to be understood.

When you genuinely need a dashboard

None of this says observability is pointless. If you operate many instances, aggregate capacity, or long trends, a dashboard is the right view to see the forest. The claim is narrower: when the question is “is this one release healthy at the public edge”, the answer lives in a reproducible terminal read, and bolting a panel on top adds latency and aggregation loss without adding evidence.

The decision heuristic is simple. If a statement can be checked with one command and its output is the proof, prefer the command and keep dashboard only where you genuinely need the aggregate. That keeps the site’s own operational story as auditable as the site’s content.

Sources

Curl documentation, reading response headers from a single invocation, checked 2026-08-18.
Twelve-Factor: Logs, treating logs as event streams an operator can read, checked 2026-08-18.
OpenTelemetry: Observability primer, observability as high-cardinality, contextual output, checked 2026-08-18.

Observability · XB Field Notes

Baggage carries context across an explicit boundary

Choose what crosses and what gets recorded

When it breaks

Check it quickly

Sources

Annotations explain an alert without changing its identity

Separate identity from the current explanation

A group does not necessarily share its summary

When it breaks

Check it quickly

Sources

pg_stat_statements finds the queries that cost the most

Normalize, then aggregate

Total time, not mean time

Counters accumulate, so compare windows

When it breaks

Check it quickly

Sources

Read procfs as documented views of a changing process

When it breaks

Check it quickly

Match the view to the question

Sources

Trace sampling reduces volume without fixing a cost ceiling

Keep a decision consistent across boundaries

When it breaks

Check it quickly

Sources

Prometheus pulls metrics and records scrape outcomes

A job contains independently scraped instances

Read the generated metric precisely

When it breaks

Check it quickly

Turn the outcome into a useful investigation

Sources

Dashboards summarize; they do not magnify

Give the page a reading order

Preserve weighting and coverage

When it breaks

Check it quickly

Sources

Structured logs make their query contract explicit

Put context on the correct record

When it breaks

Check it quickly

Further reading

Sources

An access log separates client outcomes from upstream attempts

When it breaks

Check it quickly

Keep diagnostic meaning in the schema

Sources

Traces connect the recorded parts of a request

Read relationships before adding durations

When it breaks

Check it quickly

Sources

Put the unit in the metric name

Convert at a defined boundary

Unit and type answer different questions

When it breaks

Check it quickly

Sources

Readable evidence before dashboards

The dashboard’s aggregation is the risk

A probe you can read

Replace signal loss with a checkable statement

When you genuinely need a dashboard

Sources