Tracing · XB Field Notes

Baggage carries context across an explicit boundary

Mon, 27 Mar 2028 00:00:00 +0000

OpenTelemetry baggage is a context-associated collection of string key-value pairs. It lets information known near a request’s entry point travel to downstream operations: a deployment region or a rollout cohort, for example. Unlike a span attribute, baggage is intended for propagation. Unlike trace context, it describes application-selected information rather than the identifiers and flags connecting spans. These mechanisms can travel together without being the same data structure.

Propagation still needs a working boundary. An outgoing client injects context into a carrier such as HTTP headers; the receiver extracts that carrier into its local context. Instrumentation can perform these steps automatically when configured, while manually instrumented code must do them explicitly. Merely creating baggage in one process does not deliver it to an unrelated process, a background task, or an external service.

Choose what crosses and what gets recorded

Baggage entries do not automatically become span, log, or metric attributes. A downstream operation must read the value and copy it deliberately, or use a configured processor implementing that policy. This distinction matters when a team expects a searchable attribute but has only enabled propagation. It also prevents the opposite mistake: copying every incoming key into telemetry without considering its meaning, size, or cardinality.

Context operations return a new context rather than changing every existing reference. Keep the returned context and pass or attach it at the intended boundary. When removing entries for an outgoing request, preserve the trace context that should accompany that request. Treat an outbound allowlist and a telemetry-attribute allowlist as explicit decisions; the values appropriate for one receiver need not be suitable for every later destination.

When it breaks

Baggage has no built-in proof that a value was issued by a trusted component. A caller can supply a header claiming an expensive account tier or a privileged role. Parsing that header successfully establishes its representation, not its authority. Authentication and authorization must use their own trusted inputs. Small, plausible-looking values deserve the same trust analysis as obviously suspicious ones.

Unbounded baggage also increases request-header sizes and can spread sensitive data to unintended recipients, including third-party APIs reached by automatic instrumentation. Limit names and values, review the destinations that receive them, and use propagators for wire encoding. A successful round trip between two copies of one SDK is narrower evidence than interoperability across every language and proxy in a production path. Test those actual boundaries when they matter.

Check it quickly

Create a virtual environment, install opentelemetry-sdk==1.35.0, save the script as baggage.py, and execute it with that environment’s Python. This exercise ran on Python 3.14.4. It uses the real W3C trace-context and baggage propagators with an in-memory span exporter. It does not start an HTTP server: the dictionary is the carrier that an HTTP client would subsequently send.

import json

from opentelemetry import baggage, trace
from opentelemetry.context import Context
from opentelemetry.baggage.propagation import W3CBaggagePropagator
from opentelemetry.propagators.composite import CompositePropagator
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

base = Context()
raw = baggage.set_baggage("region", "eu-west", context=base)
raw = baggage.set_baggage("release", "canary/2", context=raw)
raw = baggage.set_baggage("internal_note", "synthetic-only", context=raw)
assert not baggage.get_all(base)
propagator = CompositePropagator([TraceContextTextMapPropagator(), W3CBaggagePropagator()])
provider = TracerProvider()
exporter = InMemorySpanExporter()
provider.add_span_processor(SimpleSpanProcessor(exporter))
tracer = provider.get_tracer("baggage-lab")
with tracer.start_as_current_span("parent", context=Context()) as parent:
 outgoing = trace.set_span_in_context(parent, baggage.clear(context=raw))
 for key in ("region", "release"):
 outgoing = baggage.set_baggage(key, baggage.get_baggage(key, raw), context=outgoing)
 carrier = {}
 propagator.inject(carrier, context=outgoing)
 received = propagator.extract(carrier, context=Context())
 assert set(baggage.get_all(received)) == {"region", "release"}
 assert baggage.get_baggage("region", received) == "eu-west"
 assert baggage.get_baggage("internal_note", raw) == "synthetic-only"
 assert trace.get_current_span(received).get_span_context().trace_id == parent.get_span_context().trace_id
 with tracer.start_as_current_span("inherited", context=received):
 pass
 with tracer.start_as_current_span("copied", context=received) as span:
 span.set_attribute("deployment.region", baggage.get_baggage("region", received))
assert provider.force_flush()
spans = {span.name: span for span in exporter.get_finished_spans()}
assert not spans["inherited"].attributes
assert spans["copied"].attributes == {"deployment.region": "eu-west"}
independent = propagator.extract({"traceparent": "invalid", "baggage": "region=eu-west"},
 context=Context())
assert not trace.get_current_span(independent).get_span_context().is_valid
assert baggage.get_baggage("region", independent) == "eu-west"
untrusted = propagator.extract({"baggage": "tier=gold"}, context=Context())
assert baggage.get_baggage("tier", untrusted) == "gold"
print(json.dumps({"wire_fields": sorted(carrier),
 "wire_baggage": carrier["baggage"],
 "automatic_span_attributes": dict(spans["inherited"].attributes),
 "explicit_span_attributes": dict(spans["copied"].attributes),
 "baggage_survives_invalid_traceparent": True,
 "caller_supplied_tier_was_parsed": True}, indent=2))
provider.shutdown()

The original context contains region, release, and a synthetic internal note. The outgoing context keeps only the first two entries while retaining the parent span. The extracted region is eu-west, and the release value canary/2 round-trips through the encoded header. Assertions also confirm that constructing the filtered context did not remove the note from the original context or change the empty base context.

Two child spans demonstrate the recording boundary. inherited receives baggage but has no attributes; copied contains only the explicitly assigned deployment.region. A malformed traceparent is tested separately: its trace context is invalid, while the independently encoded baggage still extracts successfully. Finally, a caller-supplied tier=gold is accepted as data without any identity check. These results establish serialization, filtering, and explicit recording behavior. Before relying on the same policy across services, verify the real client instrumentation, receiver extraction, outbound filtering, and the attribute mapping used by the destination backend.

Sources

OpenTelemetry baggage, baggage is separate from telemetry attributes and lacks built-in integrity checks, checked 2026-09-13.
W3C Baggage, HTTP representation, value encoding, and propagation limits, checked 2026-09-13.
W3C Trace Context, trace identity travels separately from application baggage, checked 2026-09-13.

BPF tracepoints peek inside the running kernel

Mon, 06 Mar 2028 00:00:00 +0000

BPF and kernel tracepoints let you observe behavior at the source with a tiny footprint: no agent, no restart, no stopping the workload. Traditional profiling means installing something, restarting with flags, or pausing the process — each of which perturbs the system under study or is forbidden in production. A BPF probe attaches to an already-running kernel, collects its answer, and detaches, leaving the workload exactly as it found it.

For the questions “what is slow” and “what is happening right now,” a minute with the right one-liner beats an hour of log archaeology.

Probe without pausing

A small eBPF program attaches to an existing tracepoint and accumulates counts or histograms while the work keeps running. Because it runs in kernel context, the probe rarely becomes the bottleneck. The program is verified before loading — the kernel checks memory safety and termination — so a buggy probe is rejected rather than crashing the machine. Aggregation happens in kernel maps, meaning millions of events collapse into a small histogram before anything crosses to userspace; the common case costs single-digit percent overhead, often less than the logging the team already runs.

Start from questions, not tools. “Which process opens the most files” maps to one tracing shape, “p99 disk latency by device” to another, and the bpftrace one-liner gallery covers both without writing C. Prefer high-level frontends — bpftrace, bcc tools — over raw BPF unless the measurement truly needs custom logic; the one-liner you can write during an incident beats the perfect program you finish next week.

Tracepoints are the stable names

Kernel tracepoints mark events such as block I/O, scheduling, syscall entry/exit that are less brittle than raw kprobes, so probes survive kernel updates. A kprobe pins a kernel function’s address, which shifts between versions; a tracepoint names a semantic event the kernel promises to keep, so the same probe works after Tuesday’s patch reboot. Attach to the ones closest to your behavior: block tracepoints for disk questions, scheduler tracepoints for latency-off-CPU questions, syscall tracepoints for “who calls what” questions.

List what exists before guessing names — tracepoints are enumerated under the kernel’s tracing interface, and every standard tool can print them. When no tracepoint covers the event, fall back to kprobes or uprobes deliberately and pin the probe to kernel versions in comments, accepting the maintenance cost with open eyes. Stability is a spectrum: tracepoints first, well-known kprobes second, raw addresses never in anything you intend to keep.

Read a histogram, not a firehose

Aggregate a noisy event stream into a latency or count histogram to expose the top offenders without dumping each event. Use that shape to find the culprit, then drill in. A per-event dump of a million syscalls answers nothing a human can read; a log-scale latency histogram shows the bimodal split — fast path versus the slow one percent — in a single screen. The histogram is the diagnosis; individual events are only the confirmation, sampled after the shape tells you where to look.

Keep the aggregation key tight: by process, by file, by syscall — one dimension at a time. Two-dimensional breakdowns explode cardinality and bury the signal, and full-argument dumps reintroduce the firehose through the back door. When the histogram fingers a suspect, switch to targeted tracing on that process or file alone, with a time bound, and stop once the confirming sample exists.

When it breaks

The classic failure is the probe that never attaches: wrong tracepoint name, missing kernel headers, or a locked-down environment blocking BPF loading — and the tool exits quietly or with a cryptic verifier message. Verify the environment first (tracepoints listed, loader permissions present) before doubting the probe logic; most BPF failures are environmental, not logical. In containers, remember the probe usually needs host-level privileges the container runtime does not grant by default.

The second failure is observer effect at scale: an unfiltered probe on a hot path — every syscall on a 100k-rps box — burns whole cores and skews the measurement it was taking. Always aggregate in-kernel, filter to the suspect early, and time-box production probes. If overhead looks suspicious, halve the sampling or narrow the filter and check the answer stays stable; a measurement that changes shape when observed more cheaply was never trustworthy.

Check it quickly

Confirm the tooling works and take a first histogram:

bpftrace -l "tracepoint:syscalls:*" | head -n 5
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @opens[comm] = count(); }' &
BPF_PID=$!; sleep 10; kill -INT $BPF_PID; wait

The first proves tracepoints are visible, and the ten-second sample ranks processes by file opens — the shape of every BPF investigation in miniature.

Sources

Kernel tracing docs on bpftrace, Official bpftrace reference, checked 2026-06-01.
eBPF introduction, Official eBPF documentation, checked 2026-06-01.

Trace sampling reduces volume without fixing a cost ceiling

Sat, 23 Oct 2027 00:00:00 +0000

Trace sampling selects which recorded work reaches the tracing pipeline. It can reduce export traffic and storage, but a configured percentage is not a strict ceiling on bytes, processor time, or money. Requests produce different numbers of spans, attributes vary in size, and traffic itself changes. A one-percent policy applied to a sudden tenfold traffic increase can still produce roughly ten times as much sampled data.

Head sampling makes a decision near the start of a trace. A trace-identifier ratio sampler is useful when identifiers have the randomness its algorithm requires. Its probability describes selection across suitable traffic; it does not promise exactly half of every sixteen requests, or equal coverage of every route. Small populations and rare failures can be poorly represented even when the overall percentage looks reasonable.

Keep a decision consistent across boundaries

Downstream instrumentation should follow the accepted parent decision when that is the intended policy. In the Python SDK, ParentBased delegates root decisions to its configured root sampler and uses separate policies for sampled and unsampled parents. Wrapping TraceIdRatioBased(0.5) therefore differs from treating every downstream span as an independent new coin flip. Context propagation and the receiving sampler configuration must agree.

An unsampled trace still involves application work, context handling, and potentially instrumentation overhead. Dropping export does not make the business request free. Conversely, preserving a parent decision does not guarantee that every child will be visible: a library might be uninstrumented, a service might lose context, or an exporter might fail. Sampling consistency addresses one source of missing data, not the whole delivery path.

When it breaks

A request can look ordinary at its start and fail much later. A head sampler that already dropped it cannot recover the missing spans merely because application code subsequently sets an error status. Tail sampling can inspect more of the completed trace before deciding, but requires buffering, routing related spans together, and managing delayed or missing data. It also cannot reconstruct spans discarded upstream before they reached it.

Another failure is measuring request reliability directly from a biased trace collection. Policies that deliberately retain slow requests or errors change the population being observed. Keep request outcome metrics with a clear denominator, and describe the trace selection policy alongside investigations. For cost control, observe actual exported spans and bytes, queue behavior, collector resources, and retention; a sampling percentage alone is not evidence that a spending limit holds.

Check it quickly

In a Python virtual environment, install opentelemetry-sdk==1.35.0. Save this as trace-sampling.py and run it using that environment; Python 3.14.4 was used here. The custom identifier generator is exclusively a deterministic test fixture. Its sixteen identifiers deliberately straddle this SDK version’s ratio threshold. Do not copy that sequential identifier scheme into production instrumentation, where the sampler relies on appropriate randomness.

import json
from collections import Counter

from opentelemetry.context import Context
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.sdk.trace.id_generator import IdGenerator
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased
from opentelemetry.trace import Status, StatusCode
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator


class TestIds(IdGenerator):
 # Controlled test inputs only; production IDs need suitable randomness.
 def __init__(self):
 self.trace_index = 0
 self.span_index = 0

 def generate_trace_id(self):
 index = self.trace_index
 self.trace_index += 1
 return ((index + 1) << 64) | (index * (1 << 60) + 1)

 def generate_span_id(self):
 self.span_index += 1
 return self.span_index


exporter = InMemorySpanExporter()
provider = TracerProvider(sampler=ParentBased(TraceIdRatioBased(0.5)),
 id_generator=TestIds())
provider.add_span_processor(SimpleSpanProcessor(exporter))
tracer = provider.get_tracer("sampling-lab")
executed_children = 0
late_error_recording = None
for index in range(16):
 with tracer.start_as_current_span(f"root-{index}", context=Context()) as root:
 for child in range(2):
 with tracer.start_as_current_span(f"child-{index}-{child}"):
 executed_children += 1
 if index == 15:
 late_error_recording = root.is_recording()
 root.set_status(Status(StatusCode.ERROR, "error discovered after start"))
assert provider.force_flush()
spans = exporter.get_finished_spans()
counts = Counter(span.context.trace_id for span in spans)
assert len(counts) == 8 and set(counts.values()) == {3} and len(spans) == 24
assert {s.name for s in spans if s.parent is None} == {f"root-{i}" for i in range(8)}
assert executed_children == 32 and late_error_recording is False
assert not any(span.name == "root-15" for span in spans)

exporter.clear()
propagator = TraceContextTextMapPropagator()
for flag, label in (("01", "sampled"), ("00", "unsampled")):
 # High trace-ID bits do not override an accepted remote parent's flag.
 header = f"00-1111111111111111ffffffffffffffff-2222222222222222-{flag}"
 context = propagator.extract({"traceparent": header}, context=Context())
 with tracer.start_as_current_span(label, context=context):
 with tracer.start_as_current_span(label + "-child"):
 pass
assert provider.force_flush()
remote_spans = exporter.get_finished_spans()
assert {span.name for span in remote_spans} == {"sampled", "sampled-child"}
assert next(span for span in remote_spans if span.name == "sampled").parent.is_remote
print(json.dumps({"controlled_roots": 16, "exported_traces": len(counts),
 "exported_spans": len(spans), "executed_child_operations": executed_children,
 "late_error_was_recording": late_error_recording,
 "remote_parent_exported_spans": len(remote_spans)}, indent=2))
provider.shutdown()

The sixteen root operations each start two children. Exactly eight traces, containing twenty-four spans, are exported because of the chosen test inputs; all thirty-two child operations still execute. Root fifteen is deliberately outside the retained set. Its later error status does not cause it to become recording or appear in the exporter. These assertions test behavior at known inputs rather than estimating a sampling percentage statistically.

The second phase extracts two remote parent headers with the same high trace identifier and different sampled flags. Only the sampled parent’s child tree is exported, producing two spans despite that identifier being above the root ratio threshold. The unsampled parent produces none. No collector or tail sampler runs in this experiment. When applying the policy to a real service, test accepted remote context separately from new roots, then validate the resulting volume and diagnostic coverage against the workload that actually matters.

Sources

OpenTelemetry sampling, head and tail decisions and the risk of losing rare failures, checked 2026-09-13.
OpenTelemetry Python sampling reference, ParentBased and TraceIdRatioBased sampler behavior, checked 2026-09-13.
W3C Trace Context, propagated sampling flags and remote parent context, checked 2026-09-13.

Traces connect the recorded parts of a request

Tue, 15 Dec 2026 00:00:00 +0000

A distributed trace connects recorded operations through a shared trace identifier and explicit relationships. A span represents one operation, with start and end timestamps, attributes, events, and status. This makes a trace useful when a checkout request crosses an application, an inventory service, and a payment service. The resulting picture depends on which operations were instrumented, retained, and exported; it is not an automatic inventory of every action the request performed.

The trace identifier groups related spans. A span identifier distinguishes one operation within that trace, while the parent identifier connects a child to its immediate predecessor. Across HTTP, W3C traceparent carries the trace identifier, the sending span identifier, and trace flags. The receiving instrumentation extracts that context and creates a new span. Copying a request identifier into unrelated log messages does not create these span relationships. For work with several causal inputs, span links can express additional relationships without assigning multiple parents to one span.

Read relationships before adding durations

A server span can contain database spans and several overlapping outbound requests. Adding all those durations counts overlapping time repeatedly. Even subtracting the sum of child durations from the parent can produce nonsense when siblings run concurrently. Read the timeline and identify intervals on the critical path. Cross-machine clock differences and missing instrumentation also limit how precisely apparent gaps can be interpreted.

Span status needs its own interpretation. An HTTP response code, a recorded exception event, and an OpenTelemetry error status are separate pieces of data. Instrumentation may connect them according to its conventions; manually recording an exception does not justify assuming that every relevant status field changed. Service resource attributes identify the emitter, whereas an individual span describes an operation performed there.

When it breaks

A proxy can drop propagation headers, or a service can fail to extract them. The downstream work then appears as a separate trace even though the business request reached it successfully. A shared trace identifier with the wrong parent is another failure: related work is grouped together, but its causal position is wrong. Test both identifiers at a boundary instead of checking only that some spans arrived.

An absent span has several possible causes, including sampling, missing instrumentation, asynchronous context loss, and export failure. It does not prove that the operation never ran. Likewise, a green trace view does not establish that every downstream dependency succeeded. Correlate it with request outcome metrics and relevant logs, and check collector or exporter health before interpreting an incomplete picture as complete evidence.

Check it quickly

Create a Python virtual environment and install opentelemetry-sdk==1.35.0, which installs the matching API dependency. Save this as trace-flow.py and run it with that environment’s Python. It was executed with Python 3.14.4. The script starts a loopback HTTP server, uses separate caller and worker tracer providers, and exports to memory. It closes the server after the requests; no collector is required.

import json
from http.server import BaseHTTPRequestHandler, HTTPServer
from threading import Thread
from urllib.error import HTTPError
from urllib.request import Request, urlopen

from opentelemetry.context import Context
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.trace import SpanKind, Status, StatusCode
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

exporter = InMemorySpanExporter()
providers = [TracerProvider(resource=Resource({"service.name": name}))
 for name in ("caller", "worker")]
for provider in providers:
 provider.add_span_processor(SimpleSpanProcessor(exporter))
caller, worker = [p.get_tracer("trace-lab") for p in providers]
propagator = TraceContextTextMapPropagator()


class Handler(BaseHTTPRequestHandler):
 def log_message(self, *args):
 pass

 def do_GET(self):
 carrier = {key.lower(): value for key, value in self.headers.items()}
 parent = propagator.extract(carrier, context=Context())
 with worker.start_as_current_span("handle" + self.path,
 context=parent, kind=SpanKind.SERVER) as span:
 status = 200
 if self.path == "/fail":
 try:
 raise ValueError("synthetic inventory failure")
 except ValueError as error:
 span.record_exception(error)
 span.set_status(Status(StatusCode.ERROR, str(error)))
 status = 503
 body = b"lab response\n"
 self.send_response(status)
 self.send_header("Content-Length", str(len(body)))
 self.end_headers()
 self.wfile.write(body)


server = HTTPServer(("127.0.0.1", 0), Handler)
thread = Thread(target=server.serve_forever, daemon=True)
thread.start()
base = f"http://127.0.0.1:{server.server_port}"
try:
 with caller.start_as_current_span("request", context=Context()):
 with caller.start_as_current_span("outbound", kind=SpanKind.CLIENT):
 headers = {}
 propagator.inject(headers)
 with urlopen(Request(base + "/ok", headers=headers), timeout=5) as response:
 assert response.status == 200
 response.read()
 try:
 with urlopen(base + "/fail", timeout=5) as response:
 raise AssertionError("Expected HTTP 503")
 except HTTPError as response:
 with response:
 assert response.code == 503
 response.read()
finally:
 server.shutdown()
 thread.join(timeout=5)
 server.server_close()

for provider in providers:
 assert provider.force_flush()
spans = {span.name: span for span in exporter.get_finished_spans()}
assert len(spans) == 4
root, client, received, orphan = [spans[name]
 for name in ("request", "outbound", "handle/ok", "handle/fail")]
assert root.parent is None
assert root.context.trace_id == client.context.trace_id == received.context.trace_id
assert client.parent.span_id == root.context.span_id
assert received.parent.span_id == client.context.span_id and received.parent.is_remote
assert orphan.parent is None and orphan.context.trace_id != root.context.trace_id
assert orphan.status.status_code == StatusCode.ERROR
assert [event.name for event in orphan.events] == ["exception"]
assert received.resource.attributes["service.name"] == "worker"
assert client.resource.attributes["service.name"] == "caller"
print(json.dumps({"exported_spans": len(spans), "linked_spans": 3,
 "unpropagated_request_started_new_trace": True,
 "failed_span_status": orphan.status.status_code.name,
 "exception_events": len(orphan.events)}, indent=2))
for provider in providers:
 provider.shutdown()

The first request creates three linked spans: request, outbound, and handle/ok. The assertions verify one trace identifier, two correct parent links, a remote parent on the server span, and different service names on the two providers. The second request deliberately sends no propagation header. It returns HTTP 503, creates a separate root span, and explicitly records both an exception event and an ERROR status. Four spans are exported altogether.

These are two providers in one Python process communicating over real local HTTP. The exercise verifies propagation and recording behavior, not deployment-wide clock synchronization, collector reliability, or concurrent critical-path analysis. In a deployed service, repeat the boundary checks through the actual ingress and client libraries, then confirm export health and sampling configuration. A useful trace is one whose recorded relationships and limitations are understood before its timeline is used to explain an incident.

Sources

OpenTelemetry traces, span structure, relationships, and trace identity, checked 2026-09-13.
W3C Trace Context, traceparent fields and propagation across process boundaries, checked 2026-09-13.
OpenTelemetry Python instrumentation, manual spans, processors, and explicit exception recording, checked 2026-09-13.