Trace sampling selects which recorded work reaches the tracing pipeline. It can reduce export traffic and storage, but a configured percentage is not a strict ceiling on bytes, processor time, or money. Requests produce different numbers of spans, attributes vary in size, and traffic itself changes. A one-percent policy applied to a sudden tenfold traffic increase can still produce roughly ten times as much sampled data.

Head sampling makes a decision near the start of a trace. A trace-identifier ratio sampler is useful when identifiers have the randomness its algorithm requires. Its probability describes selection across suitable traffic; it does not promise exactly half of every sixteen requests, or equal coverage of every route. Small populations and rare failures can be poorly represented even when the overall percentage looks reasonable.

Keep a decision consistent across boundaries

Downstream instrumentation should follow the accepted parent decision when that is the intended policy. In the Python SDK, ParentBased delegates root decisions to its configured root sampler and uses separate policies for sampled and unsampled parents. Wrapping TraceIdRatioBased(0.5) therefore differs from treating every downstream span as an independent new coin flip. Context propagation and the receiving sampler configuration must agree.

An unsampled trace still involves application work, context handling, and potentially instrumentation overhead. Dropping export does not make the business request free. Conversely, preserving a parent decision does not guarantee that every child will be visible: a library might be uninstrumented, a service might lose context, or an exporter might fail. Sampling consistency addresses one source of missing data, not the whole delivery path.

When it breaks

A request can look ordinary at its start and fail much later. A head sampler that already dropped it cannot recover the missing spans merely because application code subsequently sets an error status. Tail sampling can inspect more of the completed trace before deciding, but requires buffering, routing related spans together, and managing delayed or missing data. It also cannot reconstruct spans discarded upstream before they reached it.

Another failure is measuring request reliability directly from a biased trace collection. Policies that deliberately retain slow requests or errors change the population being observed. Keep request outcome metrics with a clear denominator, and describe the trace selection policy alongside investigations. For cost control, observe actual exported spans and bytes, queue behavior, collector resources, and retention; a sampling percentage alone is not evidence that a spending limit holds.

Check it quickly

In a Python virtual environment, install opentelemetry-sdk==1.35.0. Save this as trace-sampling.py and run it using that environment; Python 3.14.4 was used here. The custom identifier generator is exclusively a deterministic test fixture. Its sixteen identifiers deliberately straddle this SDK version’s ratio threshold. Do not copy that sequential identifier scheme into production instrumentation, where the sampler relies on appropriate randomness.

import json
from collections import Counter

from opentelemetry.context import Context
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.sdk.trace.id_generator import IdGenerator
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased
from opentelemetry.trace import Status, StatusCode
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator


class TestIds(IdGenerator):
    # Controlled test inputs only; production IDs need suitable randomness.
    def __init__(self):
        self.trace_index = 0
        self.span_index = 0

    def generate_trace_id(self):
        index = self.trace_index
        self.trace_index += 1
        return ((index + 1) << 64) | (index * (1 << 60) + 1)

    def generate_span_id(self):
        self.span_index += 1
        return self.span_index


exporter = InMemorySpanExporter()
provider = TracerProvider(sampler=ParentBased(TraceIdRatioBased(0.5)),
                          id_generator=TestIds())
provider.add_span_processor(SimpleSpanProcessor(exporter))
tracer = provider.get_tracer("sampling-lab")
executed_children = 0
late_error_recording = None
for index in range(16):
    with tracer.start_as_current_span(f"root-{index}", context=Context()) as root:
        for child in range(2):
            with tracer.start_as_current_span(f"child-{index}-{child}"):
                executed_children += 1
        if index == 15:
            late_error_recording = root.is_recording()
            root.set_status(Status(StatusCode.ERROR, "error discovered after start"))
assert provider.force_flush()
spans = exporter.get_finished_spans()
counts = Counter(span.context.trace_id for span in spans)
assert len(counts) == 8 and set(counts.values()) == {3} and len(spans) == 24
assert {s.name for s in spans if s.parent is None} == {f"root-{i}" for i in range(8)}
assert executed_children == 32 and late_error_recording is False
assert not any(span.name == "root-15" for span in spans)

exporter.clear()
propagator = TraceContextTextMapPropagator()
for flag, label in (("01", "sampled"), ("00", "unsampled")):
    # High trace-ID bits do not override an accepted remote parent's flag.
    header = f"00-1111111111111111ffffffffffffffff-2222222222222222-{flag}"
    context = propagator.extract({"traceparent": header}, context=Context())
    with tracer.start_as_current_span(label, context=context):
        with tracer.start_as_current_span(label + "-child"):
            pass
assert provider.force_flush()
remote_spans = exporter.get_finished_spans()
assert {span.name for span in remote_spans} == {"sampled", "sampled-child"}
assert next(span for span in remote_spans if span.name == "sampled").parent.is_remote
print(json.dumps({"controlled_roots": 16, "exported_traces": len(counts),
                  "exported_spans": len(spans), "executed_child_operations": executed_children,
                  "late_error_was_recording": late_error_recording,
                  "remote_parent_exported_spans": len(remote_spans)}, indent=2))
provider.shutdown()

The sixteen root operations each start two children. Exactly eight traces, containing twenty-four spans, are exported because of the chosen test inputs; all thirty-two child operations still execute. Root fifteen is deliberately outside the retained set. Its later error status does not cause it to become recording or appear in the exporter. These assertions test behavior at known inputs rather than estimating a sampling percentage statistically.

The second phase extracts two remote parent headers with the same high trace identifier and different sampled flags. Only the sampled parent’s child tree is exported, producing two spans despite that identifier being above the root ratio threshold. The unsampled parent produces none. No collector or tail sampler runs in this experiment. When applying the policy to a real service, test accepted remote context separately from new roots, then validate the resulting volume and diagnostic coverage against the workload that actually matters.

Sources