A distributed trace connects recorded operations through a shared trace identifier and explicit relationships. A span represents one operation, with start and end timestamps, attributes, events, and status. This makes a trace useful when a checkout request crosses an application, an inventory service, and a payment service. The resulting picture depends on which operations were instrumented, retained, and exported; it is not an automatic inventory of every action the request performed.

The trace identifier groups related spans. A span identifier distinguishes one operation within that trace, while the parent identifier connects a child to its immediate predecessor. Across HTTP, W3C traceparent carries the trace identifier, the sending span identifier, and trace flags. The receiving instrumentation extracts that context and creates a new span. Copying a request identifier into unrelated log messages does not create these span relationships. For work with several causal inputs, span links can express additional relationships without assigning multiple parents to one span.

Read relationships before adding durations

A server span can contain database spans and several overlapping outbound requests. Adding all those durations counts overlapping time repeatedly. Even subtracting the sum of child durations from the parent can produce nonsense when siblings run concurrently. Read the timeline and identify intervals on the critical path. Cross-machine clock differences and missing instrumentation also limit how precisely apparent gaps can be interpreted.

Span status needs its own interpretation. An HTTP response code, a recorded exception event, and an OpenTelemetry error status are separate pieces of data. Instrumentation may connect them according to its conventions; manually recording an exception does not justify assuming that every relevant status field changed. Service resource attributes identify the emitter, whereas an individual span describes an operation performed there.

When it breaks

A proxy can drop propagation headers, or a service can fail to extract them. The downstream work then appears as a separate trace even though the business request reached it successfully. A shared trace identifier with the wrong parent is another failure: related work is grouped together, but its causal position is wrong. Test both identifiers at a boundary instead of checking only that some spans arrived.

An absent span has several possible causes, including sampling, missing instrumentation, asynchronous context loss, and export failure. It does not prove that the operation never ran. Likewise, a green trace view does not establish that every downstream dependency succeeded. Correlate it with request outcome metrics and relevant logs, and check collector or exporter health before interpreting an incomplete picture as complete evidence.

Check it quickly

Create a Python virtual environment and install opentelemetry-sdk==1.35.0, which installs the matching API dependency. Save this as trace-flow.py and run it with that environment’s Python. It was executed with Python 3.14.4. The script starts a loopback HTTP server, uses separate caller and worker tracer providers, and exports to memory. It closes the server after the requests; no collector is required.

import json
from http.server import BaseHTTPRequestHandler, HTTPServer
from threading import Thread
from urllib.error import HTTPError
from urllib.request import Request, urlopen

from opentelemetry.context import Context
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.trace import SpanKind, Status, StatusCode
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

exporter = InMemorySpanExporter()
providers = [TracerProvider(resource=Resource({"service.name": name}))
             for name in ("caller", "worker")]
for provider in providers:
    provider.add_span_processor(SimpleSpanProcessor(exporter))
caller, worker = [p.get_tracer("trace-lab") for p in providers]
propagator = TraceContextTextMapPropagator()


class Handler(BaseHTTPRequestHandler):
    def log_message(self, *args):
        pass

    def do_GET(self):
        carrier = {key.lower(): value for key, value in self.headers.items()}
        parent = propagator.extract(carrier, context=Context())
        with worker.start_as_current_span("handle" + self.path,
                                         context=parent, kind=SpanKind.SERVER) as span:
            status = 200
            if self.path == "/fail":
                try:
                    raise ValueError("synthetic inventory failure")
                except ValueError as error:
                    span.record_exception(error)
                    span.set_status(Status(StatusCode.ERROR, str(error)))
                    status = 503
            body = b"lab response\n"
            self.send_response(status)
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)


server = HTTPServer(("127.0.0.1", 0), Handler)
thread = Thread(target=server.serve_forever, daemon=True)
thread.start()
base = f"http://127.0.0.1:{server.server_port}"
try:
    with caller.start_as_current_span("request", context=Context()):
        with caller.start_as_current_span("outbound", kind=SpanKind.CLIENT):
            headers = {}
            propagator.inject(headers)
            with urlopen(Request(base + "/ok", headers=headers), timeout=5) as response:
                assert response.status == 200
                response.read()
    try:
        with urlopen(base + "/fail", timeout=5) as response:
            raise AssertionError("Expected HTTP 503")
    except HTTPError as response:
        with response:
            assert response.code == 503
            response.read()
finally:
    server.shutdown()
    thread.join(timeout=5)
    server.server_close()

for provider in providers:
    assert provider.force_flush()
spans = {span.name: span for span in exporter.get_finished_spans()}
assert len(spans) == 4
root, client, received, orphan = [spans[name]
    for name in ("request", "outbound", "handle/ok", "handle/fail")]
assert root.parent is None
assert root.context.trace_id == client.context.trace_id == received.context.trace_id
assert client.parent.span_id == root.context.span_id
assert received.parent.span_id == client.context.span_id and received.parent.is_remote
assert orphan.parent is None and orphan.context.trace_id != root.context.trace_id
assert orphan.status.status_code == StatusCode.ERROR
assert [event.name for event in orphan.events] == ["exception"]
assert received.resource.attributes["service.name"] == "worker"
assert client.resource.attributes["service.name"] == "caller"
print(json.dumps({"exported_spans": len(spans), "linked_spans": 3,
                  "unpropagated_request_started_new_trace": True,
                  "failed_span_status": orphan.status.status_code.name,
                  "exception_events": len(orphan.events)}, indent=2))
for provider in providers:
    provider.shutdown()

The first request creates three linked spans: request, outbound, and handle/ok. The assertions verify one trace identifier, two correct parent links, a remote parent on the server span, and different service names on the two providers. The second request deliberately sends no propagation header. It returns HTTP 503, creates a separate root span, and explicitly records both an exception event and an ERROR status. Four spans are exported altogether.

These are two providers in one Python process communicating over real local HTTP. The exercise verifies propagation and recording behavior, not deployment-wide clock synchronization, collector reliability, or concurrent critical-path analysis. In a deployed service, repeat the boundary checks through the actual ingress and client libraries, then confirm export health and sampling configuration. A useful trace is one whose recorded relationships and limitations are understood before its timeline is used to explain an incident.

Sources