Chapter 7 — Distributed Tracing with OpenTelemetry

A trace is a directed acyclic graph of spans that share a common TraceId. Each span represents one unit of work — an HTTP handler, a database query, a queue publish — and carries a name, a start/end timestamp, a parent reference, attributes, events, status, and a kind. The mental model: a span is to a trace what a stack frame is to a stack trace, except spans cross process boundaries and overlap in time when work happens in parallel.

SpanContext: the wire envelope

• TraceId — 128-bit (32 hex chars), globally unique per trace; must not be all zeros.
• SpanId — 64-bit (16 hex chars), unique within a trace; the parent's SpanId becomes the child's parent_span_id.
• TraceFlags — 8 bits; only bit 0 is defined (01 = sampled).

Analogy: TraceId = conference badge color (everyone shares it), SpanId = individual badge number, TraceFlags = whether the photographer may publish your photo.

Span kinds

The SERVER/CLIENT pairing is what lets Tempo's service-graph processor build a dependency map automatically — without correct kinds, the map is guesswork.

Status, events, links

• Status is UNSET, OK, or ERROR. OTel does not auto-infer status from HTTP codes: a 4xx is generally not an error on a SERVER span (the client sent a bad request), but it is from a CLIENT span's perspective. You must set it deliberately.
• Events are timestamped annotations within a span. recordException(e) adds an exception event with exception.type, exception.message, and exception.stacktrace attributes.
• Links reference other related SpanContexts that are not the strict parent — the right tool for fan-in (one batch span with 1,000 links instead of 1,000 parent references).

Figure 7.1 — Parent-child span tree

A trace only works if every service in the path reads, preserves, and forwards the SpanContext. That cross-process handoff is context propagation, implemented by propagators that inject context into outbound carriers and extract it from inbound carriers.

W3C Trace Context — traceparent

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             |  |                                |                |
             |  |                                |                +-- trace-flags (01 = sampled)
             |  |                                +-- parent span-id (16 hex)
             |  +-- trace-id (32 hex, globally unique per trace)
             +-- version (currently "00")

The optional tracestate header carries up to ~32 vendor-specific key=value entries; leftmost has highest precedence; total ~512 chars; no commas or equals signs in values.

Legacy formats: B3 and Jaeger

Composite propagators let one service emit and accept all three formats simultaneously. On extract, the first propagator that succeeds wins; on inject, every enabled propagator writes its own header. That redundancy is the migration trick: roll out W3C alongside B3/Jaeger, let downstream services read whichever they understand, then drop the legacy formats.

Baggage — cross-cutting request data

baggage: user.id=12345, tenant=acme-corp, feature.checkout_v2=enabled

Security note: untrusted clients can send any baggage they want. Sanitize at edges with an allowlist, strip internal baggage on outbound calls to third parties, and never put secrets, tokens, or PII in baggage.

Figure 7.3 — Composite propagation sequence

Auto-instrumentation will produce spans for every HTTP request and DB call out of the box. The difference between a noisy trace and a debuggable one comes down to names, attributes, error recording, and knowing when not to create a span.

Naming spans for searchability

• HTTP server: route template, not raw URL. GET /users/{id}, not GET /users/12345.
• HTTP client: method plus route or host. POST api.payments.svc.
• Database: operation + target. SELECT users, full statement in db.statement.
• Messaging: <destination> <operation>. orders.created publish.
• Internal: stable verb-noun. validate_cart, compute_shipping_quote.

Test: imagine 10,000 spans — how many distinct names should appear? Tens or low hundreds, not millions. If your span name embeds a UUID, it is too specific.

Attributes vs. events vs. status

Recording exceptions

with tracer.start_as_current_span("charge_payment") as span:
    span.set_attribute("payment.amount_cents", amount)
    try:
        gateway.charge(card, amount)
    except PaymentDeclined as e:
        span.record_exception(e)
        span.set_status(trace.StatusCode.ERROR, "payment declined")
        raise

• Record then re-raise — swallowing without re-raising hides bugs.
• Status ERROR is what backends color red and what Tempo's metrics-generator counts in RED metrics.
• HTTP 4xx is not automatically a server-side error; it usually is on the client side.

Span explosion vs. discipline

The most common mistake is a span per loop iteration. Better options: one span per batch + events for failures; sample inside the loop; use counters for counts; use links for fan-in references.

Rule of thumb: a span should represent a unit of work big enough that you might one day look at it in a UI.

Jaeger and Grafana Tempo dominate the open-source tracing backends. Both ingest OTLP, both render Gantt waterfalls, and both produce RED-style metrics — but they differ in storage and integration.

Jaeger UI

• Search by service, operation, time, duration, tags (http.status_code=500, error=true), or free-form trace-id lookup.
• Trace timeline — the Gantt chart with attributes, events, and stack traces from record_exception.
• Trace graph — nodes/edges of one trace.
• System architecture — aggregated dependency graph.
• Service Performance Monitoring (SPM) — RED metrics, typically via an OTel Collector spanmetrics processor.

Storage: Cassandra, Elasticsearch, or OpenSearch.

Grafana Tempo

Tempo stores spans in object storage (S3/GCS/Azure Blob) and indexes only the TraceId — per-trace lookup is cheap; full-text search is expensive. The bet: most queries come from exemplars (a metric or log line already gives you the TraceId).

metrics_generator:
  processor:
    service_graphs:
      enabled: true
      wait: 10s
      max_items: 10000
      peer_attributes:
        - peer.service
        - db.name
        - messaging.system
    span_metrics:
      enabled: true
      dimensions:
        - http.method
        - http.status_code
        - rpc.system
      include_span_kinds:
        - server
        - consumer

Service maps

An accurate service map requires three things: consistent service.name resource attribute, correct SpanKind, and peer.service (or db.name, messaging.system) on outbound spans so uninstrumented downstreams can still be inferred.

RED via PromQL

rate(tempo_span_calls_total{span_kind="server"}[5m]) by (service_name)

rate(tempo_span_calls_total{span_kind="server", status_code!="OK"}[5m]) by (service_name)

histogram_quantile(0.95,
  sum by (service_name, le) (
    rate(tempo_span_duration_seconds_bucket{span_kind="server"}[5m])
  )
)

Caveats for trace-derived metrics

• Sampling distorts rate. Head sampling at 10% reports ~1/10 of true rate. Tail sampling that preferentially keeps errors/slow traces over-represents errors. Treat trace-derived metrics as a correlation tool; keep direct application metrics as the SLO source of truth.
• Cardinality. Don't add user_id or request_id as span_metrics dimensions — it will crash your TSDB. Stick to bounded labels: service, operation, status, method, coarse path.