Chapter 11: Sampling, Performance, and Cost Control

Sampling is the single most powerful lever you have for controlling telemetry cost and overhead. It is also the most misunderstood. The trick is to sample in a way that preserves what matters — errors, slow traces, unusual tenants — while discarding the redundant majority.

Head-based: ParentBased, TraceIdRatio, AlwaysOn/Off

Head-based sampling makes the keep-or-drop decision at the first service that creates the root span — typically the SDK in your front-door API. Because the decision is made up front, downstream services never have to record, store, or ship the spans of a dropped trace. This is the cheapest possible form of sampling.

The canonical head sampler in OTel is TraceIdRatioBased. It hashes the trace ID to produce a deterministic sample with probability p. Wrapped in ParentBased, child spans honor the parent's decision, ensuring coherent distributed traces:

ParentBased(root=TraceIdRatioBased(0.1))

Extreme samplers exist too: AlwaysOn for dev/staging and AlwaysOff as a kill switch.

Tail-based at the Collector

Tail-based sampling lives in the OTel Collector's tail_sampling processor: SDKs ship every span, the Collector buffers spans by trace ID, waits a decision_wait window (often 5–30 s) for the trace to complete, evaluates policies (error, latency, tenant), then flushes the keepers. The cost is memory:

buffered_bytes ≈ spans/sec × spans/trace × span_size × wait_seconds

At 10,000 spans/sec × 20 spans/trace × 1 KB × 10 s = 2 GB of in-memory spans. Tail sampling also adds 5–30 s of observability latency — not great if SREs depend on live traces for incident detection.

Hybrid head + tail sampling

The common production pattern: light head sample (e.g. ParentBased(TraceIdRatioBased(0.2))) plus tail sampling that further refines the 20% to keep errors and slow traces. Caveat: anything the head sampler drops is gone forever — including errors that lost the dice roll.

Comparison: When to use which

If sampling controls trace cost, cardinality controls metric cost. In a Prometheus-style TSDB the unit of storage is not the metric — it's the unique combination of metric name plus label set. Each unique combination is one time series, with its own memory footprint and storage row.

Add a user_id label to http_requests_total{method, status} with 100,000 users and the math turns multiplicative very quickly.

Identifying high-cardinality labels

The most useful PromQL forensic query is the “top-N by series count”:

topk(20, count by (__name__)({__name__=~".+"}))

For deeper analysis use promtool tsdb analyze, watch the meta-metrics prometheus_tsdb_head_series and prometheus_tsdb_head_series_created_total, and (for Mimir users) the Cardinality Explorer UI — you can even fail CI builds when a new metric crosses, say, 50,000 series.

Allow/deny lists and attribute drops

metric_relabel_configs:
  - source_labels: [uri]
    regex: ".+"
    action: labeldrop

  - regex: "user_id|session_id|request_id"
    action: labeldrop

  - source_labels: [path]
    regex: "/api/v1/users/[0-9]+/(.*)"
    target_label: path
    replacement: "/api/v1/users/:id/$1"
    action: replace

That last rule — collapsing /users/12345/orders into /users/:id/orders — is one of the most useful tricks in the playbook.

Aggregating away unbounded dimensions

groups:
  - name: myapp_aggregates
    interval: 30s
    rules:
      - record: myapp:http_request_duration_seconds_bucket:service
        expr: sum by (service, le) (myapp_http_request_duration_seconds_bucket)

Keep the high-cardinality raw metric at short retention (2 h) for live debugging, and the rolled-up recording rule for 90 days at a tiny fraction of the storage. Long-term: never emit unbounded labels (user_id, session_id, email) on metrics — those belong on traces or logs.

The third lever is the cost of generating telemetry — the CPU and memory the instrumentation itself consumes inside your application. Usually small if sampling and cardinality are under control, but worth measuring.

CPU and memory cost of SDK instrumentation

Java auto-instrumentation (the OTel Java Agent) is the heaviest common case because it uses bytecode instrumentation. The Elastic EDOT Java benchmark on a sample JVM service:

Expect 1–5% additional CPU for most well-tuned JVM services and tens of MB of extra heap. The worst case from community benchmarks is up to ~20% CPU and ≥0.5 ms latency per instrumented hop at high QPS without sampling — driven mostly by GC pressure.

Go auto-instrumentation is library-based, so there's no startup penalty and no class-loading hit. Typical Go OTel overhead is low single-digit CPU percent with a few to tens of MB of additional RSS.

Batching, async export, and buffer tuning

The BatchSpanProcessor (BSP) decouples span generation from span export. Spans go into an in-memory queue, a background thread pulls them in batches, and the exporter ships them. Tuning knobs:

Back-pressure and benchmarking

Watch otel_sdk_span_processor_dropped_spans — a sustained nonzero rate means you're losing trace data and need to sample harder or grow the queue. To make capacity decisions, benchmark your own workload: baseline without instrumentation, then enable it at the intended sample rate, then drive 2–3× peak load to find the breakpoint. Plan capacity with +5–20% CPU headroom for Java, low single-digit for Go.

Sampling, cardinality, and overhead controls are tactical. The strategic layer is architecture: where in the pipeline you do what work, how telemetry flows from edge to backend, and how long each tier retains data. Think cold-chain logistics for telemetry: fresh data needs fast/expensive storage, ageing data shifts to progressively cheaper tiers.

Metric aggregation at the edge

The further telemetry travels before it is reduced, the more expensive it becomes. Do reduction as close to the source as possible:

• In the application: SDK views aggregate buckets, drop attributes, convert deltas before export.
• In a sidecar or node agent: per-node OTel Collector / Prometheus Agent does first-round relabel and sample.
• In a regional collector tier: consolidates across nodes, applies tail sampling, forwards to central backend.

Each tier reduces volume by 2–10×.

Log volume reduction

• Structured logging only — JSON, so the pipeline can filter and route without regex.
• Severity-based routing — DEBUG/INFO short retention, ERROR+ to always-on alerting.
• Sample DEBUG/INFO in production at 1–10%, just like traces.
• Drop or hash unbounded fields — deduplicate stack traces by hash.
• Convert logs to metrics — db_timeout_total{service="orders"} beats ten thousand log lines.
• Suppress duplicate spam with rate-limiting processors.

Tiered storage and retention

Reasonable default retention by pillar:

The architectural lever that ties this together is back-pressure handling. Every component must have a documented behavior under overload — drop, buffer, downsample, or block. The right default is “drop with a metric” so the application stays healthy and operators see the overload via clear telemetry-about-telemetry counters.