Chapter 2: The Three Signals: Metrics, Logs, and Traces in Depth

A metric is a numeric measurement of a system property recorded at a point in time. Metrics are the cheapest of the three signals because the storage system records aggregates, not individual events. A counter incremented one billion times occupies the same space as one incremented ten times — both are just a current value plus a timestamp series.

Think of metrics as the dashboard gauges in a car: speed, RPM, fuel level. They tell you the state of the system at a glance, but they don't tell you why the engine started knocking three miles back.

Counters, gauges, histograms, and summaries

• Counter — monotonically increasing (only goes up, or resets on restart). Examples: http_requests_total, errors_total. Answers "how many?" and "at what rate?" via rate().
• Gauge — can go up or down. Examples: memory_bytes_in_use, queue_depth. Represents instantaneous state.
• Histogram — a distribution stored as cumulative buckets *_bucket{le="..."} plus _sum and _count. Quantiles like p99 are computed server-side in PromQL with histogram_quantile().
• Summary — quantiles computed client-side using a sliding-window algorithm and exposed as {quantile="0.99"} series.

Histogram vs summary — the one trap to remember

The critical difference: summary quantiles cannot be aggregated across instances. Averaging the p99 from each of ten pods does not give a true global p99. You can safely sum only _sum and _count from a summary. Histogram buckets, in contrast, are fully aggregatable — sum() buckets across instances and then call histogram_quantile() to get meaningful service-wide tail latency.

Classic histograms have a drawback: bucket count multiplies cardinality. Native histograms (Prometheus 2.40+) encode dynamic log-spaced buckets inside a single time series, dramatically reducing series count while supporting high resolution.

Cardinality — the silent budget killer

A Prometheus time series is uniquely identified by its metric name plus the set of label key/value pairs:

http_requests_total{method="GET", path="/api/orders", status="200", service="checkout"}

Each unique combination of labels creates a new series. The math is brutal — consider these labels:

• method: 5 (GET, POST, PUT, DELETE, PATCH)
• path: 50 endpoints
• status: 6 (status code groups)
• service: 20
• region: 4

Total potential series: 5 × 50 × 6 × 20 × 4 = 120,000 for one metric. Add user_id with 10,000 unique values and you reach 1.2 billion potential series. This is why veterans say "never label by user ID, request ID, or trace ID." It's also why exemplars exist — they reference a trace ID without making it a series-defining label.

A log is a discrete event record emitted by an application at a specific moment. Where a metric says "23,481 requests in the last minute," a log says "request #382749 from user 42 to /api/orders failed at 10:03:21 with NullPointerException." Logs preserve the individual story that metrics aggregate away.

Structured vs unstructured logging

For decades, application logs looked like this:

2024-05-10T10:00:00Z my-host app: ERROR user 42 not found

This is unstructured logging. Humans can read it; computers struggle. Searching for "all errors involving user 42" forces a downstream tool to parse the string with regex and hope every team used the same format — an information-destruction pipeline.

Structured logging flips this around: the application emits machine-readable key/value records from the start.

{
  "ts": "2024-05-10T10:00:00Z",
  "level": "info",
  "service": "billing",
  "request_id": "abc-123",
  "user_id": 42,
  "msg": "charged credit card",
  "amount": 99.99
}

OpenTelemetry formalizes this with a typed LogRecord schema. Key fields:

• timestamp — when the event occurred (application time).
• observed_timestamp — when the collector saw it (pipeline time).
• severity_number — normalized 1–24 scale: TRACE (1–4), DEBUG (5–8), INFO (9–12), WARN (13–16), ERROR (17–20), FATAL (21–24).
• severity_text — original severity string ("WARN", "Warning", "warn").
• body — main content, typed as AnyValue (string, number, map, or list).
• attributes — structured context (http.method, db.system, ...).
• resource — service/host/cluster metadata shared with traces and metrics.
• trace_id, span_id, trace_flags — first-class trace context fields.

The genius of severity_number is normalization: a backend query severity_number >= 17 means "ERROR or worse" whether the source used Python's logging.ERROR, Java's WARN, Go's log.Error, or .NET's LogLevel.Error.

Figure 2.2: OpenTelemetry LogRecord schema

Indexing strategies and pipelines

A typical pipeline: application writes structured logs to stdout → node agent (Fluent Bit, Vector, OTel Collector) enriches with Kubernetes metadata and ships → aggregator buffers/samples → storage backend indexes for search.

Elasticsearch-style full-text indexing lets you grep any word in any log instantly, but the index can be larger than the data itself. Loki-style label-only indexing keeps a tiny index over a handful of labels (service, pod, namespace) and stores raw log lines compressed; queries grep through chunks — cheaper to operate but slower for arbitrary substring searches.

Why logs alone cannot do root-cause analysis

A checkout request times out. The checkout-service log says "called inventory: timeout after 5s." The inventory-service log says "received call, query took 4.8s." The postgres log says "lock wait." Three lines, three streams, one causal chain — but the chain is implicit. Reconstructing it requires propagating a request ID through every boundary (which most teams botch at least one of), aligning clocks that drift, and guessing the causal order from timestamps that don't capture parent/child relationships.

This is exactly the problem traces solve. Logs complement traces by carrying rich per-event context; they do not replace the causal graph.

A trace is the recorded journey of a single request through a distributed system. If metrics are the dashboard and logs are the event log, traces are the GPS track — not just that the trip happened but the exact route, which segments were slow, and where the detours were.

Trace, span, and span context

• A trace is identified by a 128-bit trace_id (32 hex chars), e.g. 4bf92f3577b34da6a3ce929d0e0e4736. All work performed in service of one logical request shares this ID.
• A span is a named unit of work within the trace with a 64-bit span_id (16 hex chars). Each span has a start, end, name, status, and attributes.
• A span context is the immutable envelope carrying trace_id, current span_id, and trace_flags across process boundaries. The standard wire format is the W3C traceparent header:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
              |       trace_id              |   span_id    |flags

When checkout-service calls inventory-service, it copies its current span context into a traceparent header. The receiver reads it, knows the parent's trace_id and span_id, and starts a child span under the same trace. This is how one trace stitches itself together across N services with no central coordinator.

Span kinds and auto-derived service maps

OpenTelemetry classifies spans by kind:

• INTERNAL — work entirely within one process.
• SERVER — receives an incoming RPC (callee side).
• CLIENT — sends an outgoing RPC (caller side).
• PRODUCER — emits a message to a queue.
• CONSUMER — receives a message from a queue.

A typical HTTP call produces two spans: a CLIENT span on the caller and a SERVER span on the callee, linked by the same trace_id and a parent-child relationship. Because every cross-service call produces a tagged CLIENT/SERVER pair, a backend can build a live service map automatically: services are nodes, observed parent-child cross-service relationships are edges, edge weight = call rate, edge color = error rate or latency.

Figure 2.4: Service map auto-derived from CLIENT→SERVER span pairs

Each signal in isolation is useful; together they become an investigative narrative. The mechanics of correlation — how a dot on a Grafana chart links to a trace, which in turn surfaces the relevant log lines — rely on three standardized hooks.

Exemplars in Prometheus histograms

The naive way to link a latency spike to a slow request is to put trace_id in the metric labels — which blows up cardinality. Exemplars solve this by attaching a few representative trace_id/span_id pointers alongside metric samples without making them series-defining labels.

The OpenMetrics text exposition format appends an exemplar after the sample value, prefixed by #:

http_server_request_duration_seconds_bucket{le="0.5",method="GET",service="api"} 240 1700000100 \
  # {trace_id="4bf92f3577b34da6a3ce929d0e0e4736",span_id="00f067aa0ba902b7"} 1.700000099e+09

Constraints worth memorizing:

• At most one exemplar per series per scrape interval is typically exported.
• Exemplars attach mostly to histogram buckets (great for "find me a trace in this latency bucket") and sometimes to counter increments.
• Exemplars live in a separate storage path from ordinary series in the Prometheus TSDB — specifically to avoid cardinality explosion.

Figure 2.5: Metric-to-trace exemplar drill-down workflow

Trace-to-logs joins via trace_id and span_id

The second leg is trace-to-logs, made possible because the OpenTelemetry LogRecord schema reserves dedicated trace_id and span_id fields at the top level — not buried in attributes. When an application logs inside an active span, the OpenTelemetry logging integration automatically copies the current span's trace_id and span_id into the LogRecord. In Java with log4j2, in Python with the OTel logging instrumentation, in Go with otelslog — the SDK does it for you.

From the trace view, "show logs for this span" is a single derived-field query:

{service="checkout"} |= "4bf92f3577b34da6a3ce929d0e0e4736"

Sampling pitfall: if you head-sample traces at 10% but log unconditionally, 90% of your log trace_ids point to traces that don't exist in Tempo. Use tail-based sampling (decide based on outcomes — errors and slow traces always kept) or ensure the sampling decision propagates early so non-sampled traces don't leave stale IDs in logs.

Unified resource attributes across all signals

The third hook is the OpenTelemetry Resource — a small, fixed set of attributes describing where the telemetry came from, attached to every metric, log, and trace emitted by a process:

service.name = "checkout-service"
service.namespace = "payments"
service.instance.id = "pod-abc123"
service.version = "1.4.2"
deployment.environment = "prod"
k8s.pod.name = "checkout-7f8b9-abcde"
k8s.namespace.name = "payments-prod"
host.name = "ip-10-0-1-42.ec2.internal"
cloud.region = "us-east-1"

Because all three signals share these attributes verbatim, the query "all telemetry for service.name = checkout-service in deployment.environment = prod" returns matching metrics, logs, and traces from a single filter — no vendor-specific tag mapping. Semantic conventions extend this consistency to operation-level attributes (http.request.method, db.system.name, rpc.service) so queries port across backends.

Figure 2.6: Correlation hub — three standardized hooks unifying the signals