Cloud-native Observability with Prometheus and OpenTelemetry

An intermediate, hands-on guide to instrumenting, collecting, querying, and operationalizing telemetry in cloud-native systems using Prometheus and OpenTelemetry.

Table of Contents

• Chapter 1: Foundations of Cloud-native Observability
• Chapter 2: The Three Signals: Metrics, Logs, and Traces in Depth
• Chapter 3: Prometheus Architecture and Data Model
• Chapter 4: PromQL: Querying Time-Series Data
• Chapter 5: OpenTelemetry Architecture: API, SDK, and Collector
• Chapter 6: Instrumentation: Manual, Automatic, and Zero-Code
• Chapter 7: Distributed Tracing with OpenTelemetry
• Chapter 8: Metrics Pipeline: Bridging OpenTelemetry and Prometheus
• Chapter 9: Logs, Events, and Cross-Signal Correlation
• Chapter 10: The OpenTelemetry Collector in Depth
• Chapter 11: Sampling, Performance, and Cost Control
• Chapter 12: SLOs, Alerting, and Operational Excellence

Chapter 1: Foundations of Cloud-native Observability

Learning Objectives

• Distinguish between monitoring and observability, and explain why cloud-native systems require the latter to operate safely at scale.
• Identify the three pillars of observability — metrics, logs, and traces — and articulate the role each plays during incident response.
• Describe the cloud-native operational challenges (ephemeral workloads, distributed services, polyglot stacks, service meshes) that motivate Prometheus and OpenTelemetry as foundational tooling.

From Monitoring to Observability

Definition of observability borrowed from control theory

The word observability originated in control theory, where it describes the degree to which the internal state of a system can be inferred from its external outputs. Modern software borrows that idea: a system is observable when its emitted telemetry — metrics, logs, and traces — is rich enough that engineers can reconstruct what is happening inside it, even for situations no one anticipated [Source: https://opentelemetry.io].

This stands in contrast to monitoring, which is fundamentally about checking known signals against predefined thresholds. Monitoring answers questions you wrote down ahead of time: “Is checkout-service returning too many 5xx errors?” or “Is CPU usage on node node-3 above 90%?” [Source: https://prometheus.io]. It is a detection mechanism — a tripwire — built around failure modes you already understand.

A useful analogy: monitoring is like the dashboard of a car. Speed, fuel level, engine temperature, and a few warning lights cover the most common failure cases. Observability is more like having a full diagnostic port plus a mechanic’s toolkit on hand. When the car behaves strangely in a way the dashboard cannot explain — a vibration only at 73 mph in the rain — you need the ability to probe, query, and correlate signals that were never wired up to a warning light.

Key Takeaway: Monitoring tells you that something is wrong by checking predefined signals; observability gives you the raw material to figure out what and why by enabling ad-hoc exploration of correlated telemetry. Both are necessary, but observability is what makes complex cloud-native systems debuggable.

Known-unknowns vs unknown-unknowns

The classic framing for this distinction is known unknowns versus unknown unknowns. A known unknown is a failure mode you can name in advance: “the database might run out of connections,” “the disk might fill up,” “a pod might enter CrashLoopBackOff.” For each of these, you can pre-build a metric, threshold, and alert [Source: https://prometheus.io].

Unknown unknowns are the failures that nobody on the team has imagined yet. Consider a real-world scenario: sporadic 500 errors appear from checkout-service only during peak traffic, only for the /checkout/card endpoint, only on version v2, and only for tenant_type="enterprise". No engineer wrote an alert for that combination of conditions. With pure monitoring you might see only a vague error-rate spike. With observability, you can slice the metric by labels, pivot to the traces matching the failing requests, and finally jump to the logs for the specific failing spans to discover that a misconfigured PAYMENT_TIMEOUT=200ms environment variable was deployed in v2 [Source: https://opentelemetry.io].

That investigative path — metric → trace → log — is the practical essence of observability. It lets you discover the question you didn’t know to ask.

Key Takeaway: Monitoring is sufficient when you can enumerate failure modes in advance. Distributed cloud-native systems generate too many novel failure combinations for that approach, which is why observability — high-cardinality, correlated, queryable telemetry — has become a baseline requirement rather than a luxury.

Why dashboards alone are insufficient in distributed systems

Dashboards are fundamentally a summarization tool: they aggregate raw telemetry into a small number of charts chosen ahead of time. That works beautifully when the number of important questions is small and stable. It breaks down when the number of meaningful slices through your data explodes.

In a microservice deployment with 30 services, 5 versions in flight at any moment, dozens of tenants, multiple regions, feature flags, and Kubernetes-supplied attributes (namespace, deployment, pod, node), the dimensional space of interesting views runs into the millions. No fixed dashboard can pre-render every relevant slice [Source: https://www.cncf.io].

Figure 1.1: Monitoring dashboards vs. observability query interfaces

flowchart LR
    subgraph Monitoring["Monitoring (pre-built dashboards)"]
        direction TB
        G1["Gauge: CPU %"]
        G2["Gauge: Error rate"]
        G3["Gauge: Latency p99"]
        G4["Fixed thresholds<br/>and alerts"]
    end
    subgraph Observability["Observability (ad-hoc query interface)"]
        direction TB
        Q["Free-form query<br/>by service, version,<br/>tenant, region, endpoint"]
        Q --> R1["Slice metrics"]
        Q --> R2["Pivot to traces"]
        Q --> R3["Drill into logs"]
    end
    Monitoring -. "answers known questions" .-> Known["Known unknowns"]
    Observability -. "discovers new questions" .-> Unknown["Unknown unknowns"]

A second problem is that dashboards summarize signals, but rarely correlate across signals. When an alert fires on a latency metric, the human operator must mentally bridge from “the chart looks bad” to “let me find a representative trace” to “let me read the logs for that trace.” Observability tooling closes that loop with exemplars and trace IDs that make the pivot a single click rather than a sprawling investigation.

Key Takeaway: Dashboards are a useful summary surface, but they cannot anticipate every question a distributed system will provoke. Observability shifts the workflow from “look at pre-built charts” to “ask arbitrary questions of correlated telemetry.”

The Three Pillars: Metrics, Logs, and Traces

Strengths and weaknesses of each signal

The three pillars of observability — metrics, logs, and traces — are complementary because each has different strengths along the axes of cardinality, cost, and temporal granularity.

Metrics are numeric time series: counters, gauges, and histograms sampled at fixed intervals. They are cheap, summarizable, and ideal for alerting. A typical Kubernetes metric looks like http_requests_total{service="api",status="500"} or a latency histogram like request_duration_seconds_bucket{le="0.1",service="checkout"} [Source: https://prometheus.io]. Their weakness is that they are inherently aggregated: you trade per-event detail for compactness.

Logs are discrete events — usually structured JSON in cloud-native systems — emitted by applications and infrastructure. In Kubernetes this includes application logs from containers, kubelet logs, API server logs, and ingress controller logs. Logs are excellent for capturing what exactly happened in a single moment, including exception stack traces, parameter values, and business context. Their weakness is volume: at scale, indexing and storing every log line is expensive [Source: https://opentelemetry.io].

Traces are end-to-end records of a request as it flows through multiple services. A trace is composed of spans, each representing a unit of work in one service, with parent-child relationships representing the call hierarchy. Trace context is propagated across processes via HTTP headers like W3C traceparent. Traces are the only signal type that natively captures the structure of a distributed request. Their weakness is also volume, which is why traces are usually sampled rather than collected exhaustively [Source: https://opentelemetry.io].

Figure 1.2: The three pillars and their connective tissue

flowchart TD
    Obs["Observability"]
    Obs --> M["Metrics<br/>aggregated numeric<br/>time series"]
    Obs --> L["Logs<br/>discrete structured<br/>events"]
    Obs --> T["Traces<br/>causal request flow<br/>across services"]
    M -- "exemplars<br/>(metric point to trace)" --> T
    T -- "trace_id in log lines" --> L
    L -- "span_id back to trace" --> T
    M -. "resource attributes<br/>(k8s.namespace, service, version)" .- L
    M -. "resource attributes" .- T
    L -. "resource attributes" .- T

Key Takeaway: No single pillar is sufficient on its own. Metrics tell you that something is wrong at scale; logs tell you what exactly happened in one event; traces tell you where in the call chain the problem lives. A mature observability stack uses all three deliberately.

Correlation between signals via exemplars and trace IDs

The leap from “three separate pillars” to “one observability fabric” happens when the signals are correlated. Two mechanisms make this possible.

First, exemplars are pointers attached to metric data points that link a specific bucket of a histogram (for example, a latency point at 2.3 seconds) to one concrete trace that contributed to that bucket. When you see a tall bar in a latency histogram, an exemplar lets you click straight to a representative slow trace, rather than guessing which of millions of requests it was [Source: https://prometheus.io].

Second, trace IDs embedded in structured logs allow you to jump from any log line to the full trace it belongs to — and from any span in a trace to the exact log lines that span emitted. When a trace shows a span failing with a TimeoutError, you can pivot directly to that span’s logs to see parameters, exception details, and surrounding context [Source: https://opentelemetry.io].

Resource attributes such as k8s.namespace.name, k8s.deployment.name, and k8s.pod.name tie all three pillars to the underlying Kubernetes workload, so a single label can pivot you from a metric chart to a log query to a trace search without losing context.

Key Takeaway: Exemplars connect metrics to traces; trace IDs in logs connect logs to traces; resource attributes connect everything to the workload. Together these three correlations let an operator move fluidly between pillars during an incident.

Cardinality, sampling, and cost trade-offs

Cardinality is the number of unique combinations of labels in a telemetry dataset. It is the single most important operational concept in cloud-native observability, because each pillar handles it differently — and getting it wrong is expensive [Source: https://prometheus.io].

In a Prometheus-style metrics system, every unique combination of label values creates a new time series. A counter like http_requests_total{service, endpoint, status, pod} with 10 services × 50 endpoints × 5 statuses × 200 pods produces 500,000 time series. Add user_id to the label set and the count explodes to millions, causing memory pressure, slow queries, and even out-of-memory crashes on the Prometheus server [Source: https://prometheus.io].

The practical rule is:

• Metrics should carry low-cardinality, operationally meaningful labels: service, namespace, deployment, version, region, endpoint (templated, not raw), status. Avoid user_id, request_id, raw URLs, and pod UIDs.
• Logs are built to handle high-cardinality data — request IDs, user IDs, dynamic fields — because they are indexed and stored differently than time series. Keep them structured so selected fields can be indexed.
• Traces are inherently high-cardinality (every span has many attributes) and are usually sampled: only a fraction of requests have their full trace retained.

Sampling strategies vary. Head-based sampling decides at the start of a request whether to keep it; it is cheap but blind to errors. Tail-based sampling buffers spans and decides after the request completes, allowing you to retain 100% of errors and slow traces while sampling normal traffic at a low rate.

Key Takeaway: Cardinality is a first-class design concern. Push low-cardinality dimensions to metrics, high-cardinality detail to logs and traces, and sample aggressively to control cost. Misplacing a label can quietly bankrupt a metrics backend.

Cloud-native Operational Context

Kubernetes, containers, and ephemeral workloads

Traditional monitoring was built for a world of long-lived hosts with stable identities: hostnames, IPs, and static service inventories. Kubernetes turns those assumptions upside down. Pods are short-lived by design — they are created and destroyed in seconds due to autoscaling, rolling deployments, crash loops, and health-probe failures. Pod names and IPs are not stable identifiers; each restart can produce new ones [Source: https://kubernetes.io].

Several practical pathologies follow from this. A dashboard keyed by pod name becomes unreadable as identities churn every few minutes. Historical time series for a specific pod are meaningless once that pod is gone. Alerts that fire on a pod that has already terminated produce “resource not found” errors when an operator clicks through. Batch jobs or crashing pods may exist for only seconds — too briefly for a legacy agent to discover them at all [Source: https://kubernetes.io].

Figure 1.3: Rolling deployment re-keys metrics from pods to the Deployment

flowchart LR
    subgraph T0["t=0: v1 steady state"]
        P1A["pod checkout-v1-a"]
        P1B["pod checkout-v1-b"]
        P1C["pod checkout-v1-c"]
    end
    subgraph T1["t=1: rolling update"]
        P1B2["pod checkout-v1-b"]
        P2A["pod checkout-v2-a"]
        P2B["pod checkout-v2-b"]
    end
    subgraph T2["t=2: v2 steady state"]
        P2A2["pod checkout-v2-a"]
        P2B2["pod checkout-v2-b"]
        P2C["pod checkout-v2-c"]
    end
    T0 --> T1 --> T2
    T0 --> Q["Stable query:<br/>sum by (service, version)<br/>(rate(http_requests_total))"]
    T1 --> Q
    T2 --> Q
    Q --> Dash["Continuous series<br/>keyed by Deployment +<br/>version, not pod name"]

The cloud-native response is to treat workloads as the unit of observability, not pods or hosts. Aggregate metrics across all pods in a Deployment using labels like service, namespace, and version. Use Prometheus’s built-in Kubernetes service discovery to find scrape targets dynamically via the Kubernetes API rather than maintaining a static target list. Send logs from containers via a node-level DaemonSet collector (Fluent Bit, Vector) to a centralized store so that records survive the pod that created them [Source: https://prometheus.io].

A concrete example: during a rolling deployment from v1 to v2, you should be able to issue the query sum(rate(http_requests_total{service="checkout", version="v2"}[5m])) and trust the answer even though the underlying pods that contributed to it may all have been replaced during the query window.

Key Takeaway: Ephemeral pods break host-centric monitoring. Cloud-native observability aggregates at the workload level using stable labels (deployment, namespace, version) and relies on dynamic Kubernetes service discovery rather than static target lists.

Microservices and the death of the call stack

In a monolith, debugging a user request usually means reading a single process’s stack trace. Almost all interesting work happens in-process, in a single language runtime, so a JVM profiler or .NET APM agent can see the whole transaction.

Microservices destroy this comfortable assumption. A single user request can traverse dozens of services, each potentially written in a different language (Go, Java, Node.js, Python, Rust, .NET) and communicating over multiple protocols (HTTP, gRPC, Kafka, SQS, gRPC streaming) [Source: https://opentelemetry.io]. There is no single stack trace — there is only a distributed call hierarchy that lives in no one process’s memory.

Traditional APM tools were designed around proprietary agents tightly coupled to specific runtimes. They cannot reliably stitch a request together when it crosses language boundaries or hops onto a message queue. The result is that engineers see traces only within one service, and cross-team debugging becomes guesswork: “Is it my service or theirs?” without a shared trace ID [Source: https://opentelemetry.io].

The cloud-native answer is distributed tracing built on open standards:

• Use OpenTelemetry SDKs in every language so all services produce traces in the same data model.
• Adopt W3C Trace Context (traceparent, tracestate) so trace IDs propagate consistently across HTTP, gRPC, and (where possible) message queues.
• Configure HTTP clients, gRPC interceptors, and message producers/consumers to automatically inject and extract trace context — engineers should not have to remember to do this manually.

Key Takeaway: The call stack does not span service boundaries in a microservices architecture. Distributed tracing — with vendor-neutral SDKs and W3C Trace Context propagation — is the only practical mechanism for reconstructing a request’s full path across polyglot services.

Service mesh, sidecars, and infrastructure telemetry

A service mesh like Istio, Linkerd, or Consul Connect injects a sidecar proxy (commonly Envoy) next to each application pod. Critical request behaviors — mTLS termination, retries, timeouts, circuit breakers, traffic splitting, routing — execute in the proxy, not in the application code [Source: https://istio.io].

This creates an observability gap for any tool that only watches application processes. The app may report a healthy error rate while the mesh is silently retrying 503 responses, throttling traffic, or failing TLS handshakes. A misconfigured destination rule or an aggressive outlier-detection policy can degrade user experience without ever touching application metrics [Source: https://istio.io].

The implication is that mesh proxies must be treated as first-class observability targets. Scrape Envoy stats endpoints alongside application metrics. Collect mesh access logs to see per-request behavior, including retries and route decisions. Combine application spans with mesh spans (via OpenTelemetry or Envoy tracing) so that an end-to-end trace shows exactly where time was spent — in the app, in the sidecar, or on the wire between them.

Figure 1.4: Application and sidecar emit separate, correlated telemetry streams

flowchart LR
    subgraph Pod["Kubernetes Pod"]
        App["Application container<br/>(business logic)"]
        Envoy["Envoy sidecar<br/>(mTLS, retries, routing)"]
        App <-->|"localhost"| Envoy
    end
    App -->|"app metrics<br/>(/metrics)"| Prom["Prometheus"]
    App -->|"app traces<br/>(OTLP spans)"| Backend["Tracing backend<br/>(Jaeger / Tempo)"]
    Envoy -->|"mesh metrics<br/>(Envoy stats)"| Prom
    Envoy -->|"mesh access logs"| Logs["Log backend<br/>(Loki)"]
    Envoy -->|"mesh spans"| Backend
    Prom --> Graf["Grafana<br/>correlated view"]
    Backend --> Graf
    Logs --> Graf

Beyond the mesh, cloud-native observability must cover infrastructure telemetry as well: kubelet and control-plane metrics from kube-state-metrics, node-level metrics from the node exporter, and Kubernetes events that describe pod restarts, evictions, and scheduling decisions. Correlating these with application telemetry is what lets you say, “the latency spike at 03:14 UTC coincided with the autoscaler evicting three pods from node-7.”

Key Takeaway: Modern request behavior is distributed across application code, sidecar proxies, and Kubernetes control-plane components. Cloud-native observability must instrument all three layers — and correlate them through shared labels and trace IDs — to give a complete picture of what happened.

The CNCF Observability Landscape

Prometheus as the de-facto metrics standard

Prometheus is the dominant metrics monitoring system in the CNCF ecosystem. It is a CNCF Graduated project — among the earliest to reach that maturity tier, alongside Kubernetes itself [Source: https://www.cncf.io]. Graduation signals widespread, production-grade adoption, strong governance, and a healthy ecosystem of integrations.

Architecturally, Prometheus is a pull-based system. It expects services to expose HTTP /metrics endpoints in the Prometheus exposition format (or the equivalent OpenMetrics format) and scrapes them on a schedule. The scraped data is stored in a purpose-built time-series database, and engineers query it with PromQL, a domain-specific language designed for metrics analysis. Alerts are evaluated as PromQL rules and routed through the Alertmanager component [Source: https://prometheus.io].

Prometheus is metrics-only. It does not natively handle logs or traces — and that single-responsibility focus is part of why it is so durable. Its strengths are:

• Fast, local metrics scraping inside a Kubernetes cluster.
• Real-time evaluation of alert rules.
• A mature ecosystem of exporters (for databases, queues, hardware, and cloud services) and Kubernetes-native integrations like ServiceMonitor resources from the kube-prometheus-stack.
• A query language (PromQL) that most SREs in cloud-native shops already know.

For scaling beyond a single Prometheus instance, the ecosystem provides compatible long-term storage and federation systems — Thanos, Cortex, Mimir, and VictoriaMetrics — all of which speak PromQL and the Prometheus remote-write protocol.

Key Takeaway: Prometheus is the de-facto metrics standard in cloud-native systems: a CNCF Graduated, pull-based metrics database with PromQL, Alertmanager, and a vast exporter ecosystem. It excels at metrics — and only metrics.

OpenTelemetry as the vendor-neutral instrumentation standard

OpenTelemetry (often abbreviated OTel) is the CNCF’s answer to the historical chaos of proprietary, vendor-specific instrumentation. It is also a CNCF Graduated project as of 2025, reflecting the maturity of its specifications, SDKs, and Collector [Source: https://opentelemetry.io].

OpenTelemetry’s scope is fundamentally different from Prometheus’s. It is not a backend; it is an instrumentation and pipeline standard. Its components are:

• SDKs in every major language (Go, Java, Python, Node.js, Rust, .NET, Ruby, and more) for emitting traces, metrics, and logs.
• Auto-instrumentation packages that wrap common libraries (HTTP clients, gRPC, database drivers, message queues) so applications can be instrumented with minimal code changes.
• A unified data model and semantic conventions so that attributes like http.route, k8s.namespace.name, and service.version mean the same thing across languages and signals.
• The OpenTelemetry Collector, a vendor-neutral process that receives, processes, and exports telemetry — deployable as a sidecar, DaemonSet, or central gateway.
• OTLP (OpenTelemetry Protocol), the standard wire protocol for traces, metrics, and logs, transported over gRPC or HTTP [Source: https://opentelemetry.io].

The vendor-neutrality argument is OpenTelemetry’s signature value. Instrument once with OpenTelemetry, and the choice of backend becomes a configuration decision rather than a code rewrite. Migrating from one tracing vendor to another — or running multiple in parallel — requires changes only in Collector pipelines, not in application source [Source: https://opentelemetry.io].

Key Takeaway: OpenTelemetry is the vendor-neutral instrumentation standard for cloud-native telemetry. It provides SDKs, auto-instrumentation, semantic conventions, a Collector, and the OTLP wire protocol — covering all three pillars (traces, metrics, logs) with one consistent model.

Backends: Grafana, Jaeger, Tempo, Loki, Mimir, and commercial vendors

Once telemetry is instrumented (via OpenTelemetry) and metrics are scraped or collected (via Prometheus or the Collector), it has to land somewhere. The CNCF backend ecosystem in 2025 is rich and specialized:

Beyond the open-source landscape, every major commercial observability vendor — Datadog, New Relic, Splunk, Honeycomb, Dynatrace, Chronosphere, Lightstep, and others — natively ingests OTLP and most also support Prometheus remote-write [Source: https://opentelemetry.io]. This is the practical payoff of vendor neutrality: a team can move between open-source and SaaS backends, or even run hybrid configurations, without rewriting instrumentation.

The dominant 2025 pattern is hybrid:

• Instrument with OpenTelemetry SDKs in application code.
• Collect and route with the OpenTelemetry Collector deployed as a DaemonSet or gateway.
• Store metrics in Prometheus (and a long-term Prometheus-compatible backend like Mimir or VictoriaMetrics for retention).
• Store traces in Jaeger or Tempo.
• Store logs in Loki, OpenSearch, or a commercial backend.
• Visualize and alert in Grafana (or a vendor UI), with PromQL for metrics queries and trace/log queries cross-linked via shared IDs.

Figure 1.5: End-to-end cloud-native observability stack

flowchart LR
    Apps["Applications<br/>(Go, Java, Python, ...)"] --> SDK["OpenTelemetry SDK<br/>+ auto-instrumentation"]
    SDK -->|"OTLP (gRPC/HTTP)"| Col["OpenTelemetry Collector<br/>(DaemonSet / gateway)"]
    Scrape["Prometheus scrape<br/>(/metrics endpoints)"] --> Prom["Prometheus<br/>(local TSDB + PromQL)"]
    Apps -. "expose /metrics" .-> Scrape
    Col -->|"metrics<br/>(remote-write)"| Mimir["Mimir / Thanos /<br/>VictoriaMetrics<br/>(long-term metrics)"]
    Prom -->|"remote-write"| Mimir
    Col -->|"traces"| Tempo["Tempo / Jaeger<br/>(trace storage)"]
    Col -->|"logs"| Loki["Loki / OpenSearch<br/>(log storage)"]
    Prom --> Graf["Grafana<br/>(dashboards + alerts)"]
    Mimir --> Graf
    Tempo --> Graf
    Loki --> Graf
    Prom --> AM["Alertmanager"]

In this arrangement Prometheus and OpenTelemetry are complementary, not competitors. As the CNCF SIG Observability community puts it in spirit: use OpenTelemetry for how you instrument and move telemetry; use Prometheus for where metrics live and how you query and alert on them [Source: https://www.cncf.io].

Key Takeaway: The 2025 CNCF observability stack pairs OpenTelemetry (for instrumentation and routing) with Prometheus (for metrics storage and PromQL alerting), feeding specialized backends — Jaeger or Tempo for traces, Loki for logs, Grafana for visualization — and remaining portable between open-source and commercial vendors through OTLP.

Chapter Summary

Observability is a property of a system: the degree to which an operator can infer its internal state from its external outputs. In cloud-native environments — where workloads are ephemeral, requests cross dozens of services in different languages, and infrastructure behavior lives partially in sidecar proxies — that property must be designed in deliberately. Traditional monitoring, which checks predefined signals against thresholds, remains valuable for catching known failure modes but cannot explain the novel, unknown-unknown incidents that distributed systems routinely produce.

The three pillars — metrics, logs, and traces — are the raw materials of observability, each with different cardinality tolerance and cost characteristics. Metrics are low-cardinality time series ideal for alerting and SLOs; logs are high-cardinality discrete events ideal for detailed per-request context; traces are end-to-end records of a request’s path across services. The killer feature is correlation: exemplars link metrics to traces, trace IDs in structured logs link logs to traces, and Kubernetes resource attributes tie everything to the underlying workload, letting an operator pivot from “this chart looks bad” to “this exact log line in this exact span” in seconds.

In the CNCF ecosystem of 2025, two graduated projects anchor the stack. Prometheus provides metrics scraping, the PromQL query language, and Alertmanager — the battle-tested metrics brain. OpenTelemetry provides language SDKs, semantic conventions, the Collector, and the OTLP wire protocol — the vendor-neutral nervous system for all three pillars. The dominant pattern is to instrument with OpenTelemetry, store metrics in Prometheus (or a Prometheus-compatible long-term store), and route traces and logs to specialized backends like Jaeger, Tempo, and Loki, with Grafana stitching it all together. The rest of this book builds out the practical mechanics of that stack.

Key Terms

Chapter 2: The Three Signals: Metrics, Logs, and Traces in Depth

Learning Objectives

• Compare the data models of metrics, logs, and traces and identify when each is the right tool for a given investigative question.
• Explain how exemplars and trace IDs correlate signals into a single investigative narrative across Prometheus, OpenTelemetry, and trace backends like Tempo or Jaeger.
• Quantify the storage and cardinality cost of each signal type, including the trade-offs of Prometheus histograms versus summaries and classic versus native histograms.

Chapter 1 introduced observability as a property that emerges from three complementary signals: metrics, logs, and traces. That framing is useful, but it can be misleading if you treat the three as interchangeable buckets of “telemetry.” They are not. Each signal has a distinct data model, distinct ingestion economics, and distinct questions it can answer well. Choosing the wrong signal for a question is like trying to measure room temperature with a video camera — you can sort of do it, but you’re paying enormous storage and processing costs for information that a $5 thermometer would deliver instantly.

This chapter goes one level deeper. We dissect the wire format of each signal, examine the cardinality math that drives cost, and then show how exemplars and W3C Trace Context turn three independent data streams into a single navigable investigative narrative.

Figure 2.1: The three signals and their correlation hooks

graph TD
    M["Metrics<br/>counts &amp; aggregates<br/>'how many / how fast'"]
    L["Logs<br/>discrete events<br/>'what happened'"]
    T["Traces<br/>causal span tree<br/>'why was it slow'"]
    M -- "exemplars<br/>(trace_id pointers)" --- T
    T -- "trace_id / span_id<br/>in LogRecord" --- L
    M -- "shared Resource<br/>(service.name, k8s.*)" --- L
    Hub(("Unified<br/>investigation"))
    M --> Hub
    L --> Hub
    T --> Hub

Metrics — Numbers Over Time

A metric is a numeric measurement of a system property recorded at a point in time. Metrics are the cheapest of the three signals because the storage system records aggregates, not individual events. A counter that has been incremented one billion times occupies the same space as one that has been incremented ten times — both are just a current value plus a timestamp series.

Think of metrics as the dashboard gauges in a car: speed, RPM, fuel level. They tell you the state of the system at a glance, but they don’t tell you why the engine started knocking three miles back.

Counters, gauges, histograms, and summaries

Prometheus, which has become the de facto standard for cloud-native metrics, defines four core instrument types:

• Counter — a monotonically increasing value that only goes up (or resets to zero on process restart). Examples: http_requests_total, errors_total, bytes_sent_total. Counters answer “how many?” and “at what rate?” via PromQL’s rate() function.
• Gauge — a value that can go up or down. Examples: memory_bytes_in_use, queue_depth, temperature_celsius. Gauges represent instantaneous state.
• Histogram — a distribution recorded as a set of cumulative buckets. Exposes *_bucket{le="0.1"}, *_bucket{le="0.5"}, etc., plus _sum and _count. Quantiles such as p90 and p99 are computed server-side in PromQL using histogram_quantile() over these buckets [Source: https://community.openai.com/t/how-to-confirm-that-you-got-the-correct-value-from-a-text-other-than-repeating-the-same-prompt-over-and-over/922371].
• Summary — a distribution where the client process computes quantiles locally (using a sliding-window algorithm like CKMS) and exposes them directly as series like request_duration_seconds{quantile="0.99"} [Source: https://blog.codinghorror.com/the-problem-with-logging/].

The histogram-versus-summary choice is one of the most consequential decisions in Prometheus instrumentation, and it trips up nearly every team at least once. The critical difference: summary quantiles cannot be aggregated across instances. Averaging the p99 from each of your ten pods does not give you the true global p99. You can safely aggregate only _sum and _count from a summary. Histogram buckets, in contrast, are fully aggregatable — you can sum() buckets across instances, zones, or services and then call histogram_quantile() on the aggregate to get a meaningful service-wide tail latency [Source: https://natesnewsletter.substack.com/p/context-windows-are-a-lie-the-myth].

The practical rule: for SLO-critical latency where you need fleet-wide p99 and the ability to choose new quantiles later without redeploying code, use histograms. For low-cardinality, per-process diagnostics where you only ever care about one box at a time, summaries are acceptable [Source: https://magazine.sebastianraschka.com/p/llm-evaluation-4-approaches].

Classic histograms have a well-known drawback: bucket count multiplies cardinality. Native histograms, introduced in Prometheus 2.40, encode dynamic log-spaced buckets inside a single time series using a binary format, dramatically reducing series count while supporting high resolution [Source: https://pubmed.ncbi.nlm.nih.gov/help/]. If your stack supports them end-to-end (client library, server, remote storage), they are usually the better default for new metrics.

Time-series labels and dimensionality

A Prometheus time series is uniquely identified by its metric name plus the set of label key-value pairs:

http_requests_total{method="GET", path="/api/orders", status="200", service="checkout"}

Each unique combination of labels creates a new time series with its own storage allocation. This is the cardinality of a metric, and it is the single biggest cost driver in any Prometheus deployment.

The math is brutal. Suppose http_requests_total carries five labels with these cardinalities:

• method: 5 (GET, POST, PUT, DELETE, PATCH)
• path: 50 endpoints
• status: 6 (2xx, 3xx, 4xx grouped)
• service: 20
• region: 4

Total potential series: 5 × 50 × 6 × 20 × 4 = 120,000 series for one metric. Add a user_id label with even 10,000 unique values and you have 1.2 billion potential series. This is why veterans say “never label by user ID, request ID, or trace ID.” It’s also why exemplars exist — they let you reference a trace ID without making it a series-defining label.

A useful analogy: labels are dimensions of a spreadsheet. Each dimension multiplies the number of cells. A 2D sheet of methods × paths is manageable; a 7D sheet quickly exceeds the heat death of the universe.

Aggregation, downsampling, and retention

Once metrics are in the time-series database, the next concern is keeping them around without going broke. Prometheus by default scrapes every 15 seconds and stores raw samples for ~15 days. For longer retention, the standard pattern is:

1. Recording rules evaluate expensive PromQL queries periodically and store the result as a new, cheaper time series. For example, instance:cpu_usage:rate5m is far cheaper to query a year of than re-aggregating raw CPU samples each time.
2. Remote write ships data to long-term storage backends like Thanos, Cortex, or Mimir, which handle compaction and downsampling.
3. Downsampling reduces resolution as data ages: raw 15-second samples become 5-minute averages after 30 days and 1-hour averages after a year.

For histograms, downsampling is more nuanced. Classic histogram buckets compose cleanly — sum the buckets, then compute the quantile. Native histograms can merge layouts more flexibly. Summaries cannot be downsampled meaningfully at all, because the quantile series are already lossy projections.

Key Takeaway: Metrics are cheap, aggregated, and ideal for “how many” and “how fast” questions. The cost is dominated by cardinality — label combinations multiply storage — and the histogram-versus-summary choice determines whether you can meaningfully ask fleet-wide tail-latency questions.

Logs — Structured Events

A log is a discrete event record emitted by an application at a specific moment. Where a metric says “23,481 requests in the last minute,” a log says “request #382749 from user 42 to /api/orders failed at 10:03:21 with NullPointerException.” Logs preserve the individual story that metrics aggregate away.

The car analogy: if metrics are the dashboard, logs are the diagnostic trouble code log that the mechanic plugs into. They record specific events with their context.

Structured vs unstructured logging

For decades, application logs looked like this:

2024-05-10T10:00:00Z my-host app: ERROR user 42 not found

This is unstructured logging — a free-form string. Humans can read it; computers struggle. To search for “all errors involving user 42 across the fleet,” some downstream tool has to parse that string with regular expressions, hope every team used the same format, and reconstruct fields that the application already knew. It’s an information-destruction pipeline.

Structured logging flips this around: the application emits machine-readable key-value records from the start.

{
  "ts": "2024-05-10T10:00:00Z",
  "level": "info",
  "service": "billing",
  "request_id": "abc-123",
  "user_id": 42,
  "msg": "charged credit card",
  "amount": 99.99
}

Now “all errors involving user 42” is an indexed field lookup, not a regex hunt. OpenTelemetry takes structured logging a step further by defining a formal LogRecord data model [Source: https://blog.codinghorror.com/the-problem-with-logging/]. A LogRecord includes:

• timestamp — when the event occurred (application time).
• observed_timestamp — when the collector saw it (pipeline time).
• severity_number — a normalized numeric severity on a 1–24 scale: TRACE (1–4), DEBUG (5–8), INFO (9–12), WARN (13–16), ERROR (17–20), FATAL (21–24).
• severity_text — the original severity string from the source (“WARN”, “Warning”, “warn”).
• body — the main content, typed as AnyValue (string, number, map, or list).
• attributes — structured key-value context (e.g., http.method = "GET", db.system = "postgresql").
• resource — service/host/cluster metadata shared with traces and metrics from the same process.
• trace_id, span_id, trace_flags — first-class trace context fields.
• instrumentation_scope — which library produced the record.
• dropped_attributes_count — bookkeeping for attribute limits.

The genius of severity_number is that it normalizes severities across logging frameworks. A backend query like severity_number >= 17 (“ERROR or worse”) works whether the source used Python’s logging.ERROR, Java’s WARN, Go’s log.Error, or .NET’s LogLevel.Error [Source: https://natesnewsletter.substack.com/p/context-windows-are-a-lie-the-myth].

Figure 2.2: OpenTelemetry LogRecord schema

graph TD
    LR["LogRecord"]
    LR --> TS["timestamp<br/>application time"]
    LR --> OTS["observed_timestamp<br/>pipeline time"]
    LR --> SEV["Severity"]
    SEV --> SN["severity_number<br/>1-24 normalized"]
    SEV --> ST["severity_text<br/>'WARN', 'Warning'..."]
    LR --> B["body : AnyValue<br/>string / num / map / list"]
    LR --> A["attributes<br/>http.method, db.system..."]
    LR --> R["resource<br/>service.name, k8s.*, host.*"]
    LR --> TC["Trace Context"]
    TC --> TID["trace_id"]
    TC --> SID["span_id"]
    TC --> TF["trace_flags"]
    LR --> IS["instrumentation_scope"]
    LR --> DA["dropped_attributes_count"]

Log levels, ingestion pipelines, and indexing strategies

Application code chooses a severity level at the log call site:

• DEBUG/TRACE — verbose internal state, off in production.
• INFO — normal lifecycle events: startup, request completed.
• WARN — degraded behavior that did not fail (retry succeeded, fallback used).
• ERROR — operation failed, often actionable.
• FATAL — process-ending or critical.

A typical pipeline looks like this:

1. Application writes structured logs to stdout (the Twelve-Factor recommendation).
2. Node agent (Fluent Bit, Vector, or OpenTelemetry Collector) tails container stdout, parses, enriches with Kubernetes metadata (pod name, namespace, node), and ships.
3. Aggregator/gateway buffers, batches, and may sample or redact.
4. Storage backend indexes for search. Common choices: Elasticsearch (full-text inverted index), Loki (label-only index over chunked raw lines), or vendor services.

Indexing strategy matters enormously for cost. Elasticsearch-style full-text indexing lets you search any word in any log instantly, but the index can be larger than the data itself and grows roughly linearly with log volume. Loki-style label-only indexing keeps the index tiny by only indexing a handful of labels (service, pod, namespace) and storing raw log lines compressed; queries then grep through chunks. The trade-off: Loki is dramatically cheaper to operate but slower for arbitrary substring searches.

Compared to traditional shippers like Fluentd or Logstash, the OpenTelemetry approach standardizes the data model itself. Fluentd and Logstash are pipelines — powerful at routing and transforming, but they impose no global schema. Each route defines its own JSON shape, severity semantics, and field names. The OpenTelemetry Collector plays a similar pipeline role but operates on the typed OTLP model across all three signals [Source: http://susandumais.com/CHI2012-12-tailanswers-chi2012.pdf].

Why logs alone are insufficient for distributed root cause analysis

A common misconception is that “good logs” suffice for observability. They don’t, for a fundamental architectural reason: logs are emitted per service, but root causes in distributed systems live in the relationships between services.

Consider a checkout request that times out. The checkout-service log shows “called inventory: timeout after 5s.” The inventory-service log shows “received call, query took 4.8s.” The postgres log shows “lock wait.” These three lines, scattered across three different log streams, describe one causal chain. Reconstructing it requires:

1. Knowing the request ID and consistently propagating it through every service (which most teams botch at least one boundary of).
2. Time-aligning logs whose clocks may differ by milliseconds or seconds.
3. Guessing the causal order from timestamps that don’t capture parent/child relationships.
4. Repeating this for every layer of the call graph.

This is exactly the problem traces solve, and it’s why logs alone — no matter how structured — cannot answer “what made this specific slow request slow?” in a microservices architecture [Source: https://arxiv.org/html/2501.11709v3]. Logs complement traces by carrying the rich context of individual events; they do not replace the causal graph.

Key Takeaway: Structured logs with a standardized data model (the OpenTelemetry LogRecord) capture rich per-event detail with normalized severity and shared resource context, but they cannot reconstruct causality across services on their own — that’s what traces are for.

Traces — Causally-linked Spans

A trace is the recorded journey of a single request through a distributed system. If metrics are the dashboard and logs are the event log, traces are the GPS track — they tell you not just that the trip happened but the exact route taken, which segments were slow, and where the detours were.

Trace, span, and span context

The core data model is a small hierarchy:

• A trace is identified by a 128-bit trace_id (32 hex characters), e.g. 4bf92f3577b34da6a3ce929d0e0e4736. All work performed in service of a single logical request shares this ID.
• A span is a single named unit of work within the trace, identified by a 64-bit span_id (16 hex characters), e.g. 00f067aa0ba902b7. Each span has a start time, end time, name (often the operation, like GET /api/orders), status, and attributes.
• A span context is the immutable propagation envelope that carries trace_id, current span_id, and trace_flags (such as the sampling bit) across process boundaries. The standard wire format is the W3C Trace Context traceparent HTTP header [Source: https://developers.openai.com/cookbook/examples/gpt4-1_prompting_guide]:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
              |       trace_id              |   span_id    |flags

When checkout-service calls inventory-service, it copies its current span context into a traceparent header on the outbound HTTP request. The receiving service reads that header, knows the parent’s trace_id and span_id, and starts a child span under the same trace. This is how a single trace stitches itself together across N services with no central coordinator.

A concrete example: a user clicks “place order.” The browser hits checkout-service, which calls inventory-service, which calls postgres, which calls payment-service. Five spans, all sharing one trace_id, each pointing to its parent’s span_id. Drawn on a timeline, this is the familiar flame graph view that tracing UIs render.

Figure 2.3: Span tree for a checkout request (one trace_id, parent-child via span_id)

graph TD
    Root["POST /checkout<br/>SERVER · checkout-service<br/>span_id=a1 · 980ms"]
    Root --> Inv["inventory.check<br/>CLIENT · checkout-service<br/>span_id=b2 · parent=a1 · 620ms"]
    Inv --> InvS["GET /inventory<br/>SERVER · inventory-service<br/>span_id=c3 · parent=b2 · 600ms"]
    InvS --> DB["db.query SELECT stock<br/>CLIENT · inventory-service<br/>span_id=d4 · parent=c3 · 540ms"]
    Root --> Pay["payment.charge<br/>CLIENT · checkout-service<br/>span_id=e5 · parent=a1 · 320ms"]
    Pay --> PayS["POST /charge<br/>SERVER · payment-service<br/>span_id=f6 · parent=e5 · 300ms"]

Parent-child relationships and span kinds

Each span (except the root) carries the span_id of its parent. This builds an explicit directed acyclic graph rather than the implicit one you’d have to reconstruct from log timestamps. The tree captures causality — the parent caused the child to happen — not merely temporal proximity.

OpenTelemetry classifies spans by span kind, which clarifies the role of the span in a network interaction:

• INTERNAL — work entirely within a single process (e.g., a slow function).
• SERVER — a span that receives an incoming RPC (the “callee” side of a request).
• CLIENT — a span that sends an outgoing RPC (the “caller” side).
• PRODUCER — emits a message to a queue (e.g., publish to Kafka).
• CONSUMER — receives a message from a queue.

A typical HTTP call generates two spans: a CLIENT span on the caller and a SERVER span on the callee, linked by the same trace_id and a parent-child relationship via the propagated context. Span kinds let analysis tools draw service maps automatically: any CLIENT span pointing to a SERVER span in a different service implies an edge between those services.

Spans also carry events (timestamped sub-points, e.g., “cache miss”) and links (references to other traces, useful for fan-out or queue-driven workflows where one input span causes work in many traces).

Service maps derived from trace topology

Because every cross-service call produces a CLIENT/SERVER pair tagged with service.name, a tracing backend can build a live service map by aggregating recent traces:

• Each service.name becomes a node.
• Each observed parent-child cross-service relationship becomes an edge.
• Edge weight = call rate; edge color = error rate or latency.

You no longer maintain a hand-drawn architecture diagram. The system draws its own current architecture from the traces it processes, automatically reflecting newly deployed services, retired ones, and unexpected dependencies (the classic “wait, why is checkout-service calling legacy-billing directly?” discovery).

Figure 2.4: Service map auto-derived from CLIENT→SERVER span pairs

graph LR
    Web["web-frontend"]
    Co["checkout-service"]
    Inv["inventory-service"]
    Pay["payment-service"]
    PG[("postgres")]
    Rd[("redis")]
    Web -- "1200 rps<br/>err 0.1%" --> Co
    Co -- "1100 rps<br/>err 0.4%" --> Inv
    Co -- "900 rps<br/>err 2.1% (hot)" --> Pay
    Inv -- "1100 rps<br/>p99 540ms" --> PG
    Co -- "1200 rps<br/>p99 5ms" --> Rd
    Pay -- "900 rps<br/>err 1.8%" --> PG

Key Takeaway: Traces capture causality across services via trace_id and parent span_id propagation through W3C Trace Context, enabling per-request flame graphs and auto-derived service maps — the one capability metrics and logs cannot deliver.

Correlating the Three

Each signal in isolation is useful; together they become an investigative narrative. The mechanics of correlation — how a dot on a Grafana chart links to a trace, which in turn surfaces the relevant log lines — rely on three standardized hooks.

Exemplars in Prometheus histograms

The naive way to link a latency spike to a slow request is to put trace_id in the metric labels. That blows up cardinality instantly: every unique trace_id creates a new series, and you’ll have billions in a day. Exemplars solve this by attaching a few representative trace_id/span_id pointers alongside metric samples without making them series-defining labels [Source: https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/prompt-engineering].

The OpenMetrics text exposition format appends an exemplar after the sample value, prefixed by #:

http_server_request_duration_seconds_bucket{le="0.5",method="GET",service="api"} 240 1700000100 \
  # {trace_id="4bf92f3577b34da6a3ce929d0e0e4736",span_id="00f067aa0ba902b7"} 1.700000099e+09

Constraints worth memorizing:

• At most one exemplar per series per scrape interval is typically exported.
• Exemplars attach most commonly to histogram buckets (great for “find me a trace in this latency bucket”) and sometimes to counter increments (great for “find me a trace for this kind of error”).
• Exemplars live in a separate storage path from ordinary time series in Prometheus TSDB, specifically to avoid the cardinality explosion of storing high-cardinality IDs alongside normal series.

The trace ID format is exactly the W3C Trace Context format used by OpenTelemetry — 32 hex chars for trace_id, 16 hex chars for span_id — and the values must be the same IDs your tracer is shipping to Tempo or Jaeger. Mismatched propagation is the most common reason exemplars appear in Grafana but the “View trace” link returns “trace not found.”

The end-to-end workflow when a developer investigates a latency spike at 10:05:

1. Grafana renders the p99 latency line from histogram_quantile(0.99, sum by (le) (rate(http_server_request_duration_seconds_bucket[5m]))).
2. Small dots appear along the line wherever Prometheus has stored exemplars.
3. Developer hovers a dot at the spike; popup shows trace_id=4bf92f3577…, latency 2.3s.
4. Click “View trace.” Grafana queries the Tempo data source by trace_id.
5. The trace opens: checkout-service called inventory-service which sat on a database lock for 2.1s.

The dot was sampled — not every slow request gets an exemplar, just enough to give the investigator a starting point. This is the practical realization of the metrics-to-traces correlation that the three pillars metaphor promises.

Figure 2.5: Metric-to-trace exemplar drill-down workflow

sequenceDiagram
    participant Dev as Developer
    participant Graf as Grafana
    participant Prom as Prometheus<br/>(TSDB + exemplar store)
    participant Tempo as Tempo
    Dev->>Graf: open p99 latency dashboard
    Graf->>Prom: histogram_quantile(0.99, ...)
    Prom-->>Graf: latency series + exemplar dots
    Note over Graf: spike at 10:05<br/>dot shows trace_id=4bf92f35...
    Dev->>Graf: hover dot, click "View trace"
    Graf->>Tempo: GET /traces/4bf92f35...
    Tempo-->>Graf: span tree (checkout → inventory → db)
    Note over Dev,Tempo: db.query span = 2.1s<br/>(lock wait identified)

Trace-to-logs joins via trace_id and span_id

The second leg of the correlation triangle is trace-to-logs, made possible because the OpenTelemetry LogRecord schema reserves dedicated trace_id and span_id fields at the top level — not buried in attributes [Source: https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices].

When an application logs inside an active span, the OpenTelemetry logging integration automatically copies the current span’s trace_id and span_id into the LogRecord. In Java with log4j2, in Python with the OTel logging instrumentation, in Go with the otelslog handler — the SDK does it for you. You no longer have to remember to call logger.info("...", "trace_id", currentSpan.TraceID()) at every log site.

From the trace view in Grafana or Tempo, “show logs for this span” is then a single derived field query:

{service="checkout"} |= "4bf92f3577b34da6a3ce929d0e0e4736"

The trace_id acts as the universal join key. The same query in reverse — start in logs, jump to the trace — gives you the bidirectional navigation that makes investigations feel fluid instead of archaeological.

A subtle pitfall: sampling mismatches. If you head-sample traces at 10% but log unconditionally on every request, 90% of your log trace_ids will point to traces that don’t exist in Tempo. To avoid this, use tail-based sampling (decide which traces to keep based on what happened in them — errors and slow ones always kept) or ensure that the sampling decision is made early and propagated, so logs from non-sampled traces don’t carry stale IDs.

Unified resource attributes across all signals

The third correlation hook is the Resource — the OpenTelemetry concept of a small, fixed set of attributes describing where the telemetry came from. The same resource attributes are attached to every metric, log, and trace emitted by a process:

service.name = "checkout-service"
service.namespace = "payments"
service.instance.id = "pod-abc123"
service.version = "1.4.2"
deployment.environment = "prod"
k8s.pod.name = "checkout-7f8b9-abcde"
k8s.namespace.name = "payments-prod"
host.name = "ip-10-0-1-42.ec2.internal"
cloud.region = "us-east-1"

Because all three signals share these attributes verbatim, a query like “all telemetry for service.name = checkout-service in deployment.environment = prod” returns matching metrics, logs, and traces from a single filter — no vendor-specific tag mapping, no per-tool index naming conventions. This is the practical meaning of the OpenTelemetry promise of “vendor-neutral observability”: a service.name in one tool means the same thing in another.

Semantic conventions extend this consistency to operation-level attributes. The OpenTelemetry spec defines official keys like http.request.method, http.response.status_code, db.system.name, messaging.system, rpc.service. When everyone uses these standard names, queries portable across backends become routine.

Figure 2.6: Correlation hub — three standardized hooks unifying the signals

graph TB
    subgraph Signals
        M["Metrics<br/>Prometheus TSDB"]
        L["Logs<br/>Loki / Elasticsearch"]
        T["Traces<br/>Tempo / Jaeger"]
    end
    H["Shared Resource<br/>service.name<br/>k8s.pod.name<br/>deployment.environment"]
    EX["Exemplars<br/>trace_id appended<br/>to histogram samples"]
    TC["LogRecord trace context<br/>top-level trace_id / span_id<br/>auto-populated by SDK"]
    M -- "drill via" --- EX
    EX -- "to" --- T
    T -- "join via" --- TC
    TC -- "to" --- L
    M --- H
    L --- H
    T --- H
    H --- Q["query: service.name=checkout<br/>returns metrics + logs + traces"]

Key Takeaway: The three signals become one investigative narrative through three standardized hooks — exemplars connect metrics to traces without cardinality blowup, OpenTelemetry’s LogRecord puts trace_id/span_id as top-level fields for trace-to-log joins, and a shared Resource model means service.name and friends mean the same thing across all signals.

Chapter Summary

This chapter dissected the three observability signals at the data-model level and showed how they interlock.

Metrics are cheap aggregates that answer “how many” and “how fast.” Prometheus offers four instrument types: counters, gauges, histograms, and summaries. The critical operational decision is histogram-vs-summary: summaries compute quantiles per-instance and cannot be aggregated across the fleet, while histogram buckets are fully aggregatable and let you compute new quantiles at query time. Native histograms (Prometheus 2.40+) further reduce series count via dynamic log-spaced buckets in a single time series. Cardinality — the product of unique label combinations — is the dominant cost, so labels like user_id or trace_id must never be used as series-defining metric labels.

Logs are discrete events with rich per-event context. The OpenTelemetry LogRecord defines a typed schema — timestamp, severity_number (normalized 1–24), severity_text, body (AnyValue), attributes, resource, trace context — that turns “lines of text” into structured telemetry events. This contrasts with Fluentd/Logstash pipelines, which transport JSON or syslog without a global data model. Despite their richness, logs alone cannot reconstruct causality across services in microservices systems.

Traces capture causality. A trace is identified by a 128-bit trace_id; each unit of work is a span with a 64-bit span_id and a parent pointer. W3C Trace Context propagates these IDs across process boundaries via the traceparent header. Span kinds (SERVER/CLIENT/PRODUCER/CONSUMER/INTERNAL) let tracing backends auto-derive service maps from the topology of observed spans.

Correlation is the payoff. Exemplars attach trace_id/span_id pointers to Prometheus histogram samples in OpenMetrics format, stored separately from normal series to avoid cardinality explosion. Trace IDs in LogRecord as top-level fields make trace-to-logs navigation a single join. A shared Resource — service.name, k8s.pod.name, deployment.environment — ties all three signals to the same process identity. Semantic conventions standardize the rest. Together these hooks turn three independent data streams into one coherent investigative narrative: spike on a graph → hover an exemplar dot → open the trace → see the slow span → click through to its logs → identify the root cause.

Key Terms

Chapter 3: Prometheus Architecture and Data Model

If chapter 2 explained why metrics matter, this chapter is about how one of the most influential metrics systems in the world actually works. Prometheus is more than a “metrics database.” It is an opinionated bundle of design choices — a pull-based scraper, a multi-dimensional data model, a custom time-series database (TSDB), and a query engine — that together produce a system that is operationally simple at small scale and powerful at large scale.

Understanding Prometheus internals matters for two reasons. First, almost every cloud-native observability stack you will encounter borrows Prometheus’s conventions: the OpenMetrics exposition format, label-based identity, PromQL, and remote write. Second, when something goes wrong — slow queries, OOMing pods, missing data after a crash, runaway cardinality — you cannot debug it without a mental model of the scrape loop, the TSDB block layout, and the WAL.

This chapter walks through Prometheus from the outside in: the components, the pull model, the data model, and finally how data is stored, retained, and shipped to long-term storage.

Learning Objectives

By the end of this chapter, you will be able to:

• Diagram the Prometheus server, the scrape loop, the TSDB, and Alertmanager, and describe how they interact during a normal scrape and alert cycle.
• Explain the pull-based scraping model, the rationale behind it, and the specific trade-offs it makes versus a push model — including where the Pushgateway fits in.
• Map a metric name, its label set, and its timestamp onto Prometheus’s multi-dimensional data model, and read or write a valid OpenMetrics exposition payload.
• Reason about TSDB blocks, the WAL, retention, and remote-write integrations like Thanos, Cortex, and Mimir well enough to plan capacity and debug failures.

Section 1: Server Components

A single prometheus binary contains several cooperating subsystems. Architecturally, you can think of Prometheus as a small distributed system that happens to run inside one process: a retrieval subsystem pulls data, a storage subsystem persists it, a query subsystem answers PromQL questions, and a rules subsystem fires alerts to an external Alertmanager.

Figure 3.1: Prometheus server components and external integrations

flowchart LR
    SD[Service Discovery<br/>K8s, Consul, EC2, DNS] --> R[Retrieval<br/>Scrape Loop]
    R -->|HTTP GET /metrics| Targets[(Scrape Targets)]
    R --> TSDB[(TSDB<br/>head + WAL + blocks)]
    TSDB --> API[HTTP API + Web UI]
    API --> Grafana[Grafana / Clients]
    TSDB --> RE[Rule Engine<br/>recording + alerting]
    RE --> AM[Alertmanager<br/>external]
    TSDB --> RW[Remote Write]
    RW --> LTS[(Long-term Storage<br/>Thanos / Mimir / Cortex)]

Retrieval (scrape) loop

The retrieval subsystem is the heart of Prometheus’s interaction with the outside world. It maintains a set of scrape targets, each of which is essentially a URL like http://10.0.4.17:9100/metrics, a scrape interval (commonly 15s or 30s), and an optional set of labels.

For each target, Prometheus runs a small state machine on a timer:

1. Resolve the target’s address (service discovery may have changed it).
2. Open an HTTP GET to /metrics with a configured timeout.
3. Parse the response as OpenMetrics or the legacy Prometheus text format.
4. Apply relabeling rules to drop or rewrite the resulting samples.
5. Append the samples to the TSDB’s head block, tagged with the scrape timestamp.

If the scrape fails (timeout, 5xx, parse error), Prometheus still writes a synthetic series called up{job="...",instance="..."} with the value 0. On success, it writes up = 1 plus several built-in scrape_* metrics like scrape_duration_seconds and scrape_samples_scraped. These “meta” series are how you alert on monitoring itself.

A useful analogy: think of the scrape loop as a postal carrier who walks the same route every 15 seconds. The carrier does not wait for anyone to mail a letter; they pick up whatever is in the mailbox at that moment. If the mailbox is missing — the house has been demolished — the carrier files a report (up = 0) and moves on.

TSDB on-disk format and blocks

The TSDB (time-series database) is where samples land after the scrape loop. Prometheus does not use Postgres, RocksDB, or any general-purpose database for time series — it ships its own purpose-built engine designed for one workload: many millions of monotonically advancing numeric series.

The on-disk layout is straightforward. Inside the data directory you will find:

• A wal/ directory containing append-only write-ahead-log segments (00000001, 00000002, …) and possibly checkpoint.N/ directories.
• One directory per block, each named with a ULID (e.g., 01J1H1Q2K3V3Y2...). Each block contains chunks/, index, meta.json, and possibly tombstones [Source: https://natesnewsletter.substack.com/p/context-windows-are-a-lie-the-myth].

Conceptually there are two regions:

• The head block holds the most recent ~2 hours of data, lives in memory, and is protected by the WAL.
• Older data lives in immutable blocks on disk; each block covers a contiguous time range and is never modified after it is written.

We will return to blocks, the WAL, and compaction in section 4. The important takeaway here is that the storage engine is not a free-for-all key-value store — it is shaped around time-bounded immutable units, which is what makes 90-day retention with sub-second queries feasible on a single node.

HTTP API and web UI

Everything you can do to Prometheus — query it, list targets, see the configuration, fire test alerts — goes through its HTTP API. Notable endpoints:

The web UI bundled with the server is intentionally minimal: a query box, a graph view, a targets page, and an alerts page. It exists so an operator can debug from a fresh laptop with no Grafana. For day-to-day dashboards, you point Grafana (or another visualization tool) at the same HTTP API.

Service discovery integrations

Static configuration — listing IPs in a YAML file — falls over the moment your environment becomes dynamic. Prometheus solves this with service discovery (SD), where the list of scrape targets is generated from an external source of truth.

Built-in SD integrations include Kubernetes, Consul, EC2, GCE, Azure, DNS SRV records, file-based SD, and many more. A Kubernetes SD configuration, for example, asks the API server for all pods matching a selector, then turns each pod into a target with labels like __meta_kubernetes_pod_name, __meta_kubernetes_namespace, and __meta_kubernetes_pod_label_app. Relabeling rules (covered in section 3) then promote these “underscore” labels into permanent target labels or drop them entirely.

Think of SD as the address book: Prometheus’s scrape loop is the postal carrier, but SD is what tells the carrier which houses exist this morning.

Key Takeaway: Prometheus is a small distributed system in a single binary — a scrape loop driven by service discovery, a purpose-built TSDB protected by a WAL, an HTTP query API, and a rules engine that talks to an external Alertmanager. Operating Prometheus well means understanding each of those subsystems independently.

Section 2: The Pull Model

Of all the design decisions in Prometheus, the choice to pull metrics rather than have services push them is the most contentious — and the most consequential. Understanding why pull was chosen helps you reason about when push is genuinely better and when it just feels easier because it is what you already know.

Why pull was chosen over push

In a pull model, targets expose a /metrics HTTP endpoint and Prometheus periodically calls it. In a push model, targets connect to a central collector and stream samples as they are produced.

The pull design buys Prometheus several properties that are surprisingly hard to replicate in push systems [Source: https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/prompt-engineering]:

• Liveness is implicit. If Prometheus cannot reach the target, the target’s up series becomes 0 and existing series are marked stale. There is no “this metric is from a dead pod that hasn’t told us yet” ambiguity.
• Prometheus controls load. The scrape interval, timeout, and concurrency are set on the server side. A misbehaving target cannot flood Prometheus with samples.
• Operators can simulate Prometheus with curl. Anyone with shell access to the target can call /metrics and see exactly what Prometheus would see. There is no special protocol to inspect.
• No client-side buffering or retries. The target does not need to know about Prometheus’s availability or implement a queue. The carrier shows up; if no one is home, no one is home.
• Horizontal scaling by sharding targets. Splitting load across multiple Prometheus servers is “give server A half the targets, server B the other half.”

The trade-offs are real, however. Pull is awkward when:

• Targets live behind NAT or firewalls and Prometheus cannot reach them.
• Targets are very short-lived (a CI job that runs for 3 seconds with a 15s scrape interval).
• The “target” is conceptually a thing that emits events, not a thing with a current state (a webhook delivery, a batch ingest event).

Pull is therefore not universally better — it is better for the steady-state of long-lived services in trusted networks, which happens to describe most of what runs inside a Kubernetes cluster.

Figure 3.2: Scrape loop sequence (pull model with implicit liveness)

sequenceDiagram
    participant SD as Service Discovery
    participant P as Prometheus<br/>Scrape Loop
    participant T as Target /metrics
    participant DB as TSDB Head

    SD->>P: target list (IPs, labels)
    loop every 15s
        P->>T: HTTP GET /metrics
        alt target healthy
            T-->>P: 200 OK + OpenMetrics body
            P->>P: parse + relabel samples
            P->>DB: append samples + "up=1"
        else target unreachable
            T--xP: timeout / 5xx / parse error
            P->>DB: append "up=0" (stale marker)
        end
    end

The Pushgateway for short-lived jobs

For the cases pull cannot reach naturally — most importantly short-lived batch jobs — Prometheus offers a companion service called the Pushgateway. It is a small HTTP server with a deceptively simple job: accept pushed metrics, store the current value in memory, and expose them on its own /metrics endpoint so Prometheus can scrape the Pushgateway like any other target.

Used correctly, the pattern looks like this:

1. A nightly batch job starts, runs for two minutes, then finishes.
2. Right before exiting, the job pushes nightly_import_last_run_status (0 for success), nightly_import_last_run_duration_seconds, and nightly_import_last_run_timestamp_seconds to the Pushgateway, grouped by job="nightly_import".
3. Prometheus continues to scrape the Pushgateway every 15 seconds, so these “last run” metrics are visible to PromQL and Alertmanager whether or not the job is currently running.

A simple Go example illustrates the shape of this pattern:

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/push"
    "time"
)

var (
    jobDuration  = prometheus.NewGauge(prometheus.GaugeOpts{Name: "nightly_import_last_run_duration_seconds"})
    jobStatus    = prometheus.NewGauge(prometheus.GaugeOpts{Name: "nightly_import_last_run_status"})
    jobTimestamp = prometheus.NewGauge(prometheus.GaugeOpts{Name: "nightly_import_last_run_timestamp_seconds"})
)

func main() {
    start := time.Now()
    err := runImport()
    jobDuration.Set(time.Since(start).Seconds())
    jobTimestamp.Set(float64(time.Now().Unix()))
    if err != nil { jobStatus.Set(1) } else { jobStatus.Set(0) }

    _ = push.New("http://pushgateway:9091", "nightly_import").
        Collector(jobDuration).Collector(jobStatus).Collector(jobTimestamp).Push()
}

The anti-patterns are at least as important to know [Source: https://blog.codinghorror.com/the-problem-with-logging/]:

• Do not use Pushgateway for long-running services. There is no built-in staleness; if the service dies, its last pushed values sit in Pushgateway forever and dashboards look healthy.
• Do not push per-instance labels for ephemeral pods. Each pod UUID creates a new label combination that is never cleaned up.
• Do not treat Pushgateway as an event stream. Each push replaces the current value; it is a cache, not a queue.
• Do not use it for SLO metrics like latency or error rate. SLOs require continuous, high-resolution series that reflect live behavior — Pushgateway hides outages behind stale cached values.

For most modern push needs, the OpenTelemetry Collector is a better answer (chapter 5 covers it in depth): it accepts OTLP push from applications, handles retries and batching, and can expose a Prometheus-compatible scrape endpoint or remote-write to a backend.

Federation and hierarchical scraping

What if you have too many targets for one Prometheus, or you want a “global” view across regions without one giant server scraping the world? Prometheus supports federation: one Prometheus scrapes another’s /federate endpoint and pulls a subset of its series.

A typical hierarchy looks like:

Federation is best used to pull aggregated series (the output of recording rules) rather than raw metrics — pulling raw, high-cardinality data through federation will bottleneck on a single HTTP scrape. For “ship everything to long-term storage,” remote write (section 4) is almost always a better fit.

Key Takeaway: Pull gives Prometheus implicit liveness, server-side load control, and a debuggable wire protocol; push (via Pushgateway or OTel Collector) is the escape hatch for short-lived jobs and firewalled targets. Use federation for aggregated views, remote write for raw data shipping.

Section 3: Data Model and Exposition Format

Prometheus’s data model is small enough to fit on an index card, yet it underpins every PromQL query you will ever write. Internalize it once and the rest of the system stops feeling magical.

Metric name + label set = unique time series

In Prometheus, a time series is uniquely identified by its metric name plus its set of labels. Every other property — the metric type, the help text, the unit — is metadata; the identity is the name and the labels.

Consider this single sample:

http_requests_total{job="api", method="GET", status="200", path="/users"}  1873  1717520400000

That has three parts:

• Metric name: http_requests_total. Conventionally _total suffixes counters; _seconds, _bytes, _ratio suffixes give units.
• Label set: {job="api", method="GET", status="200", path="/users"}. Each label is a (name, value) pair; values are arbitrary UTF-8 strings.
• Sample: a numeric value (1873) and a Unix-millisecond timestamp (1717520400000).

Change any label value and you have a different series. http_requests_total{...,method="GET"} and http_requests_total{...,method="POST"} are entirely separate time series, stored separately, indexed separately, and counted separately against cardinality budgets.

This is the multi-dimensional data model: instead of inventing one metric per dimension (http_requests_GET_200_total, http_requests_POST_500_total, …) you have one metric name with multiple label dimensions, and PromQL lets you slice along those dimensions at query time:

sum by (status) (rate(http_requests_total{job="api"}[5m]))

A practical analogy: a metric name is a spreadsheet, labels are the columns, label values are the cell contents, and each unique row is a time series. Adding a column with high cardinality (say, user_id) is like adding a column to a spreadsheet that has one row per user — your data grows linearly in the number of distinct values, and so does Prometheus’s memory.

OpenMetrics text exposition format

Targets expose metrics in a deliberately simple line-based format. The original Prometheus text format was standardized as OpenMetrics in 2020 and is now an IETF-tracked specification. A typical /metrics payload looks like:

# HELP http_requests_total Total HTTP requests received.
# TYPE http_requests_total counter
http_requests_total{method="GET",status="200"} 1873
http_requests_total{method="GET",status="500"} 4
http_requests_total{method="POST",status="200"} 219

# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 4.2848256e+07

# HELP http_request_duration_seconds HTTP request latency.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.1"} 1500
http_request_duration_seconds_bucket{le="0.5"} 1860
http_request_duration_seconds_bucket{le="1.0"} 1872
http_request_duration_seconds_bucket{le="+Inf"} 1873
http_request_duration_seconds_sum 92.7
http_request_duration_seconds_count 1873
# EOF

Key rules:

• Lines starting with # are comments, except for # HELP <metric> <text> and # TYPE <metric> <counter|gauge|histogram|summary>, which are semantically meaningful.
• Each sample line is metric_name{labels} value [timestamp]. The timestamp is optional; if omitted, Prometheus uses the scrape time.
• Histograms expand into multiple lines: one bucket per le boundary, plus _sum and _count. Each bucket is its own series.
• OpenMetrics ends with a literal # EOF line.

The format’s simplicity is the point: anyone can write an exporter in a few dozen lines of code in any language, and anyone can debug one with curl.

Honor labels, target labels, and relabeling

When Prometheus scrapes a target, it sees two sets of labels: the labels the target exposed in its /metrics output, and target labels that Prometheus attaches automatically (most importantly job and instance, plus anything service discovery contributed).

By default, if the target’s exposed labels conflict with target labels, the target labels win. The honor_labels: true setting in a scrape config flips this: the target’s labels take precedence. This matters mostly for the Pushgateway (where the pushing job has set its own job label intentionally) and for federation (where you want to preserve the original job label from the upstream Prometheus).

Relabeling is the small declarative DSL Prometheus uses to transform labels before storing samples. There are two flavors:

• relabel_configs: run before the scrape, against target metadata labels (the __meta_* labels from service discovery). Used to select which targets to scrape and to rewrite their labels.
• metric_relabel_configs: run after the scrape, against each sample’s labels. Used to drop high-cardinality samples or rename labels.

A worked example: Kubernetes SD discovers thousands of pods. You want to scrape only pods with the annotation prometheus.io/scrape=true, set their job label from prometheus.io/job, and drop a noisy histogram bucket:

scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # Keep only pods that opt in.
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      # Set the job label from the pod's annotation.
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_job]
        target_label: job
      # Promote namespace and pod into permanent labels.
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
    metric_relabel_configs:
      # Drop a known cardinality-bomb metric.
      - source_labels: [__name__]
        regex: "go_gc_pauses_seconds_bucket"
        action: drop

Figure 3.3: Relabeling pipeline (target metadata to stored series)

flowchart LR
    SD[Service Discovery] -->|"__meta_kubernetes_*<br/>__meta_consul_*"| RC[relabel_configs<br/>keep / drop / rewrite]
    RC --> TL[Final target labels<br/>"job, instance, namespace, pod"]
    TL --> SCR[Scrape /metrics]
    SCR --> RAW[Raw samples<br/>"name + labels + value"]
    RAW --> MRC[metric_relabel_configs<br/>drop high-cardinality]
    MRC --> TSDB[(TSDB head block)]

Relabeling is the place where most operational bugs hide. Two rules of thumb: (1) always test relabeling against a known target with --log.level=debug, and (2) drop unbounded labels (user IDs, request IDs, HTTP paths with query strings) before they enter the TSDB — once they are in, they cost memory until retention removes them.

Key Takeaway: A time series is uniquely identified by metric_name + label_set; labels are how you slice data at query time but also how cardinality explodes. The OpenMetrics text format is intentionally readable, and relabeling is your control plane for shaping labels before and after each scrape.

Section 4: Storage, Retention, and Remote Write

Now we follow a sample from the moment Prometheus accepts it from the scrape loop, through the head block and WAL, into a 2-hour block on disk, through compaction, and finally out to a long-term storage system over remote write.

Local TSDB block compaction and WAL

When the scrape loop produces a sample, the TSDB does two things, in order [Source: https://natesnewsletter.substack.com/p/context-windows-are-a-lie-the-myth]:

1. Append a record to the WAL. The sample (with its series ID, timestamp, and value) is encoded as a CRC-checksummed record and written to the current WAL segment file (typically ~128 MB segments named 00000001, 00000002, …).
2. Update the in-memory head block. The TSDB looks up (or creates) an in-memory series for the label set and appends the sample to that series’ current chunk.

The WAL ensures durability: if Prometheus crashes after step 1 but before the head block is persisted as a finished block, the sample can be recovered on restart by replaying the WAL.

Every ~2 hours, the head’s accumulated samples are cut into a new immutable on-disk block. Each block directory looks like:

01J1H1Q2K3V3Y2.../
  meta.json       # ULID, minTime, maxTime, compaction level, sources, stats
  index           # symbol table + postings lists (label=value -> series IDs)
  chunks/
    000001        # concatenated XOR-compressed chunks (mmapped at query time)
    000002
  tombstones      # optional, deletion intervals

The index file is the crown jewel: it interns every label name and value into a symbol table, assigns each series an integer ID, and stores postings lists — sorted lists of series IDs for each label=value pair. When you run sum by (status) (rate(http_requests_total{job="api"}[5m])), the index lets Prometheus find every series matching __name__="http_requests_total" and job="api" by intersecting two postings lists in milliseconds, no table scan required.

Chunks themselves use XOR-based delta-of-delta compression for timestamps and Gorilla-style XOR for float values, which is brutally efficient: typical compressed sizes are ~1-2 bytes per sample. Chunks are read via memory-mapped I/O rather than copied into the heap, which is why Prometheus’s working set often shows modest RSS but a large OS page cache.

Background compaction periodically merges adjacent 2-hour blocks into larger ones (8h, then ~24h, then multi-day) [Source: https://www.youtube.com/watch?v=MqqKT6etxpQ]. Larger blocks mean fewer index files to open per query and better compression. Retention is enforced at the block level: when --storage.tsdb.retention.time=30d is set, any block whose maxTime is older than 30 days ago is deleted in one shot.

To keep WAL replay fast after a crash, Prometheus periodically writes a checkpoint (wal/checkpoint.N/) that snapshots the head state. After a successful checkpoint, all WAL segments numbered less than N can be deleted. On restart, recovery loads the latest checkpoint and replays only segments after it, stopping at the first record with an invalid CRC (which represents a torn write at the moment of crash).

Figure 3.4: TSDB block lifecycle from scrape to retention

stateDiagram-v2
    [*] --> Scrape: sample produced
    Scrape --> WAL: append CRC record
    WAL --> Head: update in-memory head chunk
    Head --> Head: accumulate ~2h of samples
    Head --> Block2h: cut immutable block<br/>(meta.json + index + chunks)
    Block2h --> Block8h: compact adjacent blocks
    Block8h --> Block24h: compact further
    Block24h --> BlockMulti: compact multi-day
    BlockMulti --> Deleted: retention horizon passed
    Deleted --> [*]
    Head --> Recovery: crash
    Recovery --> Head: replay WAL from checkpoint.N

Worked example — surviving a crash. Suppose Prometheus has wal/checkpoint.10/ plus segments 00000011, 00000012, 00000013, and crashes while writing to segment 13. On startup:

1. All existing immutable blocks load normally — they need no WAL [Source: http://susandumais.com/CHI2012-12-tailanswers-chi2012.pdf].
2. The head is rebuilt from checkpoint.10.
3. Segments 11, 12, and 13 are replayed; recovery stops at the first record in segment 13 whose CRC fails.
4. Any samples that did not complete their write are lost — but the head is consistent and queries work immediately.

If your Prometheus startup is slow, it almost always means either checkpoints are not happening (frequent restarts, repeated OOM kills) or WAL has grown because of an unusually long outage. The fix is to keep Prometheus healthy long enough to checkpoint.

Remote write protocol

Local TSDB is intentionally short-term storage — a few weeks at most. For longer retention, multi-cluster aggregation, or “metrics as a service,” Prometheus supports remote write: an HTTP-based protocol for shipping every sample Prometheus ingests to a remote backend.

The protocol is conceptually simple. Prometheus batches samples into Snappy-compressed protobuf messages and POSTs them to a configured URL:

remote_write:
  - url: https://mimir.example.com/api/v1/push
    basic_auth:
      username: tenant-42
      password_file: /etc/prometheus/mimir-token
    queue_config:
      capacity: 10000
      max_shards: 30
      min_backoff: 30ms
      max_backoff: 5s
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "go_.*"
        action: drop   # don't ship Go runtime metrics

Important properties:

• Remote write is lossy under backpressure: if the remote endpoint cannot keep up, Prometheus’s queue fills and eventually drops samples. The prometheus_remote_storage_* metrics tell you what is happening.
• Each sample is sent once; remote write is not a replay protocol. If you start it on day 30, the backend gets samples from day 30 onward — not the previous 30 days from local TSDB.
• write_relabel_configs lets you drop or rewrite labels on the way out, which is useful for sampling expensive metrics or stripping cardinality before it leaves your network.

Long-term storage with Thanos, Cortex, and Mimir

Three open-source projects dominate the “Prometheus at scale” space. Each makes different architectural trade-offs.

Thanos is the natural choice when you already operate Prometheus clusters and want to bolt on object-store-backed long-term storage with minimal disruption [Source: https://blog.codinghorror.com/the-problem-with-logging/]. A small sidecar runs next to each Prometheus, uploads each finalized 2-hour block to S3 (or GCS, Azure, MinIO), and exposes the local TSDB to a central Querier for “recent data.” A Store Gateway serves historical blocks from object storage; the Compactor merges and downsamples them to 5-minute and 1-hour resolutions, which is what makes long-range queries (90 days, 1 year) fast.

Cortex and Grafana Mimir take a different approach: they are remote-write-native, horizontally scalable, multi-tenant metric backends. Prometheus (or any OTel Collector or Agent) ships samples via remote write to a load-balanced endpoint that fans them out to distributors, then to ingesters that keep recent data in memory and a WAL before flushing TSDB blocks to object storage. Store-gateways serve historical reads, and a query-frontend / query-scheduler layer parallelizes and caches PromQL queries.

Cortex was the original of these two; Mimir is its evolved successor from Grafana Labs, with simplified storage (blocks only — no legacy chunks engine), better defaults, and significantly improved query performance at scale. In 2024-2025, most new “central metrics platform” deployments choose Mimir over Cortex, and existing Cortex shops are gradually migrating.

When choosing between them, a useful rule of thumb:

• Use Thanos if your topology is “many independent Prometheus servers, give us long-term storage and a global view.” Thanos’s sidecar pattern adds minimal moving parts and its downsampling is uniquely useful for year-long dashboards.
• Use Mimir if you are building a multi-tenant metrics service where teams or customers ship metrics in via remote write and expect per-tenant quotas, isolation, and SLOs.
• Use Cortex mostly if you already run it; for new deployments, Mimir is usually the better choice.
• VictoriaMetrics is worth mentioning as a simpler single-binary alternative, especially for teams without dedicated SREs.

Figure 3.5: Long-term storage architectures — Thanos vs. Mimir/Cortex

flowchart TB
    subgraph Thanos["Thanos (sidecar pattern)"]
        direction LR
        TP[Prometheus] --> TS[Thanos Sidecar]
        TS -->|upload 2h blocks| TOS[(Object Store<br/>S3 / GCS)]
        TQ[Thanos Querier] --> TS
        TQ --> TSG[Store Gateway]
        TSG --> TOS
        TC[Compactor<br/>downsample 5m / 1h] --> TOS
    end

    subgraph Mimir["Mimir / Cortex (remote-write platform)"]
        direction LR
        MP[Prometheus / Agent] -->|remote_write| MD[Distributor]
        MD --> MI[Ingester<br/>head + WAL]
        MI -->|flush blocks| MOS[(Object Store)]
        MQF[Query Frontend<br/>+ cache] --> MQ[Querier]
        MQ --> MI
        MQ --> MSG[Store Gateway]
        MSG --> MOS
    end

Key Takeaway: Local TSDB is fast, durable, and short-term: WAL + head + 2-hour blocks + compaction + retention. For longer horizons and multi-cluster scale, remote write ships samples to a backend like Thanos (sidecar + downsampling for existing Prometheus), Mimir (multi-tenant remote-write platform), or Cortex (legacy predecessor of Mimir).

Chapter Summary

Prometheus is a small distributed system bundled into a single binary. Its retrieval subsystem pulls metrics over HTTP from targets discovered via service discovery; its TSDB persists those samples through a WAL into 2-hour blocks that compact into larger blocks over time; its HTTP API answers PromQL queries; and its rules engine evaluates recording rules and fires alerts to an external Alertmanager.

The pull model gives Prometheus implicit liveness, server-controlled load, and a debuggable wire protocol, at the cost of awkwardness around short-lived jobs and firewalled targets. The Pushgateway is a narrow escape hatch — appropriate for job-level batch metrics, dangerous when used for service-level SLOs. The OpenTelemetry Collector is increasingly the modern answer for genuinely push-shaped workloads.

The multi-dimensional data model — metric name plus a label set defines a unique time series — is the conceptual key to PromQL and to the operational pitfalls (cardinality explosion) that come with mishandling labels. The OpenMetrics text format is intentionally simple enough to debug with curl and write with printf.

Underneath, the TSDB is one of the more elegant pieces of open-source storage engineering: immutable time-bounded blocks, an index built around label postings lists, XOR-compressed memory-mapped chunks, and a WAL with checkpoints for crash recovery. Remote write is how Prometheus integrates with Thanos, Cortex, and Mimir to extend retention from weeks to years and to scale to many tenants and clusters.

When something goes wrong in production — slow queries, OOMing servers, missing data, runaway cardinality — the mental model from this chapter is the map. The next chapter builds on it by exploring PromQL itself, the query language that makes all of this storage useful.

Key Terms

Chapter 4: PromQL — Querying Time-Series Data

PromQL (the Prometheus Query Language) is the lens through which every metric in a Prometheus-based observability stack becomes operational insight. Without PromQL, the millions of samples Prometheus diligently scrapes are just numbers on a disk. With it, you can express questions like “what is the 99th-percentile checkout latency per region, excluding canary pods, over the last five minutes?” in a single line. That power, however, comes with sharp edges: a misplaced label, a counter that resets at the wrong moment, or a quantile computed in the wrong order can turn a confident dashboard into a comforting lie.

Think of PromQL as a spreadsheet formula language for time. Where a spreadsheet operates on rows and columns of static values, PromQL operates on labeled streams of timestamps and floats. Every expression returns a vector, and every vector has a shape — number of series, set of labels, and a temporal extent. Master the shape, and the language follows.

Learning Objectives

By the end of this chapter, you will be able to:

• Write instant, range, and subquery PromQL expressions to answer operational questions about service health and capacity.
• Apply rate, histogram_quantile, and aggregation operators (sum, avg, topk, by, without) correctly, including the subtle ordering rules that govern quantile and ratio computations.
• Diagnose common PromQL pitfalls — counter resets across restarts, staleness markers and missing samples, and cardinality explosions caused by high-dimensional labels — and refactor queries to avoid them.

Figure 4.1: PromQL data-shape transitions

graph TD
    RAW["Raw samples in TSDB<br/>(timestamp, value, labels)"]
    SEL["Selector<br/>http_requests_total{job=&quot;api&quot;}"]
    IV["Instant Vector<br/>one sample per series at eval time"]
    RV["Range Vector<br/>append [5m] — many samples per series"]
    AGG["Aggregated Instant Vector<br/>sum/avg/topk by labels"]
    S["Scalar<br/>single number, no labels"]

    RAW --> SEL
    SEL --> IV
    IV -->|"append [duration]"| RV
    RV -->|"rate, increase, *_over_time"| IV
    IV -->|"sum by, avg by, topk"| AGG
    AGG -->|"scalar()"| S
    S -->|"comparison, arithmetic"| IV

4.1 PromQL Fundamentals

Before writing useful queries, you need a precise mental model of the data types PromQL manipulates. Confusion about these types is the single largest source of “why isn’t my query returning anything?” tickets.

Instant Vectors, Range Vectors, and Scalars

PromQL has four expression types, but for day-to-day work three of them dominate:

The crucial rule: most functions and operators that you think of as “PromQL math” require an instant vector. The comparison operator >, the binary operator +, and aggregation operators all reject range vectors. Range vectors exist almost exclusively to feed time-windowed functions like rate(), avg_over_time(), or increase().

# Instant vector: one sample per series at "now"
http_requests_total

# Range vector: every sample in the past 5 minutes per series
http_requests_total[5m]

# Scalar: just a number
0.95

# Functions transform range vectors back into instant vectors
rate(http_requests_total[5m])    # instant vector again

Analogy: an instant vector is a single Polaroid snapshot of all your series right now. A range vector is a flip-book of snapshots covering the last five minutes. Functions like rate() are the flip-book reader that summarizes the motion into a single number per series, handing you back a new Polaroid.

Selectors and Label Matchers

A selector chooses which series to retrieve. Every PromQL query starts with one. The selector has two parts: the metric name and an optional set of label matchers in {...} braces.

# All series for this metric, across every label combination
http_requests_total

# Filter by exact label match
http_requests_total{job="api", method="GET"}

# Regex match (note the =~ operator)
http_requests_total{code=~"5.."}

# Negative regex match
http_requests_total{code!~"2..|3.."}

# Negative exact match
http_requests_total{environment!="canary"}

Four matcher operators exist: = (equals), != (not equals), =~ (regex match), and !~ (regex doesn’t match). Regexes are anchored on both ends automatically — code=~"5.." matches 500, 503, and 599 but not 5000.

You can also select by the special __name__ label, which is how PromQL internally represents the metric name. This trick is occasionally useful when the metric name itself needs filtering:

{__name__=~"http_.*", job="api"}

Offset and @ Modifiers

Two modifiers let you shift queries through time. They’re the difference between “requests right now” and “requests at exactly 9:00 AM yesterday.”

The offset modifier shifts the query backwards by a relative duration:

# Current request rate
rate(http_requests_total[5m])

# Request rate from one week ago, same lookback
rate(http_requests_total[5m] offset 1w)

# Week-over-week ratio
rate(http_requests_total[5m])
  /
rate(http_requests_total[5m] offset 1w)

The @ modifier (introduced in Prometheus 2.25) pins the query to an absolute Unix timestamp, which is invaluable for reproducible alerts and post-incident analysis:

# Request rate as observed at 2026-01-15 14:30:00 UTC
rate(http_requests_total[5m] @ 1736951400)

# Combined: 5-minute rate, 1 hour before a fixed timestamp
rate(http_requests_total[5m] @ 1736951400 offset 1h)

Subqueries extend this further by letting you build a range vector out of an instant-vector expression on the fly, evaluated at a step you specify:

# Max 5-minute request rate observed over the last hour, sampled every 1m
max_over_time(
  rate(http_requests_total[5m])[1h:1m]
)

The [1h:1m] syntax says: “evaluate the inner expression every 1 minute over the last 1 hour and assemble those results into a range vector.” Subqueries are powerful but expensive — they multiply query work — so prefer recording rules for anything you run repeatedly.

Key Takeaway: Every PromQL expression has a shape: instant vector, range vector, or scalar. Most operators require instant vectors; range vectors exist to feed time-window functions like rate(). Master the shape transitions and most “why doesn’t this work?” errors disappear.

4.2 Functions and Operators

This is where PromQL goes from “selecting data” to “answering questions.” The functions in this section are the workhorses of every production dashboard and alert.

rate, irate, and increase on Counters

Counters are the most common Prometheus metric type — monotonically increasing numbers like http_requests_total or node_network_transmit_bytes_total. Raw counter values are almost never useful on their own; what you care about is how fast they’re growing. Three functions answer that question with subtly different semantics [Source: https://developers.openai.com/cookbook/examples/gpt4-1_prompting_guide].

All three operate only on counters and automatically handle counter resets — when the value drops (say, from 12345 back to 0 after a pod restart), the negative jump is ignored and only post-reset increments count. Critically, none of them work correctly on gauges; for gauges, reach for avg_over_time, max_over_time, delta, or deriv.

# Smooth requests-per-second dashboard, aggregated by service
sum by (service) (
  rate(http_requests_total[5m])
)

# Spike-detection alert: instantaneous error rate above 10%
sum by (service) (irate(http_requests_total{code=~"5.."}[1m]))
  /
sum by (service) (irate(http_requests_total[1m]))
  > 0.1

# SLO accounting: total 5xx errors and total requests over 30 days
sum(increase(http_requests_total{code=~"5.."}[30d]))
  /
sum(increase(http_requests_total[30d]))

A common stumble is dividing before aggregating. The ratio of two rates is not the rate of two ratios. Always aggregate the numerator and denominator separately, then divide [Source: https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents]:

# WRONG — produces per-instance ratios that get summed nonsensically
sum by (service) (
  rate(http_requests_total{code=~"5.."}[5m])
    /
  rate(http_requests_total[5m])
)

# RIGHT — aggregate first, then divide
sum by (service) (rate(http_requests_total{code=~"5.."}[5m]))
  /
sum by (service) (rate(http_requests_total[5m]))

Choosing the window size matters too. A good rule of thumb is 3–5× the scrape interval: with a 15s scrape, use at least [1m] for rate to have meaningful samples; [5m] is the standard dashboard window because it smooths jitter without hiding real outages. Going too short produces noisy graphs; going too long ([1h]) hides brief incidents [Source: https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices].

Figure 4.2: rate vs irate vs increase over a range vector

flowchart LR
    C["Counter samples in [5m] window<br/>t-5m … t-4m … t-3m … t-2m … t-1m … t"]

    C --> R["rate(v[5m])<br/>linear regression across<br/>all samples in window"]
    C --> I["irate(v[5m])<br/>uses only the LAST<br/>two samples"]
    C --> N["increase(v[5m])<br/>= rate(v[5m]) * 300s<br/>total increments in window"]

    R --> RO["Smooth per-second rate<br/>good for dashboards and alerts"]
    I --> IO["Spiky per-second rate<br/>good for short spike detection"]
    N --> NO["Total event count over window<br/>good for SLO budgets"]

histogram_quantile and Bucket Math

Latency, response sizes, and queue depths are typically tracked with histograms, not gauges or counters. A classic Prometheus histogram exposes three families of series for each underlying metric:

• *_bucket{le="0.1", ...} through le="+Inf" — cumulative counters of observations falling at or below each upper bound.
• *_sum — total of all observed values.
• *_count — equal to the +Inf bucket; total observation count.

The histogram_quantile() function reconstructs an approximate distribution from these buckets and linearly interpolates within the bucket that contains your target quantile [Source: https://natesnewsletter.substack.com/p/context-windows-are-a-lie-the-myth].

The math, briefly: for quantile q with cumulative counts b_i at upper bounds u_i and total count N = b_n,

1. Compute rank r = q × N.
2. Find the first bucket k where b_k ≥ r.
3. Interpolate: x_q = u_{k-1} + ((r − b_{k-1}) / (b_k − b_{k-1})) × (u_k − u_{k-1}).

This assumes a uniform distribution within each bucket, which is why bucket boundary design matters so much.

The canonical p99 latency pattern looks like this:

histogram_quantile(
  0.99,
  sum by (le, service) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

Four things must be right for this to produce a meaningful answer [Source: https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/prompt-engineering]:

1. Apply rate() to the buckets first. Buckets are counters, so you almost always want a rate of observations (or increase() for a fixed window), not the raw cumulative count since process start.
2. Preserve the le label in aggregation. If you write sum by (service) without including le, you collapse all buckets into a single value and destroy the histogram structure.
3. Aggregate before taking the quantile, never after. Quantiles are not linear; you cannot average p99s across instances.
4. Ensure all instances share the same bucket boundaries. Mixing layouts produces meaningless interpolation.

Here is the wrong-vs-right comparison in code:

# WRONG — compute per-instance p99, then sum/avg quantiles (statistically invalid)
sum by (service) (
  histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))
)

# WRONG — drops the `le` label, breaking histogram reconstruction
histogram_quantile(
  0.99,
  sum by (service) (rate(http_request_duration_seconds_bucket[5m]))
)

# RIGHT — aggregate buckets preserving `le`, then take one quantile
histogram_quantile(
  0.99,
  sum by (le, service) (rate(http_request_duration_seconds_bucket[5m]))
)

A related gotcha: if the true p99 falls within the +Inf bucket, histogram_quantile() returns +Inf (or, depending on bucket bounds, flattens at the largest finite upper bound). The fix is better bucket design — clustering boundaries tightly around your SLO threshold. If your SLO is 300ms p99, you want buckets like 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.4, 0.5, 1.0 rather than the default 0.005, 0.01, ..., 10.

For average latency, combine _sum and _count:

sum by (service) (rate(http_request_duration_seconds_sum[5m]))
  /
sum by (service) (rate(http_request_duration_seconds_count[5m]))

Native histograms (introduced as experimental in Prometheus 2.40+ and stabilized later) eliminate the le label entirely by storing the bucket structure compactly inside a single series. Querying them is simpler:

# Native histogram p99 — no `le`, no _bucket suffix
histogram_quantile(
  0.99,
  sum by (service) (
    rate(http_request_duration_seconds[5m])
  )
)

Aggregation Operators

Aggregation operators collapse many series into fewer series. They’re the workhorse of every dashboard panel. PromQL has a fixed set: sum, avg, min, max, count, count_values, stddev, stdvar, topk, bottomk, quantile, and group.

Each can be modified by by (label_list) (keep only these labels) or without (label_list) (drop these labels, keep the rest).

# Total requests per second across the whole fleet
sum(rate(http_requests_total[5m]))

# Grouped by service — one result series per service
sum by (service) (rate(http_requests_total[5m]))

# Equivalent: drop the instance and pod labels, keep everything else
sum without (instance, pod) (rate(http_requests_total[5m]))

# Top 5 noisiest services by error rate
topk(5,
  sum by (service) (rate(http_requests_total{code=~"5.."}[5m]))
)

# Average CPU usage per node
avg by (node) (rate(node_cpu_seconds_total{mode!="idle"}[5m]))

A subtle but important point: by and without are complementary. sum by (service) keeps only the service label and drops everything else. sum without (pod, instance) drops just those two labels and keeps everything else. In high-cardinality environments, without is often safer — it doesn’t accidentally hide labels you forgot to mention.

Key Takeaway: Use rate() for smooth dashboards, irate() for spike alerts, and increase() for “how many events” questions. For histograms, always rate() the buckets first, preserve le in aggregation, and compute the quantile after aggregating. For ratios, aggregate numerator and denominator separately, then divide.

4.3 Recording and Alerting Rules

PromQL queries can be slow. A histogram_quantile() over a 30-day window touching millions of series can take seconds to evaluate — far too slow for a dashboard that refreshes every 10s or an alert that fires every 30s. The fix is rules: queries that Prometheus evaluates on a fixed schedule and stores the results as either new metrics (recording rules) or alert states (alerting rules) [Source: https://sre.google/sre-book/service-best-practices/].

Recording Rules for Expensive Queries

A recording rule pre-computes a query and writes the result back into Prometheus as a new time series. The first dashboard load is slow; every subsequent load reads a single pre-computed series.

groups:
  - name: http_slos
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      - record: job:http_errors:rate5m
        expr: sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))

      - record: job:http_error_ratio:5m
        expr: |
          job:http_errors:rate5m
            /
          job:http_requests:rate5m

Notice the naming convention: level:metric:operation. The level (job, namespace, cluster) identifies the aggregation scope; the metric identifies what’s being measured; the operation describes the transformation (rate5m, histogram_quantile99). This convention is part of the Prometheus operational vocabulary and pays dividends in dashboards and downstream rules [Source: https://www.dynatrace.com/news/blog/site-reliability-done-right/].

# Without recording rule — slow, runs every dashboard refresh
histogram_quantile(0.99,
  sum by (le, service) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

# With recording rule — fast lookup of a single pre-computed series
service:http_request_duration_seconds:histogram_quantile99_5m

A few rules-of-thumb for rule chains:

• Keep chains 2–3 levels deep. Pipelines like raw → base → service → cluster work well; deeper chains become hard to debug.
• Don’t rate a rate. If job:http_requests:rate5m is already a per-second rate, writing rate(job:http_requests:rate5m[5m]) is meaningless.
• Group rules by data source and scrape interval. Co-locate rules that depend on each other so they evaluate consistently.

Alerting Rule Syntax and the for Clause

Alerting rules look like recording rules but produce alert states instead of new time series. The expr field must return an instant vector; any series in that vector with a non-zero, non-NaN value triggers an alert.

groups:
  - name: http_alerts
    interval: 30s
    rules:
      - alert: HighRequestErrorRate
        expr: job:http_error_ratio:5m > 0.05
        for: 5m
        labels:
          severity: critical
          team: payments
        annotations:
          summary: "High error rate on {{ $labels.job }}"
          description: |
            Error ratio is {{ $value | humanizePercentage }} on job
            {{ $labels.job }} (threshold 5%). See runbook for triage.
          runbook_url: "https://runbooks.example.com/HighRequestErrorRate"

The for clause is the single most important alerting-rule feature for reducing noise. It requires the alert condition to be continuously true for the specified duration before the alert fires. With for: 5m, a one-minute blip in error rate won’t page anyone; a sustained five-minute problem will.

Guidelines for for durations [Source: https://www.splunk.com/en_us/blog/learn/sre-metrics-four-golden-signals-of-monitoring.html]:

A critical anti-pattern: don’t use for to mask noisy query design. If your expression is flapping because the underlying rate window is too short, fix the query (longer rate window, more aggregation) rather than lengthening for. The for clause delays alerts; it doesn’t make them more accurate.

Best Practices for Rule Organization

At scale, rules become code. Treating them like code is the single biggest lever for keeping alerting sane [Source: https://cloud.google.com/blog/products/devops-sre/how-sre-teams-are-organized-and-how-to-get-started].

• Version control everything. Rules live in Git, reviewed by service owners and SREs, deployed by CI/CD.
• Lint in CI. promtool check rules validates syntax; custom linters enforce naming conventions, required labels (severity, team, service, runbook_url), and forbidden patterns (no topk in alerts, no high-cardinality grouping).
• Standard libraries. Share templates for common patterns: SLO burn-rate alerts with FAST and SLOW windows, four-golden-signals dashboards, exporter-specific rule packs.
• Tier alerts by severity. critical pages a human; warning opens a ticket; info logs to a channel. Every critical alert must have a runbook URL in annotations.
• Safe rollout. New rules ship as recording rules first, then as warning alerts to a non-paging channel with a long for, and only graduate to critical after they prove reliable.

A sample rule-organization layout might look like:

rules/
├── recording/
│   ├── http_aggregates.yaml      # job:http_requests:rate5m, etc.
│   ├── slo_aggregates.yaml       # service:availability:ratio_rate30d
│   └── infra_aggregates.yaml     # node:cpu:usage_ratio_avg5m
├── alerts/
│   ├── slo_burn.yaml             # multi-window SLO alerts
│   ├── infra_saturation.yaml     # disk, memory, CPU
│   └── platform_health.yaml      # control-plane symptoms
└── tests/
    └── *.test.yaml                # promtool test rules fixtures

Key Takeaway: Recording rules turn expensive queries into single-series lookups; alerting rules turn those queries into pages. Use the level:metric:operation naming convention, keep rule chains shallow, and tune the for clause to alert category — never to mask noisy queries. Treat rules as code: Git, CI lint, runbook annotations.

Figure 4.3: Recording-rule chain feeding an alert

sequenceDiagram
    participant T as "Target /metrics"
    participant P as "Prometheus scraper"
    participant R1 as "Recording rule (base)<br/>job:http_requests:rate5m"
    participant R2 as "Recording rule (ratio)<br/>job:http_error_ratio:5m"
    participant A as "Alerting rule<br/>HighRequestErrorRate"
    participant AM as "Alertmanager"

    T->>P: "scrape every 15s"
    P->>P: "write samples to TSDB"
    Note over R1: "every 30s eval interval"
    P->>R1: "read raw counters"
    R1->>R1: "sum by (job) (rate(...))"
    R1->>P: "write new series"
    Note over R2: "every 30s eval interval"
    R1->>R2: "read rate5m series"
    R2->>R2: "errors / requests"
    R2->>P: "write ratio series"
    Note over A: "every 30s eval interval"
    R2->>A: "read ratio series"
    A->>A: "expr > 0.05 sustained for 5m"
    A->>AM: "fire alert with labels + annotations"

4.4 Common Pitfalls

PromQL is a small language with a lot of footguns. The pitfalls below are the ones that show up most often in incident postmortems.

Counter Resets Across Restarts

Counters should never decrease, but processes restart. When a counter resets from 12345 back to 0, every rate-family function (rate, irate, increase) detects the drop and treats it as a reset, counting only the post-reset increase. This works automatically and is one of PromQL’s most pleasant surprises.

But there are edge cases:

• Frequent restarts within a short window. A pod that restarts every 30 seconds inside a 1-minute rate window produces noisy, misleading rates. Investigate the restart loop; the metric is correctly reflecting the chaos.
• Counters that aren’t actually counters. Some metrics named _total are gauges in disguise (badly designed exporters). Rate functions silently produce nonsense. Always verify metric type via the Prometheus UI or /metrics endpoint.
• Aggregating across reset boundaries. If you sum two counter series and one resets, the sum jumps. Always apply rate() or increase() before aggregating counters:

# WRONG — sum first, then rate. Reset on one instance corrupts the sum.
rate(sum by (job)(http_requests_total)[5m:])

# RIGHT — rate per series first, then aggregate
sum by (job)(rate(http_requests_total[5m]))

Staleness Markers and Missing Samples

Prometheus 2.0+ writes an explicit staleness marker when a target disappears or a series stops being reported. Five minutes after the last sample (the default lookback delta), instant queries return no result for that series rather than the last known value. This is usually what you want — but it has consequences:

• Alerts can silently disable themselves. If your alert is up == 0 and the up series stops being reported (e.g., because the scrape config no longer targets that job), the alert never fires.
• Dashboards show gaps. Sparse metrics (events that only occur occasionally) appear missing rather than zero. Use or vector(0) or absent() to make missing data explicit:

# Detect when a critical metric is missing
absent(up{job="payments-api"})

# Default to zero when no data
sum by (service)(rate(payment_failures_total[5m])) or vector(0)

• Subqueries cross staleness boundaries. A [1h:30s] subquery includes 30-second snapshots; if data is missing for part of that hour, the inner expression evaluates to no result for those steps. Functions like avg_over_time skip those steps rather than treating them as zero.

The lookback-delta setting (default 5 minutes) controls how far back PromQL searches for the most recent sample of a series. Don’t change this without strong reason — it has cascading effects on every query in your environment.

Cardinality Explosions from High-Dimensional Labels

Cardinality — the number of unique time series — is the single largest scalability constraint in Prometheus. Each unique combination of metric name and label values is one series; each series consumes memory, disk, and query CPU. A metric with 10 labels each having 100 possible values can in principle produce 10^20 series. In practice, a few hundred thousand series per Prometheus instance is healthy; a few million is painful; tens of millions usually crashes.

Cardinality explodes when a label can take values from an unbounded set:

A common real-world failure: an exporter that uses raw request paths as labels generates one series per (method, status, path) tuple. Add /users/{id}/posts/{id} and you have one series per user per post — millions of series before lunch. The fix is to normalize at the source: bucket paths into templates (/users/:id/posts/:id), drop high-cardinality labels before exposing them, or move that information into logs and traces (where cardinality is cheap) instead of metrics.

For recording rules, the discipline tightens. A recording rule that retains pod as a grouping label persists one new series per pod per evaluation interval — a hidden multiplier. Standard practice [Source: https://www.dynatrace.com/news/blog/site-reliability-done-right/]:

# RISKY — preserves pod label, creates pod-cardinality series
- record: pod:http_requests:rate5m
  expr: sum by (pod, service)(rate(http_requests_total[5m]))

# SAFER — aggregate pod away in the recording rule
- record: service:http_requests:rate5m
  expr: sum by (service)(rate(http_requests_total[5m]))

Tools for diagnosis:

• topk(20, count by (__name__)({__name__=~".+"})) — top 20 metrics by series count.
• count(count by (label_name)(metric_name)) — distinct values for a specific label.
• prometheus_tsdb_head_series — total active series; alert when this grows fast.

Key Takeaway: Three pitfalls dominate PromQL incidents: counter resets handled by aggregating after rate() (not before); staleness handled by absent()/or vector(0) for missing data; and cardinality controlled by aggregating away unbounded labels in recording rules and never exposing raw user/path/trace IDs as label values.

Figure 4.4: How label dimensions multiply series count

graph LR
    M["http_requests_total<br/>1 metric, no labels<br/>1 series"]
    M --> L1["+ method<br/>(~10 values)<br/>10 series"]
    L1 --> L2["+ status_code<br/>(~40 values)<br/>400 series"]
    L2 --> L3["+ pod<br/>(~500 churning values)<br/>200,000 series"]
    L3 --> L4["+ request_path raw URL<br/>(unbounded, 50k+)<br/>10,000,000+ series<br/>Prometheus OOM"]
    L4 --> FIX["Fix: normalize path to template<br/>route=&quot;/users/:id/posts/:id&quot;<br/>or drop label entirely"]

Chapter Summary

PromQL is the language of operational truth in a Prometheus-based observability stack. In this chapter you learned to:

• Reason about data shape. Every PromQL expression returns an instant vector, range vector, or scalar; the shape determines which operators are valid. Range vectors exist to feed time-window functions; everything else operates on instant vectors.
• Pick the right rate function. rate(m[5m]) for dashboards and most alerts; irate(m[1m]) for spike detection; increase(m[30d]) for SLO budgets. All three handle counter resets automatically and demand counter (not gauge) inputs.
• Compute histogram quantiles correctly. rate() the buckets, sum by (le, ...) preserving the le label, then histogram_quantile() on the aggregated result. Never aggregate after quantiles, never drop le, never mix bucket layouts.
• Aggregate with intent. Use by to keep a small label set, without to drop specific labels in high-cardinality environments. Always aggregate ratios numerator-and-denominator-separately, then divide.
• Move expensive work into recording rules. Apply the level:metric:operation naming convention, keep rule chains 2–3 levels deep, and never rate a rate.
• Tune alerting rules. Use for clauses sized to alert category (1–5m for symptoms, 5–15m for saturation, longer for SLO burn). Treat rules as code: Git, CI lint, runbooks linked from annotations.
• Avoid the big pitfalls. Counter resets (rate first, then aggregate), staleness (use absent and explicit defaults), and cardinality (drop unbounded labels in recording rules; never expose user/path/trace IDs as labels).

The next chapter takes the data we now know how to query and shows how to push it into dashboards, alert receivers, and downstream systems — bringing PromQL into the operational loop of the SRE team.

Key Terms

[Source: https://sre.google/sre-book/service-best-practices/] [Source: https://www.dynatrace.com/news/blog/site-reliability-done-right/] [Source: https://natesnewsletter.substack.com/p/context-windows-are-a-lie-the-myth]

Chapter 5: OpenTelemetry Architecture: API, SDK, and Collector

OpenTelemetry is often described as a single project, but in practice it is three loosely coupled layers working in concert: a small, stable API that instrumentation code calls; a configurable SDK that turns those calls into real spans, metrics, and log records; and an out-of-process Collector that receives, processes, and forwards telemetry. Understanding how these layers fit together — and where the seams are — is the foundation for every decision you will make later about samplers, exporters, deployment topologies, and vendor choice.

This chapter zooms out from individual signals (Chapters 2–4) and looks at OpenTelemetry as a system. By the end, you should be able to read an OpenTelemetry architecture diagram, predict where a given piece of configuration belongs, and choose a Collector deployment topology appropriate for the workload in front of you.

Learning Objectives

By the end of this chapter, you will be able to:

• Differentiate the OpenTelemetry API, SDK, and Collector and explain why the API/SDK split is a deliberate architectural decision rather than an accident of history.
• Trace telemetry data from an instrumented application through SDK exporters and the Collector to a backend, and identify which configuration knob lives at each layer.
• Select an appropriate Collector deployment topology — agent (sidecar or DaemonSet), gateway (centralized service), or a hybrid of both — for a given workload, citing concrete trade-offs.
• Compare OTLP transport variants (gRPC, HTTP/protobuf, HTTP/JSON) and pick one based on infrastructure constraints.
• Distinguish OpenTelemetry distributions (core, contrib, vendor, custom) and know when to build your own with the OpenTelemetry Collector Builder (ocb).

Figure 5.1: OpenTelemetry three-layer architecture and OTLP data flow

flowchart TD
    subgraph App["Application Process"]
        Lib[Library Code<br/>depends on API only]
        Code[Application Code<br/>depends on API only]
        API[OpenTelemetry API<br/>Tracer / Meter / Logger interfaces]
        SDK[OpenTelemetry SDK<br/>samplers + processors + exporters + resource]
        Lib --> API
        Code --> API
        API --> SDK
    end
    SDK -->|"OTLP/gRPC :4317 or OTLP/HTTP :4318"| Col
    subgraph Col["OpenTelemetry Collector (out-of-process)"]
        Recv[Receivers<br/>OTLP, Prometheus, filelog]
        Proc[Processors<br/>batch, memory_limiter, k8sattributes, tail_sampling]
        Exp[Exporters<br/>OTLP, vendor-specific]
        Recv --> Proc --> Exp
    end
    Exp -->|"OTLP or vendor protocol"| BE[("Backend<br/>Prometheus / Tempo / Loki / Vendor SaaS")]

1. API vs SDK vs Collector

OpenTelemetry’s most important architectural decision is splitting instrumentation surface from pipeline implementation, and then separating both from the out-of-process telemetry agent. Each layer has a distinct audience, a distinct release cadence, and a distinct dependency footprint [Source: https://opentelemetry.io/docs/concepts/components/].

1.1 The API: a stable interface for instrumentation

The API is what library authors and application code import. It defines types like TracerProvider, Tracer, Span, MeterProvider, Meter, LoggerProvider, and Logger, along with global access points (GlobalOpenTelemetry.getTracer(...) in Java, opentelemetry.trace.get_tracer(...) in Python). Crucially, the API defines interfaces only — it does not know about exporters, samplers, batching, OTLP, or any backend [Source: https://opentelemetry.io/docs/specs/otel/].

Think of the API as the plug on the back of an appliance. The shape of the plug is standardized and changes very slowly. Whether the wall socket is connected to a hydroelectric dam, a solar panel, or nothing at all is not the appliance’s problem.

A library that emits a span looks like this in Java:

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.api.trace.Span;

private static final Tracer tracer =
    GlobalOpenTelemetry.getTracer("com.example.library", "1.0.0");

void doWork() {
    Span span = tracer.spanBuilder("doWork").startSpan();
    try {
        span.setAttribute("foo", "bar");
        // library logic
    } finally {
        span.end();
    }
}

Note what is missing: no exporter, no endpoint, no sampling rate, no environment variable parsing. The library cannot accidentally pull in a vendor SDK, gRPC client, or HTTP exporter as a transitive dependency [Source: https://opentelemetry.io/docs/languages/java/instrumentation/].

If no SDK is registered, the API returns no-op implementations. Spans are created but never recorded; metric updates evaporate; log records are dropped. The cost is a few function calls and a small allocation — safe to leave instrumentation enabled even in latency-sensitive paths [Source: https://opentelemetry.io/docs/specs/otel/].

1.2 The SDK: a configurable pipeline implementation

The SDK is what application developers wire up at startup. It replaces the API’s no-op providers with concrete implementations that actually record data, apply samplers, run processors, and call exporters [Source: https://opentelemetry.io/docs/specs/otel/].

A minimal Java SDK initialization looks like this:

OtlpGrpcSpanExporter exporter = OtlpGrpcSpanExporter.builder()
    .setEndpoint("http://collector:4317")
    .build();

SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
    .addSpanProcessor(BatchSpanProcessor.builder(exporter).build())
    .setResource(Resource.getDefault().toBuilder()
        .put("service.name", "my-service")
        .build())
    .build();

OpenTelemetrySdk.builder()
    .setTracerProvider(tracerProvider)
    .buildAndRegisterGlobal();

The SDK owns four moving parts:

• Samplers (e.g., AlwaysOn, ParentBased, TraceIdRatioBased) decide whether to record a span.
• Processors (SimpleSpanProcessor, BatchSpanProcessor, metric readers, log record processors) buffer, batch, enrich, or filter data.
• Exporters serialize data and send it somewhere — OTLP, Jaeger, Zipkin, Prometheus, or a vendor-specific endpoint.
• Resource describes the entity producing telemetry (service.name, host.name, deployment.environment).

1.3 Why the split matters

The API/SDK split is not bureaucratic — it directly enables vendor neutrality:

A widely used HTTP client library depending only on opentelemetry-api adds essentially zero weight and zero opinion about your backend. The same library can be used in an app that exports to Jaeger, in an app that exports to a SaaS vendor, and in an app that disables telemetry entirely — without recompilation [Source: https://opentelemetry.io/docs/concepts/components/].

1.4 The Collector: an out-of-process pipeline

The Collector is a separate binary written in Go that runs outside your application. It speaks OTLP (and many other protocols) on its receivers, runs processors on the in-memory pipeline, and sends data out through exporters [Source: https://opentelemetry.io/docs/collector/].

receivers  →  processors  →  exporters

Figure 5.4: Collector pipeline anatomy

flowchart LR
    subgraph Receivers
        R1[OTLP gRPC :4317]
        R2[OTLP HTTP :4318]
        R3[Prometheus scrape]
        R4[filelog tail]
    end
    subgraph Processors
        P1[memory_limiter<br/>backpressure]
        P2[k8sattributes<br/>add pod/namespace]
        P3[tail_sampling<br/>keep errors + slow]
        P4[batch<br/>group for efficiency]
        P1 --> P2 --> P3 --> P4
    end
    subgraph Exporters
        E1[OTLP to backend]
        E2[Vendor exporter]
        E3[debug / logging]
    end
    R1 --> P1
    R2 --> P1
    R3 --> P1
    R4 --> P1
    P4 --> E1
    P4 --> E2
    P4 --> E3

Why move logic out of the application?

• Decoupling deploys. Sampling, redaction, and backend routing change without rebuilding every service.
• Aggregation and batching. A Collector seeing traffic from hundreds of pods can build larger, more efficient batches than any single SDK exporter.
• Vendor portability. Apps export OTLP; the Collector translates to whatever backend you happen to use this quarter.
• Centralized policy. Drop secrets, rewrite attributes, enforce per-tenant limits in one place.

Key Takeaway: OpenTelemetry deliberately separates a stable instrumentation surface (API) from a configurable in-process pipeline (SDK) from an out-of-process aggregator (Collector). Libraries depend only on the API, applications own the SDK, and operations teams own the Collector — each layer can evolve independently without breaking the others.

2. Cross-Language Architecture

OpenTelemetry promises a consistent mental model across more than a dozen languages. The architecture above repeats almost identically in Java, Python, Go, .NET, Node.js, Ruby, PHP, Rust, C++, Swift, and others. What ties them together is a shared specification, a shared set of semantic conventions, and a shared wire protocol (OTLP) [Source: https://opentelemetry.io/docs/specs/otel/].

2.1 Language support matrix and stability levels

Each language SIG (Special Interest Group) implements the spec at its own pace. The OpenTelemetry project tracks stability per signal per language: a language might be GA for traces, beta for metrics, and experimental for logs. This matters when you adopt OpenTelemetry in a polyglot environment — a Java service might emit fully GA telemetry while a sibling Node.js service is still using a beta logs SDK [Source: https://opentelemetry.io/docs/languages/].

A high-level snapshot (consult the docs for current status):

The pattern: traces stabilized first, metrics followed, logs are the most recent and still maturing in some languages.

2.2 Semantic conventions: the lingua franca

If every team picks its own attribute names — http.statusCode vs http_status vs httpResponse.code — your “vendor-neutral” telemetry becomes useless. Semantic conventions are the OpenTelemetry project’s standardized vocabulary for resource and span attributes [Source: https://opentelemetry.io/docs/concepts/semantic-conventions/].

Examples:

• service.name, service.namespace, service.instance.id
• host.name, host.id, host.arch
• deployment.environment (e.g., production, staging)
• http.request.method, http.response.status_code, url.full
• db.system, db.statement, db.name
• messaging.system, messaging.destination.name
• cloud.provider, cloud.region, k8s.namespace.name, k8s.pod.name

The payoff: a dashboard, alert, or query written against http.response.status_code works identically whether the data came from a Java service, a Go service, or a Python service. Backends like Grafana, Datadog, Honeycomb, and Tempo can build out-of-the-box visualizations because they know exactly what to look for.

Analogy: semantic conventions are to telemetry what HTTP status codes are to the web. Without them, every server could invent its own “the page worked” signal; with them, every browser, proxy, and dashboard knows what 200 means.

2.3 OTLP: the wire protocol that ties it together

OTLP (OpenTelemetry Protocol) is the bridge between SDKs, Collectors, and OTLP-compatible backends. It defines:

• A protobuf schema (ExportTraceServiceRequest, ResourceSpans, ScopeSpans, Span, plus equivalents for metrics and logs).
• A small set of service methods (TraceService.Export, MetricsService.Export, LogsService.Export).
• Three transport bindings: OTLP/gRPC, OTLP/HTTP/protobuf, and OTLP/HTTP/JSON [Source: https://opentelemetry.io/docs/specs/otlp/].

The structure of a trace export request is layered:

ExportTraceServiceRequest
  └── ResourceSpans            (one per Resource, e.g., per service)
       ├── Resource            (attributes like service.name)
       └── ScopeSpans          (one per instrumentation library)
            ├── InstrumentationScope (name, version)
            └── Span[]         (trace_id, span_id, attributes, events, status)

The same proto messages are used across all three transports — only the encoding and HTTP/gRPC framing differ [Source: https://github.com/open-telemetry/opentelemetry-proto].

2.4 gRPC vs HTTP/protobuf vs HTTP/JSON

Practical guidance:

• Default to OTLP/gRPC on port 4317 in Kubernetes and other modern infra where you control the network path [Source: https://opentelemetry.io/docs/specs/otel/protocol/exporter/].
• Switch to OTLP/HTTP/protobuf on port 4318 when traversing proxies or load balancers that don’t handle gRPC well, or backends that only expose HTTP endpoints.
• Use OTLP/HTTP/JSON for browsers (front-end telemetry), debugging with curl, or low-volume use cases where wire overhead doesn’t matter.

Default endpoints for OTLP/HTTP:

• POST /v1/traces
• POST /v1/metrics
• POST /v1/logs

A common configuration mistake is mismatching protocol and port:

# WRONG: gRPC port with HTTP exporter
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4317
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

This produces errors like “unexpected response code” or “transport error.” Use grpc with 4317 or http/protobuf with 4318, not a cross [Source: https://opentelemetry.io/docs/specs/otlp/].

Figure 5.3: OTLP export request flow across transport variants

sequenceDiagram
    participant SDK as SDK Exporter
    participant Col as Collector OTLP Receiver
    Note over SDK,Col: OTLP/gRPC on :4317
    SDK->>Col: HTTP/2 frame: TraceService.Export(ExportTraceServiceRequest, protobuf)
    Col-->>SDK: ExportTraceServiceResponse (may include partial_success)
    Note over SDK,Col: OTLP/HTTP/protobuf on :4318
    SDK->>Col: "POST /v1/traces  Content-Type: application/x-protobuf"
    Col-->>SDK: "200 OK  protobuf body (partial_success)"
    Note over SDK,Col: OTLP/HTTP/JSON on :4318
    SDK->>Col: "POST /v1/traces  Content-Type: application/json"
    Col-->>SDK: "200 OK  JSON body (partial_success)"
    Note over SDK,Col: Retry only on UNAVAILABLE / 5xx / 429 with backoff

2.5 Partial success and retry semantics

Both gRPC and HTTP OTLP support partial success: the response carries a partial_success field with rejected_spans (or rejected data points / log records) and an error_message. The transport-level status is still success — rejected items are usually permanently bad (e.g., rate-limited, malformed) and should not be retried [Source: https://opentelemetry.io/docs/specs/otlp/].

Retry policy is signaled by the transport status:

• Retry on UNAVAILABLE, DEADLINE_EXCEEDED, RESOURCE_EXHAUSTED (gRPC) or HTTP 5xx, 429 (with backoff). Use exponential backoff with jitter.
• Do not retry on INVALID_ARGUMENT, UNAUTHENTICATED, PERMISSION_DENIED (gRPC) or HTTP 400, 401, 403. The data is bad; retrying won’t help.

Key Takeaway: OpenTelemetry’s cross-language consistency rests on three pillars: the specification (which defines API/SDK semantics per signal), semantic conventions (a shared vocabulary so any backend can interpret data from any language), and OTLP (the wire protocol that carries telemetry between SDK, Collector, and backend across three transport variants).

3. Collector Deployment Topologies

The Collector is the most operationally flexible piece of OpenTelemetry. The same binary, with different configuration, can be deployed as a sidecar inside a pod, as a DaemonSet per node, or as a centralized fleet of pods behind a service. Most production Kubernetes environments combine more than one of these patterns [Source: https://opentelemetry.io/docs/collector/deployment/].

Figure 5.2: Agent, gateway, and hybrid Collector topologies

flowchart TD
    subgraph AgentMode["Agent topology (DaemonSet)"]
        direction TB
        A_App1[App Pod<br/>Node 1]
        A_App2[App Pod<br/>Node 2]
        A_Ag1[Collector Agent<br/>Node 1]
        A_Ag2[Collector Agent<br/>Node 2]
        A_BE[(Backend)]
        A_App1 -->|"OTLP localhost"| A_Ag1
        A_App2 -->|"OTLP localhost"| A_Ag2
        A_Ag1 --> A_BE
        A_Ag2 --> A_BE
    end
    subgraph GatewayMode["Gateway topology (Deployment)"]
        direction TB
        G_App1[App Pod<br/>Node 1]
        G_App2[App Pod<br/>Node 2]
        G_GW[Gateway Collectors<br/>centralized Deployment + Service]
        G_BE[(Backend)]
        G_App1 -->|"OTLP cluster Service"| G_GW
        G_App2 -->|"OTLP cluster Service"| G_GW
        G_GW --> G_BE
    end
    subgraph HybridMode["Hybrid topology (recommended)"]
        direction TB
        H_App1[App Pod<br/>Node 1]
        H_App2[App Pod<br/>Node 2]
        H_Ag1[Agent<br/>Node 1]
        H_Ag2[Agent<br/>Node 2]
        H_GW[Gateway Collectors<br/>tail sampling + tenant routing + auth]
        H_BE[(Backend)]
        H_App1 -->|"OTLP localhost"| H_Ag1
        H_App2 -->|"OTLP localhost"| H_Ag2
        H_Ag1 -->|"OTLP cross-node"| H_GW
        H_Ag2 -->|"OTLP cross-node"| H_GW
        H_GW --> H_BE
    end

3.1 Agent mode (sidecar or DaemonSet)

In agent mode, a Collector lives next to the workload:

• Sidecar: one Collector container per application pod, sharing localhost. Useful for per-service custom pipelines or strict tenant isolation between pods on the same node. Expensive at scale.
• DaemonSet: one Collector per Kubernetes node, exposed on the node’s IP or a host-network port. Apps on that node send telemetry to the local Collector.

Agents excel at:

• Node-local enrichment — adding k8s.pod.name, k8s.namespace.name, host.name from a position where that information is cheap to obtain.
• Host-level signals — scraping kubelet/cAdvisor metrics, reading container logs from /var/log/containers/*.log, collecting host CPU and memory.
• Minimal application latency — apps export to localhost or a node-local address with no cross-node hop.
• Isolated failure domain — if one agent dies, only its node’s telemetry is affected.

A minimal agent receivers/processors/exporters chain:

receivers:
  otlp:
    protocols:
      grpc:
      http:
  filelog:
    include: [ /var/log/containers/*.log ]
  kubeletstats:
    collection_interval: 10s
processors:
  memory_limiter:
    limit_percentage: 75
  k8sattributes:
    auth_type: serviceAccount
  batch:
    timeout: 5s
    send_batch_size: 8192
exporters:
  otlp:
    endpoint: otel-gateway.observability.svc.cluster.local:4317
    tls:
      insecure: true
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlp]

3.2 Gateway mode (centralized service)

In gateway mode, one or more Collectors run as a Deployment behind a Kubernetes Service (or external load balancer). Apps — or, more commonly, agents — send telemetry to this central fleet [Source: https://opentelemetry.io/docs/collector/deployment/].

Gateways excel at:

• Tail-based sampling. Sampling that depends on the full trace (e.g., “keep all traces with errors” or “keep all traces over 500 ms”) requires seeing every span in the trace. A node-local agent only sees spans for pods on its node; a gateway can see the whole trace [Source: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor].
• Efficient batching and compression. Aggregating from many sources lets the gateway build large batches that compress well and amortize backend connection overhead.
• Centralized auth and secrets. API keys, OAuth tokens, mTLS certs for the backend live in one place rather than on every node.
• Multi-tenant routing. The gateway is the natural place to route per-team or per-tenant traffic to different backends, apply per-tenant quotas, and validate tenant tokens.

For correct tail sampling across multiple gateway replicas, you need trace-ID-aware load balancing so that all spans for a given trace ID hit the same gateway pod. The loadbalancing exporter or an L7 load balancer with consistent hashing is the typical solution.

A gateway pipeline with tail sampling:

processors:
  memory_limiter:
  batch:
  tail_sampling:
    decision_wait: 5s
    num_traces: 50000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: default
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

3.3 Agent vs gateway: trade-off summary

3.4 Hybrid: agent + gateway

For most non-trivial Kubernetes clusters, the recommended pattern is DaemonSet agent + Deployment gateway:

1. DaemonSet agent receives OTLP from local pods, scrapes Prometheus and kubelet endpoints, tails container logs, adds Kubernetes metadata, and forwards OTLP to the gateway. Resilient, node-local, and cheap.
2. Deployment gateway receives OTLP from agents (and any clients that prefer to bypass the agent), runs tail sampling and tenant routing, authenticates to backends, and handles retries and backpressure.

This hybrid topology gives you the best of both worlds: node-local enrichment plus host signals from the agent, and centralized sampling, auth, and routing from the gateway. It also gives you a natural choice of where to bound failure: if the gateway is overloaded, agents buffer locally (memory or file_storage extension) until pressure subsides [Source: https://opentelemetry.io/docs/collector/deployment/].

3.5 The OpenTelemetry Operator

On Kubernetes, the OpenTelemetry Operator provides CRDs that let you declare topology declaratively [Source: https://github.com/open-telemetry/opentelemetry-operator]:

• OpenTelemetryCollector with mode: daemonset → agent per node.
• OpenTelemetryCollector with mode: sidecar → injected into matching pods.
• OpenTelemetryCollector with mode: deployment → centralized gateway.
• Instrumentation CR → auto-injects SDK and configures endpoint for Java, Python, Node.js, .NET, and Go workloads.

The Operator lets you treat Collector topology as declarative configuration, just like Deployments and Services.

Key Takeaway: Agent mode optimizes for node-local collection, host signals, and isolated failure domains; gateway mode optimizes for tail sampling, centralized auth, and multi-tenant routing. Most production Kubernetes clusters end up running both — a DaemonSet agent for cheap node-local work and a Deployment gateway for heavy centralized work.

4. Distributions and Builds

The Collector is more than one binary. The OpenTelemetry project ships distributions — pre-built bundles of receivers, processors, exporters, and extensions — and provides a builder tool for assembling your own [Source: https://opentelemetry.io/docs/collector/].

4.1 otelcol vs otelcol-contrib

The two flagship distributions:

• otelcol (core): the minimal, stable distribution. Includes the most common, well-tested receivers (OTLP, Prometheus), processors (batch, memory_limiter, attributes), and exporters (OTLP, OTLPHTTP, debug). Recommended for production pipelines that don’t need exotic components.
• otelcol-contrib: the kitchen-sink distribution. Includes hundreds of components: vendor-specific exporters (Datadog, Splunk, New Relic, Honeycomb, AWS, GCP, Azure), specialized receivers (kubeletstats, filelog, hostmetrics, redis, mysql), advanced processors (tail_sampling, transform, routing, k8sattributes), and many more.

For most teams getting started, otelcol-contrib is the practical default — almost every production pipeline ends up needing at least one component that lives in contrib (k8sattributes, tail_sampling, filelog, etc.).

4.2 Building custom distributions with ocb

The OpenTelemetry Collector Builder (ocb) is a CLI that lets you assemble your own distribution from a manifest. The motivation:

• Supply-chain hygiene. Ship only the components you actually use. Smaller binary, smaller attack surface, faster startup.
• Compliance. Some organizations require an inventory of every Go module in production binaries.
• Performance. Fewer registered components = less startup overhead, smaller config schema, fewer plugins competing for the same resources.

A builder manifest (manifest.yaml) looks like this:

dist:
  name: my-otelcol
  description: Custom OpenTelemetry Collector for ACME Corp
  output_path: ./dist
  otelcol_version: 0.95.0

receivers:
  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.95.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver v0.95.0

processors:
  - gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.95.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/k8sattributesprocessor v0.95.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor v0.95.0

exporters:
  - gomod: go.opentelemetry.io/collector/exporter/otlpexporter v0.95.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter v0.95.0

Then:

ocb --config manifest.yaml

The result is a single Go binary containing exactly the components listed — nothing more.

Analogy: otelcol-contrib is like a Linux distribution shipped with every package preinstalled. Convenient, but most servers only need a handful. A custom build with ocb is the equivalent of apt install only the packages you need, on a minimal base image.

4.3 Vendor-specific distributions

Several vendors ship their own Collector distributions, sometimes called agents or contribs:

• AWS Distro for OpenTelemetry (ADOT) — bundles upstream Collector with AWS-specific exporters (X-Ray, CloudWatch, AMP) and is supported by AWS.
• Splunk OpenTelemetry Collector — pre-configured for Splunk Observability Cloud, with Splunk-tuned defaults.
• Datadog Agent (OpenTelemetry mode) — Datadog’s agent can ingest OTLP and forward to Datadog with its own pipeline.
• Grafana Agent / Alloy — Grafana Labs’ distribution focused on Grafana Cloud and the LGTM stack.

Vendor distributions still respect the API/SDK/OTLP boundary. Your applications continue to emit standard OTLP; only the Collector itself is vendor-flavored. Switching vendors is largely a Collector configuration change — instrumentation code stays put. This is the practical payoff of the architecture in section 1.

4.4 Choosing a distribution

A decision matrix for picking a distribution:

Key Takeaway: OpenTelemetry distributions are pre-assembled bundles of Collector components. Use otelcol-contrib for most production pipelines, vendor distributions when you want first-party support, and ocb-built custom distributions when supply-chain hygiene or binary size matter. In all cases, applications continue to emit standard OTLP — the distribution choice is a Collector concern, not an instrumentation concern.

Chapter Summary

OpenTelemetry is three layers, not one: a stable API that instrumentation code calls, a configurable SDK that turns those calls into real telemetry, and an out-of-process Collector that aggregates, processes, and forwards data. The API/SDK split is what makes vendor-neutral instrumentation possible — libraries can depend only on the API, and applications choose backends at deployment time without recompiling.

Across more than a dozen languages, three pillars give OpenTelemetry consistency: the specification that defines API/SDK semantics, semantic conventions that standardize attribute names so any backend can interpret any signal, and OTLP — the wire protocol with three transport variants (gRPC on 4317, HTTP/protobuf on 4318, HTTP/JSON on 4318) — that carries telemetry between SDKs, Collectors, and backends.

Collector topology is the most operationally flexible knob. Agents (sidecars or DaemonSets) excel at node-local enrichment, host signals, and isolated failures. Gateways (centralized Deployments) excel at tail-based sampling, centralized auth, and multi-tenant routing. Most non-trivial Kubernetes deployments combine both — a DaemonSet agent for cheap local work and a Deployment gateway for heavy centralized work, often managed by the OpenTelemetry Operator.

Finally, distributions package the Collector for different audiences: otelcol for minimal stable pipelines, otelcol-contrib for the practical default, vendor distributions for first-party support, and ocb-built custom distributions for supply-chain hygiene. Regardless of distribution, applications emit standard OTLP — keeping the seam between application code and operations exactly where the architecture intends.

In the next chapter, we’ll dive deeper into instrumenting applications: auto-instrumentation versus manual instrumentation, language-specific patterns, and how to evolve from zero-touch coverage to high-value custom spans and metrics.

Key Terms

Chapter 6: Instrumentation: Manual, Automatic, and Zero-Code

Instrumentation is the act of teaching your code to talk about itself. Before any dashboard can be drawn, before any alert can fire, before any trace can be visualized, some agent — your code, a runtime hook, or the Linux kernel — must produce a signal. OpenTelemetry recognizes three broad strategies for producing those signals: manual instrumentation, where developers explicitly emit spans, metrics, and logs; automatic instrumentation, where libraries are patched at runtime to emit telemetry on your behalf; and zero-code instrumentation, where an external observer (often eBPF in the kernel, or a Kubernetes Operator injecting agents) generates telemetry with no awareness from the application itself.

Think of the three approaches as the difference between an author writing a memoir, a transcriptionist sitting beside them, and a hidden microphone in the ceiling. Each captures a story; each captures it differently; each has a place.

Learning Objectives

By the end of this chapter, you will be able to:

• Choose between manual, automatic, and zero-code instrumentation for a given language and runtime.
• Add custom spans, metrics, and attributes that follow OpenTelemetry semantic conventions.
• Configure auto-instrumentation in Kubernetes using the OpenTelemetry Operator and its Instrumentation CRD.

Figure 6.1: Three instrumentation approaches compared

graph TD
    I[Application Telemetry]
    I --> M[Manual<br/>developer writes<br/>tracer.start_span]
    I --> A[Automatic<br/>runtime agent<br/>wraps libraries]
    I --> Z[Zero-Code<br/>eBPF kernel probes<br/>or Operator injection]
    M -->|effort: high| Q1[business attributes<br/>tenant.id, order.id]
    A -->|effort: low| Q2[broad HTTP/DB/RPC<br/>library coverage]
    Z -->|effort: none| Q3[polyglot, no rebuild<br/>kernel-wide visibility]

Section 1: Manual Instrumentation

Manual instrumentation puts the developer in direct control. You acquire a Tracer, Meter, or Logger from the OpenTelemetry SDK, then explicitly call methods to start spans, record measurements, or write structured log events. Manual code is verbose, but it is the only way to express domain context — concepts like order_id, tenant_id, payment.status, and feature_flag.variant that no auto-instrumenter could ever guess.

Acquiring Tracers, Meters, and Loggers

The OpenTelemetry SDK exposes three top-level provider objects: a TracerProvider, a MeterProvider, and a LoggerProvider. From each provider you obtain a named, versioned instance scoped to your library or module. The name is conventionally the import path of the instrumented package; it becomes the instrumentation.scope.name on every signal you emit, letting backends filter by which code produced the data [Source: https://opentelemetry.io/docs/specs/semconv/db/database-spans/].

// Java: acquire scoped tracer and meter
import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.api.metrics.Meter;

Tracer tracer = GlobalOpenTelemetry.getTracer("com.acme.payments", "1.4.0");
Meter meter   = GlobalOpenTelemetry.getMeter("com.acme.payments");

# Python: acquire scoped tracer and meter
from opentelemetry import trace, metrics

tracer = trace.get_tracer("acme.payments", "1.4.0")
meter  = metrics.get_meter("acme.payments")

// Node.js: acquire scoped tracer and meter
const { trace, metrics } = require('@opentelemetry/api');

const tracer = trace.getTracer('acme-payments', '1.4.0');
const meter  = metrics.getMeter('acme-payments');

Creating Spans and Recording Attributes

A span is the basic building block of a trace: a named, timed operation with a start, an end, attributes, events, and a status. The idiomatic pattern is to wrap a unit of work in a span block so it closes automatically, even on exceptions.

// Java: a span around a business operation
Span span = tracer.spanBuilder("authorize_payment")
    .setSpanKind(SpanKind.INTERNAL)
    .setAttribute("payment.method", "card")
    .setAttribute("tenant.id", tenantId)
    .startSpan();
try (Scope scope = span.makeCurrent()) {
    boolean ok = gateway.authorize(amount);
    span.setAttribute("payment.outcome", ok ? "approved" : "declined");
    if (!ok) span.setStatus(StatusCode.ERROR, "gateway declined");
} catch (Exception e) {
    span.recordException(e);
    span.setStatus(StatusCode.ERROR);
    throw e;
} finally {
    span.end();
}

# Python: idiomatic context-manager span
with tracer.start_as_current_span("authorize_payment") as span:
    span.set_attribute("payment.method", "card")
    span.set_attribute("tenant.id", tenant_id)
    try:
        approved = gateway.authorize(amount)
        span.set_attribute("payment.outcome",
                           "approved" if approved else "declined")
    except Exception as exc:
        span.record_exception(exc)
        span.set_status(trace.StatusCode.ERROR, str(exc))
        raise

// Node.js: span around an async function
await tracer.startActiveSpan('authorize_payment', async (span) => {
  span.setAttribute('payment.method', 'card');
  span.setAttribute('tenant.id', tenantId);
  try {
    const approved = await gateway.authorize(amount);
    span.setAttribute('payment.outcome', approved ? 'approved' : 'declined');
  } catch (err) {
    span.recordException(err);
    span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
    throw err;
  } finally {
    span.end();
  }
});

Attributes are key–value pairs. Their job is twofold: provide search and grouping handles in your trace UI, and feed dashboards through trace-to-metric pipelines. Use dot-namespaced keys (payment.method, not paymentMethod), follow semantic conventions when one exists, and treat your custom namespace (acme.*, payment.*) the way you would treat a public API — once dashboards depend on it, you cannot freely rename it.

Custom Metric Instruments

Where spans tell stories of individual requests, metrics tell stories of aggregate behavior. OpenTelemetry provides four core synchronous instruments and several asynchronous ones. Choose the instrument that matches the semantics of what you are counting, not just the dashboard panel you want.

# Python: a Counter and a Histogram for HTTP server work
http_requests = meter.create_counter(
    name="http.server.requests",
    description="Number of HTTP requests served",
    unit="1",
)

http_latency = meter.create_histogram(
    name="http.server.request.duration",
    description="HTTP server request latency",
    unit="s",
)

start = time.monotonic()
try:
    response = handle(request)
    http_requests.add(1, {
        "http.request.method": request.method,
        "http.response.status_code": response.status,
        "http.route": request.matched_route,
    })
finally:
    http_latency.record(time.monotonic() - start, {
        "http.request.method": request.method,
        "http.route": request.matched_route,
    })

Three rules deserve special emphasis. First, units matter: declare seconds (s), bytes (By), or 1 for dimensionless counts so backends can convert and label correctly. Second, histogram bucket boundaries are usually picked by the SDK — override them only when you know the latency profile of your service. Third, every attribute you attach to a metric multiplies cardinality; we return to that in Section 4.

Key Takeaway: Manual instrumentation gives you the only path to business-meaningful telemetry. Acquire scoped tracers and meters once per module, wrap units of work in spans with carefully chosen attributes, and pick metric instruments by semantics — Counter for monotonic totals, UpDownCounter for things that ebb and flow, Histogram for distributions, Gauge for current values.

Section 2: Automatic Instrumentation

Automatic instrumentation is the OpenTelemetry community’s answer to a hard question: “How do I get traces from libraries I did not write?” The answer differs by runtime, because each language exposes different hooks for intercepting library code without changing it.

Bytecode Injection: Java Agent and .NET Profiler

In Java, the opentelemetry-javaagent.jar attaches to the JVM at startup using the -javaagent flag. Internally, it registers a premain method via the Java Instrumentation API and uses a bytecode library such as ByteBuddy to rewrite classes as the classloader loads them, weaving span start/end logic around methods of interest [Source: https://javapro.io/wp-content/uploads/2026/02/JAVAPRO_01-2026.pdf].

OTEL_SERVICE_NAME=checkout-service \
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \
OTEL_TRACES_EXPORTER=otlp \
OTEL_LOGS_EXPORTER=otlp \
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod,service.version=2.3.1 \
java -javaagent:/opt/otel/opentelemetry-javaagent.jar -jar /app/checkout.jar

The agent ships with dozens of instrumentation modules for Servlet, Spring MVC and WebFlux, JAX-RS, gRPC, OkHttp, Apache HttpClient, JDBC, R2DBC, Hibernate, Mongo, Cassandra, Kafka, RabbitMQ, JMS, and many more. Because the rewriting happens at class load time, it works without source changes; however, the agent must attach at JVM start — you generally cannot retrofit a running JVM — and exotic custom classloaders sometimes need extra configuration [Source: https://javapro.io/wp-content/uploads/2026/02/JAVAPRO_01-2026.pdf].

.NET uses a conceptually similar mechanism via a CLR profiler, registered through CORECLR_ENABLE_PROFILING=1 and a companion DLL that injects IL into managed methods at JIT time.

Monkey-Patching: Python and Node.js

Dynamic languages do not need bytecode rewriting — they let you replace functions at runtime. Python’s auto-instrumentation uses the opentelemetry-instrument CLI to run a bootstrap before your application code; that bootstrap loads every installed opentelemetry-instrumentation-* package, each of which monkey-patches the relevant library at import time [Source: https://lumigo.io/opentelemetry/].

pip install opentelemetry-distro opentelemetry-exporter-otlp \
            opentelemetry-instrumentation-requests \
            opentelemetry-instrumentation-psycopg2 \
            opentelemetry-instrumentation-flask

OTEL_SERVICE_NAME=orders-api \
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \
opentelemetry-instrument gunicorn orders.wsgi:application

For example, the requests instrumentation replaces requests.Session.request with a wrapper that opens a client span, records HTTP attributes, calls the original function, captures the response, and ends the span. The instrumentation must run before the first import of the patched library; otherwise the cached reference will be the unpatched one.

Node.js relies on require hooks. @opentelemetry/auto-instrumentations-node registers handlers for require() (via require-in-the-middle or similar) and patches each module’s exports as it is loaded.

// tracing.js — must be required before any other module
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({}),
  instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();

NODE_OPTIONS="--require ./tracing.js" node app.js

Coverage and the Cross-Language Comparison

All three approaches expose the same observable surface — HTTP servers and clients, gRPC, SQL and NoSQL clients, message queues — but each has a different “blast radius” and quirks.

Figure 6.2: Java agent bytecode injection lifecycle

sequenceDiagram
    participant OS as OS / shell
    participant JVM as JVM
    participant Agent as otel-javaagent.jar
    participant CL as ClassLoader
    participant App as Application code
    participant Col as OTLP Collector
    OS->>JVM: java -javaagent:otel.jar -jar app.jar
    JVM->>Agent: invoke premain(Instrumentation)
    Agent->>JVM: register ClassFileTransformer (ByteBuddy)
    JVM->>App: start main()
    App->>CL: load HttpServlet, JdbcDriver, ...
    CL->>Agent: transform(class bytes)
    Agent-->>CL: rewritten bytes with span hooks
    App->>App: first request enters Servlet.service()
    App->>Col: OTLP span exported<br/>(http.server.request)

A shared environment-variable contract spans every language: OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER, OTEL_LOGS_EXPORTER, OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_TRACES_SAMPLER, OTEL_PROPAGATORS, and OTEL_RESOURCE_ATTRIBUTES. Operators love this: one ConfigMap, one set of variables, and every workload — whether Java, Python, or Node — speaks the same dialect [Source: https://lumigo.io/opentelemetry/].

When something goes wrong, two debugging axioms apply. No traces at all usually means an exporter is set to none, the endpoint protocol is mismatched (grpc vs. http/protobuf), or the bootstrap is loading too late. Duplicate spans almost always mean a library is being captured by both auto- and manual instrumentation; disable one for that library.

Key Takeaway: Auto-instrumentation gives you HTTP, gRPC, database, and queue spans for free, but how it gets injected depends on the runtime. Java rewrites bytecode at class load; Python and Node monkey-patch at import or require. All three share the same OTEL_* configuration vocabulary, which is what makes mixed-language fleets tractable.

Section 3: Zero-Code Instrumentation

“Zero-code” is the marketing label for a stronger promise than auto-instrumentation: not only does the developer not write tracing code, the developer’s build artifact is not modified at all. Two distinct technologies live under this umbrella: eBPF agents that observe processes from the Linux kernel, and the OpenTelemetry Operator that injects auto-instrumentation agents into Kubernetes pods at admission time without changing container images.

eBPF-Based Auto-Instrumentation

eBPF (extended Berkeley Packet Filter) lets you load safe, sandboxed bytecode into the Linux kernel and attach it to kernel events — syscalls, function entry/exit, tracepoints, network events — at runtime, without recompiling the kernel [Source: https://ebpf.io/what-is-ebpf/]. An eBPF observability agent typically does the following [Source: https://logz.io/glossary/what-is-ebpf/] [Source: https://www.sysdig.com/blog/the-art-of-writing-ebpf-programs-a-primer]:

1. Attaches kprobes to network kernel functions like tcp_sendmsg, tcp_cleanup_rbuf, sys_enter_sendto, and sys_enter_recvfrom to observe every byte that crosses TCP.
2. Attaches uprobes to user-space functions in shared libraries — SSL_read / SSL_write in libssl, the Go runtime’s HTTP handlers, JVM JNI entry points — to see data before encryption or after decryption.
3. Writes structured records into eBPF maps that a user-space agent drains at high frequency.
4. Reconstructs requests in user space — matching send/recv into request–response pairs, parsing HTTP headers, gRPC HTTP/2 framing, and SQL handshakes — to produce L7 metrics and OTLP spans [Source: https://www.groundcover.com/ebpf].

Figure 6.3: eBPF zero-code instrumentation dataflow

flowchart TD
    subgraph US[User space]
        A1[App A<br/>Go binary]
        A2[App B<br/>Java JVM]
        A3[App C<br/>Python]
        SSL[libssl.so]
        DS[Beyla / Pixie DaemonSet<br/>user-space agent]
    end
    subgraph K[Linux kernel]
        KP1[kprobe: tcp_sendmsg]
        KP2[kprobe: tcp_cleanup_rbuf]
        KP3[tracepoint: sys_enter_sendto]
        UP[uprobe: SSL_read / SSL_write]
        MAP[(eBPF map<br/>ring buffer)]
    end
    A1 -->|syscalls| KP1
    A2 -->|syscalls| KP3
    A3 -->|TLS calls| SSL
    SSL --> UP
    KP1 --> MAP
    KP2 --> MAP
    KP3 --> MAP
    UP --> MAP
    MAP --> DS
    DS -->|OTLP spans + RED metrics| COL[OpenTelemetry Collector]

Because the hook points are in the kernel and in shared libraries, eBPF works for every language on the host — Go, Rust, Java, Python, Node, C++, even closed-source binaries — without touching their code [Source: https://www.contrastsecurity.com/glossary/ebpf]. The output is typically the four golden signals per service (latency, traffic, errors, saturation) plus distributed traces for common protocols [Source: https://newrelic.com/blog/observability/what-is-ebpf].

Tool landscape:

OpenTelemetry Operator and Auto-Instrumentation CRDs

For Kubernetes workloads, the OpenTelemetry Operator offers a different flavor of zero-code: it lets the cluster itself inject the auto-instrumentation agents we saw in Section 2, with no changes to your container images [Source: https://lumigo.io/opentelemetry/]. The Operator defines an Instrumentation Custom Resource that describes how to instrument, then a mutating admission webhook applies that recipe when pods are annotated for injection.

# 1. The Instrumentation CRD: a reusable recipe per language
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: default-instrumentation
  namespace: production
spec:
  exporter:
    endpoint: http://otel-collector.observability:4317
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "0.1"
  resource:
    attributes:
      deployment.environment: prod
      service.namespace: payments
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
    env:
      - name: OTEL_INSTRUMENTATION_JDBC_ENABLED
        value: "true"
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
  nodejs:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest

# 2. The Deployment opts in via pod annotations
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-java: "production/default-instrumentation"
    spec:
      containers:
        - name: app
          image: registry.example.com/checkout:2.3.1

When a pod with that annotation is created, the webhook injects an init container that copies the Java agent JAR into a shared volume, then patches the application container with JAVA_TOOL_OPTIONS=-javaagent:/otel-auto-instrumentation/javaagent.jar and the appropriate OTEL_* environment variables. Python and Node.js use analogous mechanisms — wrapping the entry command or injecting startup hooks — so the application image stays untouched [Source: https://lumigo.io/opentelemetry/].

This is the easiest path to fleet-wide instrumentation in Kubernetes: write the Instrumentation CRD once, label deployments by language, and every new pod is born observable.

Limits of Zero-Code Approaches

Zero-code is not a free lunch. Compare the three strategies:

eBPF loses when business context matters (it has no idea that the request it just saw belongs to tenant=acme-corp paying order=ORD-9182), when traffic is encrypted with libraries the agent doesn’t know how to probe, when protocols are custom or binary, or when the platform isn’t Linux [Source: https://www.stackrox.io/blog/what-is-ebpf/]. The Operator-based path inherits all the limits of the SDK auto-instrumentation it ships — no domain attributes, library-version compatibility risk — but it is fantastic for getting wide coverage quickly.

Hybrid is the production-grade answer. Run eBPF for horizontal, language-agnostic baseline coverage of every workload on every node; use the Operator to inject SDK auto-instrumentation on every K8s pod; and add manual instrumentation on the critical business flows where you need tenant_id, feature_flag, payment.outcome, and the like to debug or to model SLOs.

Key Takeaway: Zero-code means two different things: eBPF probes in the kernel that see every process on a node, and the OpenTelemetry Operator injecting SDK agents into K8s pods at admission. Both eliminate code changes; neither captures business meaning. Combine them with manual instrumentation on the flows that matter.

Section 4: Semantic Conventions in Practice

Instrumentation that nobody can query is just expensive noise. Semantic conventions are OpenTelemetry’s contract that names things the same way everywhere, so a dashboard written against http.response.status_code works whether the data came from a Java agent, a Python monkey-patch, a Beyla eBPF probe, or your own manual code [Source: https://opentelemetry.io/docs/specs/semconv/db/database-spans/].

Attributes for HTTP, RPC, Database, Messaging

The conventions divide into stable attributes (safe to anchor dashboards on) and experimental ones (subject to change). The most heavily used stable attributes:

A clean dashboard query — “p95 HTTP server latency by route and status, last 30 minutes” — is just groupby(http.route, http.response.status_code) of histogram(http.server.request.duration). Because every service emits those attribute keys, the same panel works across the entire fleet, and across vendors that ingest OTLP natively [Source: https://lumigo.io/opentelemetry/].

Resource Attributes for Service Identity

Attributes describe a single signal; resource attributes describe the emitter of every signal it produces. They live in the OTLP Resource and are typically set once, at SDK initialization, via OTEL_RESOURCE_ATTRIBUTES or runtime resource detectors.

Stable resource attributes you should always set:

OTEL_SERVICE_NAME=checkout-api
OTEL_RESOURCE_ATTRIBUTES=\
service.namespace=payments,\
service.version=2.3.1,\
service.instance.id=checkout-api-7d4f-x9w2,\
deployment.environment=prod,\
k8s.namespace.name=production,\
k8s.deployment.name=checkout-api,\
k8s.pod.name=checkout-api-7d4f-x9w2,\
cloud.provider=aws,\
cloud.region=us-east-1

The OpenTelemetry Operator can fill many of these for you automatically by reading the pod’s downward API; in Kubernetes you should rarely need to set Kubernetes resource attributes by hand.

Avoiding Label and Attribute Cardinality Bombs

Cardinality is the silent killer of observability platforms. Every unique combination of attribute values produces a distinct time series for metrics and a distinct index entry for traces. Pricing, retention, query speed, and even cluster stability degrade with cardinality. The single most important instrumentation discipline is asking, before you attach an attribute, “How many distinct values can this take?”

Figure 6.4: OpenTelemetry Operator pod injection workflow

sequenceDiagram
    participant Dev as Developer
    participant API as kube-apiserver
    participant Op as OTel Operator
    participant WH as Mutating Webhook
    participant Init as Init container
    participant App as App container
    participant Col as Collector
    Dev->>API: kubectl apply Instrumentation CR<br/>(java/python/nodejs recipes)
    Op->>API: watch + cache Instrumentation CR
    Dev->>API: apply Deployment with annotation<br/>inject-java: "ns/instr-name"
    API->>WH: AdmissionReview (Pod create)
    WH->>WH: read annotation + CR recipe
    WH-->>API: patched Pod spec<br/>(init container + JAVA_TOOL_OPTIONS + OTEL_* env)
    API->>Init: schedule init container
    Init->>App: copy javaagent.jar to shared volume
    App->>App: JVM starts with -javaagent
    App->>Col: OTLP spans, metrics, logs

Figure 6.5: Cardinality explosion from a single attribute

graph LR
    subgraph BASE[Safe baseline]
        B1[method ~10]
        B2[status ~60]
        B3[route ~200]
        B1 --> BX[10 x 60 x 200<br/>= 120K series]
        B2 --> BX
        B3 --> BX
    end
    subgraph BAD[Add user.id]
        U[user.id<br/>~1,000,000]
        BX --> E[120K x 1M<br/>= 120 billion series]
        U --> E
    end
    E -->|TSDB OOM, cost spike| X[Cardinality bomb]

Rules of thumb:

Three practical mitigations:

1. Use templates, not raw values. Push http.route=/orders/{id} to your metric labels, leaving url.full for span-only attributes you debug with.
2. Drop or hash at the Collector. If you cannot prevent a high-cardinality attribute at the source, the OpenTelemetry Collector’s attributes, transform, and redaction processors can drop, truncate, or one-way-hash before export [Source: https://www.honeycomb.io/blog/opentelemetry-best-practices-data-prep-cleansing].
3. Separate metric and span schemas. It is fine — and good — for a span to carry tenant.id and order.id while the metric derived from those spans carries only tenant.tier and payment.method. Spans are sampled and indexed; metrics are aggregated forever.

The same mitigations double as PII hygiene controls. url.full, url.query, client.address, network.peer.address, and db.statement may all contain personal data — email addresses, search terms, session tokens. The Honeycomb best-practices guide recommends a layered strategy: redact at the SDK where possible, allow-list at the Collector for everything you cannot vouch for, and hash where you need to preserve cardinality without preserving identity [Source: https://www.honeycomb.io/blog/opentelemetry-best-practices-data-prep-cleansing].

Stability and Evolution

OpenTelemetry semantic conventions evolve under a three-state model: Experimental, Stable, and Deprecated. The migration from http.method (old) to http.request.method (new) is a real-world example: both names exist for a period, the new name is preferred, and Collector transform processors can normalize older signals so your dashboards survive the transition [Source: https://opentelemetry.io/docs/specs/semconv/db/database-spans/]. Anchor dashboards on Stable attributes; treat Experimental ones as opt-in extras.

Key Takeaway: Semantic conventions are what make OpenTelemetry portable. Use stable attribute names (http.request.method, http.response.status_code, db.system, db.operation) on every signal, set resource attributes once for service identity, and treat cardinality and PII as instrumentation-time concerns — once they reach the Collector, the damage is harder to contain.

Chapter Summary

This chapter mapped the three pillars of OpenTelemetry instrumentation. Manual instrumentation — acquiring named tracers and meters, creating spans, choosing the right metric instrument — is where business meaning enters telemetry; nothing else can capture tenant_id, feature_flag, or payment.outcome. Automatic instrumentation runs in three flavors depending on the runtime: Java’s -javaagent bytecode rewriting, Python’s opentelemetry-instrument monkey-patching, and Node.js’s require-hook patching, all sharing a common OTEL_* environment-variable contract. Zero-code instrumentation comes in two forms: eBPF agents that watch kernel and library hooks for every process on a host, and the OpenTelemetry Operator’s Instrumentation CRD that injects SDK agents into Kubernetes pods at admission. Each strategy has a sweet spot: eBPF for fast, broad, polyglot coverage; the Operator for fleet-wide K8s rollouts; manual for the business-critical paths. Finally, semantic conventions — stable attribute names for HTTP, RPC, database, and messaging, plus resource attributes for service identity, plus disciplined cardinality and PII hygiene — are what turn instrumentation into vendor-portable, durable observability.

In Chapter 7 we will follow the signals after they leave the application: the OpenTelemetry Collector, its receivers, processors, and exporters, and how to build telemetry pipelines that can normalize, sample, and route data from every workload in your environment to the backends that consume it.

Key Terms

Chapter 7: Distributed Tracing with OpenTelemetry

In a monolithic application, when a user clicks “Place Order” and the page hangs, a developer can attach a debugger and walk through the call stack. The execution path is linear, the variables are local, and the entire story of the request lives in one process. In a cloud-native system, that same click might cross a dozen services, two message brokers, three databases, and a handful of language runtimes — each with its own logs, clocks, and failure modes. The stack trace is gone. What replaces it is the distributed trace: a stitched-together view of how a single request flowed through the system, who called whom, how long each hop took, and where things went wrong.

OpenTelemetry (OTel) is the open standard that makes those traces portable. It defines a data model for traces, a set of propagation formats for carrying trace identity over the wire, and APIs/SDKs for emitting trace data from instrumented applications. This chapter explains how OpenTelemetry traces are structured, how trace context is propagated across service and protocol boundaries, how to instrument code so traces are useful (and not just voluminous), and how to visualize and analyze trace data in tools like Jaeger and Grafana Tempo.

Learning Objectives

By the end of this chapter, you should be able to:

• Construct and propagate trace context across service and protocol boundaries using W3C Trace Context, B3, and Jaeger formats.
• Build readable, debuggable traces with appropriate span hierarchy, naming, attributes, status, and events.
• Visualize trace data in Jaeger or Tempo to diagnose latency and error patterns, and derive RED metrics from spans.

7.1 Trace Data Model

A trace is, formally, a directed acyclic graph of spans that share a common TraceId. Each span represents one unit of work — an HTTP handler, a database query, a queue publish — and carries a name, a start/end timestamp, a parent reference, attributes, events, status, and a kind. A trace is what you get when you collect all the spans for one request and arrange them in causal order.

The mental model worth carrying is this: a span is to a trace what a stack frame is to a stack trace, except spans cross process boundaries and overlap in time when work happens in parallel.

Figure 7.1: Parent-child span tree for a single trace

flowchart TD
    A["SERVER<br/>POST /checkout<br/>checkout-svc<br/>0 - 480ms"] --> B["CLIENT<br/>payment.charge<br/>checkout-svc<br/>20 - 310ms"]
    A --> C["CLIENT<br/>inventory.reserve<br/>checkout-svc<br/>20 - 180ms"]
    B --> D["SERVER<br/>POST /charge<br/>payments-svc<br/>30 - 300ms"]
    C --> E["SERVER<br/>POST /reserve<br/>inventory-svc<br/>30 - 170ms"]
    D --> F["CLIENT<br/>db.query users<br/>payments-svc<br/>50 - 110ms"]
    D --> G["CLIENT<br/>POST gateway<br/>payments-svc<br/>120 - 290ms"]
    E --> H["CLIENT<br/>db.update stock<br/>inventory-svc<br/>40 - 160ms"]

    classDef server fill:#1f3a5f,stroke:#58a6ff,color:#fff
    classDef client fill:#3a2f5f,stroke:#a78bfa,color:#fff
    class A,D,E server
    class B,C,F,G,H client

TraceId, SpanId, and TraceFlags

Three identifiers form the backbone of every span context:

• TraceId — A 128-bit value (32 lowercase hex characters when serialized) that is globally unique per trace. Every span in a single logical request shares the same TraceId. It must not be all zeros [Source: https://www.w3.org/TR/trace-context/].
• SpanId — A 64-bit value (16 lowercase hex characters) that uniquely identifies a single span within a trace. Each new span gets a fresh SpanId; the parent’s SpanId is recorded in the child span as its parent_span_id.
• TraceFlags — An 8-bit field where only bit 0 (LSB) is currently defined: 01 means the trace is sampled/recorded, 00 means it is not [Source: https://www.w3.org/TR/trace-context/].

Together, these three fields form the SpanContext, the minimal envelope of identity that must travel between services for a trace to remain coherent. Everything else — names, attributes, events, status — is local to each span and is exported to a backend; only the SpanContext crosses the wire.

A useful analogy: think of TraceId as the conference badge color (everyone at the same event shares it), SpanId as the individual badge number (each person is unique), and TraceFlags as whether the photographer is allowed to publish your photo (sampled or not).

Span Kinds: SERVER, CLIENT, PRODUCER, CONSUMER, INTERNAL

SpanKind is a small enum that tells backends what role the span plays in a distributed conversation. Without it, a backend cannot tell whether a span represents the inbound side or the outbound side of an RPC, and dependency graphs become guesswork.

The default kind is INTERNAL. Setting the kind correctly is what allows Tempo’s service-graph processor (covered in §7.4) to pair CLIENT spans with downstream SERVER spans and build a dependency map [Source: https://grafana.com/docs/loki/latest/query/log_queries/].

Span Status, Events, and Links

Beyond identity and kind, a span carries three additional structures that turn raw timing data into a diagnosis:

• Status — One of UNSET (default), OK, or ERROR. Crucially, OTel does not infer status from HTTP codes for SERVER spans: a 4xx is generally not an error from the server’s perspective (the client made a bad request), but it is an error from a CLIENT span’s perspective. The instrumentation must set status deliberately. A span with ERROR status should usually also have a status_description string.
• Events — Timestamped, named annotations within a span, each with its own attributes. Events are the OTel-native way to record exceptions: instrumentation typically calls span.recordException(e), which adds an event named exception with exception.type, exception.message, and exception.stacktrace attributes. Events let you mark intra-span moments — “cache miss,” “retry attempt 2,” “circuit breaker opened” — without creating new spans.
• Links — A reference from one span to one or more other SpanContext values that are causally related but not the strict parent. Links are the right tool for fan-in patterns: a batch job processing 1,000 messages should have one span with 1,000 links to the producing spans, not 1,000 parent references that would force one giant trace.

Figure 7.2: Span kinds and how they pair across a distributed call

graph LR
    subgraph svcA["Service A"]
        A1["SERVER<br/>POST /checkout"]
        A2["INTERNAL<br/>validate_cart"]
        A3["CLIENT<br/>POST /charge"]
        A4["PRODUCER<br/>orders.created publish"]
        A1 --> A2
        A1 --> A3
        A1 --> A4
    end

    subgraph svcB["Service B"]
        B1["SERVER<br/>POST /charge"]
        B2["CLIENT<br/>db.query"]
        B1 --> B2
    end

    subgraph svcC["Kafka + Consumer"]
        C1["CONSUMER<br/>orders.created process"]
    end

    A3 -.->|"HTTP<br/>traceparent"| B1
    A4 -.->|"Kafka<br/>traceparent header"| C1

    classDef server fill:#1f3a5f,stroke:#58a6ff,color:#fff
    classDef client fill:#3a2f5f,stroke:#a78bfa,color:#fff
    classDef producer fill:#1f5f3a,stroke:#34d399,color:#fff
    classDef consumer fill:#5f3a1f,stroke:#fbbf24,color:#fff
    classDef internal fill:#2a2a2a,stroke:#888,color:#fff
    class A1,B1 server
    class A3,B2 client
    class A4 producer
    class C1 consumer
    class A2 internal

Key Takeaway: A trace is a graph of spans tied together by a shared TraceId; each span carries a SpanId, a kind (SERVER/CLIENT/PRODUCER/CONSUMER/INTERNAL), a status, attributes, events, and optional links. Setting these correctly is what turns raw timing into a diagnosable picture of a request.

7.2 Context Propagation

A trace only works if every service in the request path reads, preserves, and forwards the SpanContext. That cross-process handoff is called context propagation, and it is implemented by propagators — small objects that know how to inject context into outbound carriers (HTTP headers, gRPC metadata, Kafka record headers) and extract context from inbound carriers.

OpenTelemetry’s default wire format for HTTP is the W3C Trace Context standard, but the SDKs also ship propagators for B3 (Zipkin) and Jaeger to interoperate with legacy systems.

W3C Trace Context: traceparent and tracestate

The W3C spec defines two HTTP headers [Source: https://www.w3.org/TR/trace-context/]:

The traceparent header is mandatory and has a fixed, dash-separated format:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             |  |                                |                |
             |  |                                |                +-- trace-flags (01 = sampled)
             |  |                                +-- parent span-id (16 hex)
             |  +-- trace-id (32 hex, globally unique per trace)
             +-- version (currently "00")

The four fields are:

1. version — 00 today. Implementations that see a future version should follow version-specific rules; for 00, ignore anything after the four parts.
2. trace-id — 32 lowercase hex characters (16 bytes). Maps to OTel’s TraceId.
3. span-id — 16 lowercase hex characters (8 bytes). This is the sender’s span — the parent of any span the receiver creates.
4. trace-flags — 2 hex characters. Only bit 0 is defined; 01 is sampled, 00 is not.

The tracestate header is optional and carries an ordered list of vendor-specific key–value pairs:

tracestate: ot=foo:bar,ro=1,congo=t61rcWkgMzE

Rules worth knowing [Source: https://www.w3.org/TR/trace-context/]:

• Leftmost entries have highest precedence.
• Up to ~32 entries and roughly 512 total characters.
• Keys are lowercase letters, digits, and limited punctuation; vendor-prefixed keys use vendor@tenant syntax.
• Values are ASCII; no commas or equals signs in values.

tracestate is where vendors stash routing hints, custom sampling decisions, or legacy correlation IDs without disturbing the standardized traceparent.

B3 and Jaeger Legacy Formats

Many production systems pre-date W3C Trace Context. OpenTelemetry includes propagators for two legacy formats so new W3C-aware services can interoperate with them.

B3 (Zipkin) has two variants. The multi-header form uses separate headers:

X-B3-TraceId: 4bf92f3577b34da6a3ce929d0e0e4736
X-B3-SpanId: 00f067aa0ba902b7
X-B3-ParentSpanId: 5e0c63257de34c92
X-B3-Sampled: 1
X-B3-Flags: 0

The single-header form packs everything into one header:

b3: 4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-1-5e0c63257de34c92

The X-B3-Flags: 1 value (or the equivalent in the single-header form) signals debug, which forces the trace to be sampled regardless of X-B3-Sampled.

Jaeger uses a single header named uber-trace-id:

uber-trace-id: 4bf92f3577b34da6a3ce929d0e0e4736:00f067aa0ba902b7:5e0c63257de34c92:1

The fields are colon-separated: trace-id : span-id : parent-span-id : flags, where flags is a decimal number — bit 1 (value 1) = sampled, bit 2 (value 2) = debug.

The three formats encode the same logical SpanContext but differ in surface syntax, sampling semantics, and how they handle debug traces:

OpenTelemetry SDKs let you compose multiple propagators so a single service can accept and emit several formats at once. In Go, that looks like [Source: https://www.w3.org/TR/trace-context/]:

otel.SetTextMapPropagator(
    propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{},   // W3C traceparent + tracestate
        propagation.Baggage{},        // W3C baggage
        b3.New(b3.WithSingleHeader()),
        jaeger.Jaeger{},
    ),
)

On extract, each propagator tries its own header; the first one that succeeds wins (typically W3C). On inject, every enabled propagator writes its format, so the outbound request carries traceparent, b3, and uber-trace-id simultaneously. That redundancy is the migration trick: roll out W3C alongside B3/Jaeger, let downstream services read whichever they understand, then remove legacy formats once the fleet is fully W3C-aware.

The mapping between sampled flags is straightforward but must be preserved: B3 X-B3-Flags=1 (debug) and Jaeger flags & 0x02 both force-sample and should map to W3C trace-flags=01; B3 X-B3-Sampled=1 and Jaeger flags & 0x01 map directly to W3C bit 0.

Figure 7.3: traceparent propagation across a composite-propagator chain

sequenceDiagram
    participant C as Client
    participant A as Service A<br/>(W3C + B3)
    participant B as Service B<br/>(W3C only)
    participant D as Service C<br/>(B3 only)

    C->>A: HTTP request<br/>traceparent: 00-{trace-id}-{span-C}-01
    A->>A: extract context,<br/>start SERVER span {span-A}
    A->>B: HTTP request<br/>traceparent: 00-{trace-id}-{span-A}-01<br/>b3: {trace-id}-{span-A}-1<br/>baggage: user.id=12345
    B->>B: extract traceparent,<br/>start SERVER span {span-B}
    B->>D: HTTP request<br/>traceparent: 00-{trace-id}-{span-B}-01<br/>b3: {trace-id}-{span-B}-1
    D->>D: extract b3 header,<br/>start SERVER span {span-D}
    D-->>B: response
    B-->>A: response
    A-->>C: response

    Note over C,D: Same trace-id flows through<br/>all four hops despite mixed formats

Baggage for Cross-Cutting Attributes

Baggage is a separate W3C specification that travels alongside (but independently of) trace context. It is a set of key–value pairs stored on the Context — not on any one span — and propagated via the baggage HTTP header [Source: https://blog.nimblepros.com/blogs/otel/].

baggage: user.id=12345, tenant=acme-corp, feature.checkout_v2=enabled

Baggage is for cross-cutting, request-scoped context that every service might want to attach to its own spans, logs, or metrics: user ID, tenant ID, feature-flag variant, geographic region, support-case ID. Set it once at the edge, and every downstream hop can read it.

The distinction from span attributes is important:

A critical security note: untrusted clients can send any baggage header they want. Edge services should sanitize incoming baggage and apply an allowlist of accepted keys, and outbound calls to third parties should strip internal baggage to avoid leaking identifiers [Source: https://blog.nimblepros.com/blogs/otel/]. Never put secrets, tokens, or PII into baggage; treat it as data that may be stored long-term and visible to any service in the path.

Key Takeaway: Propagation is what stitches local spans into a global trace. W3C Trace Context (traceparent + tracestate) is OpenTelemetry’s default; composite propagators let you co-emit B3 and Jaeger headers for backward compatibility; and W3C Baggage carries cross-cutting request data — but never secrets — alongside the trace.

7.3 Building Useful Traces

Instrumenting a codebase is the easy part: auto-instrumentation libraries will produce spans for every HTTP request and database call out of the box. Producing traces that engineers actually use during an outage takes more thought. The difference between a noisy trace and a debuggable one usually comes down to span names, attribute hygiene, error recording, and knowing when not to create a span.

Naming Spans for Searchability

A span name is the primary identifier users see in Jaeger or Tempo. It should be low-cardinality (so backends can index and aggregate it) but descriptive enough to identify the operation.

The OpenTelemetry semantic conventions provide good defaults:

• HTTP server spans: the route template, not the raw URL. GET /users/{id} is right; GET /users/12345 is wrong because it creates a unique span name per user — a classic cardinality bomb. The numeric ID belongs in the http.target or a custom attribute.
• HTTP client spans: the method plus the route or host: GET /users/{id} or POST api.payments.svc.
• Database spans: the operation and target: SELECT users, INSERT orders. Put the full statement in the db.statement attribute, not the name.
• Messaging spans: <destination> <operation> — orders.created publish, orders.created process.
• Internal spans: a stable verb-noun describing the work: validate_cart, compute_shipping_quote.

A simple test: if you imagine 10,000 spans being created, how many distinct names should appear? Tens or low hundreds, not millions. If your span name embeds an order ID or a UUID, it is too specific.

Attributes vs. Events vs. Status

Once a span is named, the question becomes what to attach to it. The three OTel facilities serve different purposes:

Follow the OTel semantic conventions for attribute names religiously: http.method, http.route, http.status_code, db.system, db.statement, rpc.system, rpc.service, rpc.method, messaging.system, messaging.destination. Consistent naming is what allows backends to render request panels, build service graphs, and correlate signals across the LGTM stack [Source: https://newrelic.com/blog/log/enrich-logs-with-opentelemetry-collector].

Recording Exceptions and Error Status

When an instrumented function throws, two things should happen: the exception is recorded as an event, and the span’s status is set to ERROR. Most language SDKs provide a single helper:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("charge_payment") as span:
    span.set_attribute("payment.amount_cents", amount)
    try:
        gateway.charge(card, amount)
    except PaymentDeclined as e:
        span.record_exception(e)
        span.set_status(trace.StatusCode.ERROR, "payment declined")
        raise

Three habits to internalize:

1. Record then re-raise unless you are deliberately swallowing the exception. Recording without re-raising can hide bugs.
2. Status ERROR is the signal that backends use to color spans red and that Tempo’s metrics-generator uses to count errors in RED metrics (§7.4). If you forget to set it, your error rate dashboards will lie.
3. HTTP 4xx is not automatically an error on SERVER spans — the server worked correctly; the client sent a bad request. Reserve ERROR for 5xx or unhandled exceptions on the server side. On CLIENT spans, both 4xx and 5xx are typically errors from the caller’s perspective.

Avoiding Span Explosion in Tight Loops

The most common instrumentation mistake is creating a span per iteration of a loop. Imagine processing 5,000 Kafka messages in a single poll:

# Wrong — 5,001 spans per batch, blows up trace storage and indexing
with tracer.start_as_current_span("process_batch") as batch_span:
    for msg in batch:
        with tracer.start_as_current_span("process_message") as msg_span:
            msg_span.set_attribute("messaging.message_id", msg.id)
            handle(msg)

Most messages are uninteresting, but they all get the same span treatment. Better patterns:

• One span per batch, events for noteworthy items: the parent process_batch span captures the whole loop; addEvent("message.failed", attrs={"message_id": msg.id}) records only the messages that erred.
• Sample inside the loop: create child spans only for messages that fail, or for 1-in-N successes.
• Use metrics for counts, not spans: if you want to know “how many messages were processed,” a counter is the right tool, not 5,000 spans.
• Use links for fan-in: if a downstream batch span needs to reference the producers of its inputs, use links rather than parent relationships so the trace graph stays bounded.

A useful guideline: a span should represent a unit of work big enough that you might one day look at it in a UI. If you would never click on it, do not create it.

Figure 7.4: Span explosion vs. disciplined instrumentation

flowchart TB
    subgraph wrong["Wrong: 5001 spans per batch"]
        W1["process_batch SERVER"]
        W2["process_message x 5000<br/>uniform child spans<br/>blows up trace storage"]
        W1 --> W2
    end

    subgraph right["Right: 1 span + events + metrics"]
        R1["process_batch SERVER<br/>messaging.batch.size=5000"]
        R2["handle_failed_message<br/>(child span, only on error) x 3"]
        R3["events: cache.miss,<br/>retry.attempt, dlq.send"]
        R4["counter: messages_processed_total<br/>(metrics, not spans)"]
        R1 --> R2
        R1 -.-> R3
        R1 -.-> R4
    end

    classDef bad fill:#5f1f1f,stroke:#f87171,color:#fff
    classDef good fill:#1f5f3a,stroke:#34d399,color:#fff
    classDef neutral fill:#1f3a5f,stroke:#58a6ff,color:#fff
    class W1,W2 bad
    class R1,R2 good
    class R3,R4 neutral

Key Takeaway: Useful traces follow the semantic conventions, use low-cardinality span names, distinguish attributes (stable properties) from events (timestamped moments), set ERROR status deliberately, and avoid emitting a span per loop iteration — events, counters, or links usually serve better.

7.4 Trace Visualization and Analysis

Generated traces are only valuable if engineers can find and read them. Two open-source backends dominate the OpenTelemetry ecosystem: Jaeger, the original CNCF tracing project, and Grafana Tempo, the high-scale, object-storage-backed tracing backend in the Grafana LGTM stack. Both ingest OTLP, both render traces as Gantt-style waterfalls, and both produce RED-style metrics from spans — but the way they store, scale, and integrate differs.

Jaeger UI

Jaeger’s UI is a focused trace explorer. The core views are:

• Search — filter by service, operation, time range, duration, tags, and a free-form trace ID lookup. Tag search (http.status_code=500, error=true) is how on-call engineers narrow down to relevant traces during an incident.
• Trace timeline — a Gantt chart of spans, indented by parent-child relationship, with duration bars and color-coded status. Selecting a span reveals its attributes, events (including stack traces from record_exception), and SpanContext IDs.
• Trace graph / topology — a node-and-edge view of one trace, useful for understanding which services contributed to that specific request.
• System architecture / dependency graph — an aggregated view across many traces showing which services call which.
• Service Performance Monitoring (SPM) — derived RED metrics per service and operation, typically powered by an OpenTelemetry Collector spanmetrics processor sitting in front of Jaeger.

Jaeger stores traces in Cassandra, Elasticsearch, or OpenSearch (with experimental support for object storage), and is well-suited to mid-scale deployments.

Grafana Tempo

Tempo takes a different design tack: it stores spans in object storage (S3, GCS, Azure Blob) and indexes only the TraceId, making per-trace lookup cheap but full-text span search expensive. Tempo’s bet is that most trace queries come from exemplars — a metric or log line that already gives you the TraceId — and that for ad-hoc search you can use a separate index or the TraceQL query language.

Tempo’s signature feature is the metrics-generator, a component that reads spans from the ingest pipeline and emits Prometheus metrics in real time [Source: https://grafana.com/docs/loki/latest/query/log_queries/]. It runs two processors:

• span_metrics — Per-service, per-operation counters and duration histograms.
• service_graphs — Per-edge (caller → callee) metrics built by pairing CLIENT spans with SERVER spans within a configurable wait window (typically 10s).

A minimal Tempo configuration:

metrics_generator:
  processor:
    service_graphs:
      enabled: true
      wait: 10s
      max_items: 10000
      peer_attributes:
        - peer.service
        - db.name
        - messaging.system
    span_metrics:
      enabled: true
      dimensions:
        - http.method
        - http.status_code
        - rpc.system
      include_span_kinds:
        - server
        - consumer

Service Maps and Dependency Graphs

A service map is the aggregate dependency graph derived from many traces. Both Jaeger and Tempo can render one. The accuracy of the map depends entirely on the correctness of:

• service.name resource attribute on every span (consistent across services).
• SpanKind set to CLIENT/SERVER/PRODUCER/CONSUMER rather than the default INTERNAL.
• peer.service (or db.name, messaging.system) attribute on outbound spans for inferring the callee when the downstream service is not instrumented.

When all three are right, the service map shows a directed graph with edges colored by error rate, throughput, or p95 latency — a near-real-time view of system topology that is impossible to maintain by hand.

Figure 7.5: Service map derived from trace data with RED metrics on each edge

flowchart LR
    web["web<br/>SERVER"]
    gw["api-gateway<br/>SERVER + CLIENT"]
    orders["orders<br/>SERVER + CLIENT"]
    pay["payments<br/>SERVER + CLIENT"]
    inv["inventory<br/>SERVER + CLIENT"]
    db[("db<br/>peer.service")]

    web -->|"rate 920/s<br/>err 0.1%<br/>p95 95ms"| gw
    gw -->|"rate 880/s<br/>err 0.2%<br/>p95 180ms"| orders
    gw ===>|"rate 412/s<br/>err 3.4%<br/>p95 820ms"| pay
    orders -->|"rate 720/s<br/>err 0.1%<br/>p95 60ms"| inv
    orders -->|"rate 720/s<br/>err 0.0%<br/>p95 40ms"| db
    pay -->|"rate 410/s<br/>err 0.1%<br/>p95 35ms"| db
    inv -->|"rate 720/s<br/>err 0.0%<br/>p95 30ms"| db

    classDef ok fill:#1f5f3a,stroke:#34d399,color:#fff
    classDef hot fill:#5f1f1f,stroke:#f87171,color:#fff
    classDef store fill:#3a2f5f,stroke:#a78bfa,color:#fff
    class web,gw,orders,inv ok
    class pay hot
    class db store

Trace-Based Metrics: RED and USE Generation

The RED method — Rate, Errors, Duration — is the de facto SLI vocabulary for request-driven services. Tempo’s span_metrics processor emits exactly the time series needed to compute RED via PromQL:

Rate per service:

rate(tempo_span_calls_total{span_kind="server"}[5m]) by (service_name)

Error rate per service:

rate(
  tempo_span_calls_total{
    span_kind="server",
    status_code!="OK"
  }[5m]
) by (service_name)

p95 duration per service:

histogram_quantile(
  0.95,
  sum by (service_name, le) (
    rate(tempo_span_duration_seconds_bucket{span_kind="server"}[5m])
  )
)

The service_graphs processor emits parallel metrics keyed by (client, server) so you can ask the same questions per edge rather than per service — useful when a problem isn’t in a service but in a particular dependency between two services.

The USE method (Utilization, Saturation, Errors) applies to resources rather than requests, but trace spans can contribute to USE too. A span on a database client carries db.system and peer.service attributes; aggregating its duration and error counts gives you per-database errors and saturation indicators. Resource-level utilization (CPU, memory) still comes from Prometheus exporters, but traces give you USE from the consumer’s perspective: how much of a downstream resource each caller is using.

Caveats for Trace-Derived Metrics

Two pitfalls deserve emphasis [Source: https://grafana.com/docs/loki/latest/query/log_queries/]:

1. Sampling distorts rate. Head-based sampling at 10% means trace-derived metrics report roughly 1/10 of true request rate. Tail-based sampling that preferentially keeps errors and slow traces over-represents errors in the metric stream. Many teams treat trace-derived metrics as a correlation tool and keep direct application metrics as the SLO source of truth.
2. Cardinality. Each dimension (http.method, http.status_code) becomes a Prometheus label. Adding user_id or request_id to span metrics will blow up cardinality and crash your TSDB. Stick to bounded labels: service, operation, status, method, coarse path.

Jaeger SPM vs. Tempo Metrics-Generator

The two systems converge on the same goal — RED metrics from spans — but differ in placement and tightness of integration:

Either choice still benefits from running an OTel Collector in front of the tracing backend: the Collector handles batching, retries, tail-based sampling, attribute scrubbing, and multi-backend fan-out, leaving the backend to do storage and query.

Key Takeaway: Jaeger and Tempo both render traces as Gantt waterfalls and derive RED metrics from spans, but Tempo couples object storage with a built-in metrics-generator that emits both per-service and per-edge metrics into Prometheus. Trace-derived metrics are excellent for exploration and service maps, but sampling and cardinality limits mean teams should keep direct application metrics as the SLO source of truth.

Chapter Summary

Distributed tracing makes the invisible visible: it stitches the journey of one request across services, queues, and databases into a single causal graph. OpenTelemetry defines that graph as a tree of spans sharing a TraceId, each span carrying a SpanId, a kind (SERVER/CLIENT/PRODUCER/CONSUMER/INTERNAL), a status, attributes, events, and optional links. Setting the kind correctly is what enables backends to build dependency maps; setting status and recording exceptions is what makes errors searchable.

Context propagation is the wire-level mechanism that keeps spans in the same trace. The W3C Trace Context spec defines the traceparent header (version, trace-id, parent span-id, sampled flag) and the optional tracestate for vendor-specific data, and is OpenTelemetry’s default propagator. Legacy B3 (multi-header or single-header b3:) and Jaeger (uber-trace-id) propagators are available for interoperability, and composite propagators let services emit and accept multiple formats simultaneously — the foundation of a gradual migration to W3C. Baggage is a separate, complementary propagator for cross-cutting request data (user.id, tenant, feature flags) that lives on the context rather than on any one span; it must never carry secrets and should be sanitized at trust boundaries.

Useful traces require discipline: low-cardinality span names following the OTel semantic conventions, attributes for stable properties, events for timestamped moments, deliberate ERROR status on real failures, and the willingness to not emit a span per loop iteration. Events, counters, and links usually serve the high-cardinality cases better than per-iteration spans.

Visualization closes the loop. Jaeger is the classic, focused trace explorer with strong search and dependency-graph features. Grafana Tempo stores spans cheaply in object storage and ships with a metrics-generator that turns spans into Prometheus RED metrics — per service via span_metrics and per dependency edge via service_graphs. Both tools support the RED method directly and contribute to USE-style resource views. Trace-derived metrics are powerful for correlation and dependency analysis but should not replace direct application metrics for SLO accounting, because sampling and cardinality choices distort the signal.

A well-instrumented system gives an on-call engineer three things at 3 a.m.: a metric that says “errors are up,” a log line with a TraceId, and a trace that points at the exact span where the request died. Everything in this chapter exists to make that handoff work.

Key Terms

Chapter 8: Metrics Pipeline: Bridging OpenTelemetry and Prometheus

OpenTelemetry (OTel) and Prometheus were designed by different communities to solve overlapping but distinct problems. Prometheus grew up around pull-based scraping of cumulative counters with a strict text exposition format. OpenTelemetry was conceived as a vendor-neutral push pipeline with a rich metric data model that includes deltas, exponential histograms, and full resource attribution. In production observability stacks, these two worlds must coexist — usually because teams already operate Prometheus dashboards and alerts, but want to instrument applications with OTel’s polyglot SDKs.

This chapter walks the metric all the way from the instrument call inside your application through the OpenTelemetry SDK, across the wire (OTLP or Prometheus exposition), and into a Prometheus-compatible backend. By the end, you will be able to choose between the three common bridging patterns, configure aggregation temporality correctly for each, and translate OTel attributes into Prometheus labels without quietly losing data.

Learning Objectives

By the end of this chapter, you will be able to:

• Configure the OpenTelemetry SDK to emit metrics that Prometheus can scrape, or that the Collector can forward to a Prometheus-compatible backend.
• Compare cumulative versus delta aggregation temporality and pick the right one for each backend in a multi-destination pipeline.
• Translate OpenTelemetry attributes to Prometheus labels — including the metric name and unit transformations — without losing semantic meaning.
• Decide between the Prometheus SDK exporter, the Collector’s prometheus exporter, the prometheusremotewrite exporter, and Prometheus’ native OTLP receiver.

8.1 OpenTelemetry Metrics Data Model

Before you can bridge OTel metrics into Prometheus, you have to understand what OTel actually produces. OTel’s metrics data model is richer than Prometheus’ exposition format, which is precisely why the bridge is nontrivial.

Figure 8.1: OpenTelemetry metrics data model from Meter to Exporter

flowchart TD
    Meter[Meter]
    Meter --> Sync[Synchronous Instruments]
    Meter --> Obs[Observable Instruments]
    Sync --> C[Counter]
    Sync --> UDC[UpDownCounter]
    Sync --> H[Histogram]
    Obs --> OC[ObservableCounter]
    Obs --> OUDC[ObservableUpDownCounter]
    Obs --> OG[ObservableGauge]
    C --> View[View<br/>rename, filter,<br/>change aggregation]
    UDC --> View
    H --> View
    OC --> View
    OUDC --> View
    OG --> View
    View --> Agg[Aggregation<br/>Sum / LastValue /<br/>Histogram / ExpHistogram]
    Agg --> DP[Data Point<br/>value + attributes +<br/>timestamp + temporality]
    DP --> Exp[Exporter<br/>OTLP / Prometheus / stdout]

Instruments: The Six Core Shapes

An instrument is the API surface your application code calls. OpenTelemetry defines six standard instrument types, organized along two axes: synchronous versus observable, and monotonic versus non-monotonic [Source: https://www.groundcover.com/opentelemetry/opentelemetry-metrics].

Synchronous instruments are recorded inline on the hot path: requestCounter.Add(ctx, 1, attrs). Observable instruments register a callback the SDK invokes during each collection cycle: useful when reading the underlying value is cheap and you do not want to pay for it on every request.

By analogy: synchronous instruments are like ringing a bell every time something happens, while observable instruments are like a thermostat that the SDK reads on a schedule. Both produce time series, but the cost model is very different.

Views: Customizing Aggregation at the SDK

A View is an SDK configuration mechanism that lets you intercept measurements from a specific instrument and change how they are aggregated, named, or attributed before export. Views are how you do things like:

• Rename a metric (http.server.request.duration → request_latency).
• Drop high-cardinality attributes (e.g., remove user_id from a counter).
• Change the aggregation type (e.g., explicit-bucket histogram → exponential histogram).
• Configure custom bucket boundaries on a histogram.

Views are essential for the OTel-to-Prometheus bridge because they let you operate two pipelines from one instrument: an explicit-bucket histogram for the Prometheus scrape path, and an exponential histogram for an OTLP backend that supports them [Source: https://www.groundcover.com/opentelemetry/opentelemetry-metrics].

// Go SDK: configure a View to use an exponential histogram
sdkmetric.NewView(
    sdkmetric.Instrument{Name: "request_duration_seconds"},
    sdkmetric.Stream{
        Aggregation: sdkmetric.AggregationExponentialHistogram{
            MaxSize:  160,
            MaxScale: 10,
        },
    },
)

Exponential Histograms

OTel ExponentialHistogram is a compact, base-2 exponential bucket representation. Instead of fixing bucket boundaries up front, it uses a scale parameter s such that buckets approximate [2^(i/2^s), 2^((i+1)/2^s)). A higher scale gives more buckets per power of 2 — that is, more resolution.

Exponential histograms have three useful properties:

1. Automatic dynamic range. If observed values span microseconds to minutes, you do not need to pre-pick boundaries. The aggregator chooses scale automatically.
2. Bounded memory. When the bucket count would exceed MaxSize, the aggregator downscales — lowering s and merging neighboring buckets. Memory stays constant; precision degrades gracefully.
3. Separate positive, negative, and zero buckets. Useful for instruments that can take negative values (rare, but well-defined).

Compare this to a classic explicit-bucket histogram, where if your latency suddenly grows tenfold past your last bucket boundary, everything piles into the +Inf overflow bucket and your quantiles become useless. Exponential histograms degrade much more gracefully.

Key Takeaway: OpenTelemetry’s metric model expresses the shape of a measurement (counter, gauge, histogram) and the cadence (synchronous vs. observable) separately from how it is aggregated for export. Views are the configuration knob that lets you serve multiple backends — including Prometheus — from a single instrumentation point.

8.2 Aggregation Temporality

Aggregation temporality is the single most common source of OTel-to-Prometheus bugs. It is the difference between a counter that goes up forever and one that resets every export — and Prometheus’ query language assumes the former.

What Temporality Means

Temporality describes what time window each exported data point represents [Source: https://www.groundcover.com/opentelemetry/opentelemetry-metrics]. There are two options:

• Cumulative: Each point is the total since a fixed start time (usually process start).
• Delta: Each point is the change since the previous export.

Temporality applies to sums (counters) and histograms. Gauges and last-value metrics are instantaneous snapshots and do not use temporality in the same way — they are simply “the current value at observation time.”

Crucially, temporality is a property of the exported time series, not the instrument. The same Counter can be exported as cumulative to Prometheus and delta to an OTLP backend, possibly through the same Collector.

Cumulative Temporality

A cumulative Counter at time T reports the total events since the SDK started. A cumulative Histogram bucket at time T reports the total observations that fell into that bucket since the SDK started.

The backend computes deltas by subtracting successive samples:

delta = value(T2) - value(T1)

This is exactly how Prometheus’ rate() and increase() work. Pros and cons:

• Process restart: Counter drops to zero. Backends that understand cumulative semantics detect the reset (a counter that decreased) and treat subsequent samples as a new run.
• Missed export interval: No data loss. The backend just computes the rate over a longer window when the next sample arrives.

Delta Temporality

A delta Counter at time T reports events since the previous export. Each exported point is already a per-interval increment.

• Process restart: Transparent. Each export is independent; there is nothing to reset.
• Missed export interval: Irrecoverable data loss. If the export for window [T1, T2] never arrives, those events are gone — there is no later cumulative sample from which to reconstruct them.

Side-by-Side Comparison

When Each Is the Right Choice

Use cumulative when:

• Your backend is Prometheus or any system that assumes cumulative monotonic counters.
• You want robustness to transient network issues — missing one export does not lose information.
• You want behavior ops teams already know: counters that monotonically rise.

Use delta when:

• You are pushing OTLP into a backend that expects deltas or aggregates server-side.
• You want exact per-interval statistics and accept the data-loss tradeoff.
• You have many ephemeral producers (serverless, batch jobs) where cumulative reset detection is messy.

The Collector as Temporality Translator

In a real pipeline, you rarely want to pick one and force every backend to live with it. The dominant 2025 pattern is to let the OpenTelemetry Collector convert temporality per exporter:

1. Application SDK exports OTLP with AggregationTemporality = DELTA for sums and histograms.
2. Collector receives delta points.
3. Collector forwards delta to an OTLP vendor backend that prefers deltas.
4. Collector accumulates delta into cumulative for the prometheusremotewrite exporter or the /metrics endpoint Prometheus scrapes.

Figure 8.2: Collector as temporality translator — one input, two temporalities out

flowchart LR
    App[Application SDK]
    App -->|OTLP delta| Coll[OpenTelemetry Collector]
    Coll -->|cumulative| PromExp[prometheus exporter]
    Coll -->|delta| OTLPExp[otlp exporter]
    PromExp -->|scrape| Prom[(Prometheus)]
    OTLPExp -->|push| Vendor[(Vendor OTLP backend)]

Cumulative versus delta in the time domain — same underlying event stream, two export shapes:

Figure 8.3: Cumulative vs delta temporality over time

graph LR
    subgraph Cumulative
        C1["t1: 5"] --> C2["t2: 12"] --> C3["t3: 18"] --> C4["t4: 25"]
    end
    subgraph Delta
        D1["t1: +5"] --> D2["t2: +7"] --> D3["t3: +6"] --> D4["t4: +7"]
    end

The Prometheus exporters inside the Collector maintain internal state per series so they can sum deltas into a running total. This is what makes the “delta-from-apps, cumulative-to-Prometheus” pattern work end to end.

Common Temporality Bugs

If you misconfigure temporality, the symptoms in Prometheus are distinctive:

• sum(rate(my_requests_total[5m])) returns negative values: your “counter” is actually being exported as delta to Prometheus, so each scrape sees a smaller value than the last.
• Inspecting the raw series shows a counter that drops between scrapes — same root cause.
• increase() over a long window returns a value much smaller than the true event count — Prometheus interpreted scrape-to-scrape drops as resets and discarded the “missing” data.

The fix is always the same: ensure the exporter feeding Prometheus is configured to produce cumulative, regardless of what temporality the SDK and Collector use internally.

Key Takeaway: Prometheus is built around cumulative monotonic counters; OTLP push pipelines often favor delta. Pick temporality per exporter — usually delta from the SDK, cumulative at the Prometheus boundary — and let the Collector translate between them.

8.3 Bridging to Prometheus

There are four practical ways to get OpenTelemetry metrics into a Prometheus-based observability stack. Each has tradeoffs around coupling, push-versus-pull semantics, and how many moving parts you operate.

Figure 8.4: Four bridge pipelines from OTel-instrumented apps to Prometheus-compatible storage

flowchart TD
    App1[App with OTel SDK<br/>+ Prometheus exporter]
    App2[App with OTel SDK]
    App3[App with OTel SDK]
    App4[App with OTel SDK]
    Coll2[OTel Collector<br/>prometheus exporter]
    Coll3[OTel Collector<br/>prometheusremotewrite]
    App1 -->|expose /metrics| P1Slash[/metrics endpoint/]
    P1Slash -->|scrape| Prom1[(Prometheus)]
    App2 -->|OTLP push| Coll2
    Coll2 -->|expose /metrics| P2Slash[/metrics endpoint/]
    P2Slash -->|scrape| Prom2[(Prometheus)]
    App3 -->|OTLP push| Coll3
    Coll3 -->|remote_write push| Remote[(Mimir / Cortex /<br/>Thanos / VictoriaMetrics)]
    App4 -->|OTLP push| Prom4[(Prometheus<br/>OTLP receiver)]

Option 1: Prometheus Exporter in the SDK

The simplest path is to attach a Prometheus exporter directly to the OTel SDK inside your application. The SDK accumulates measurements internally and exposes them on an HTTP /metrics endpoint in the Prometheus text format.

import (
    "net/http"
    "go.opentelemetry.io/otel"
    sdkmetric "go.opentelemetry.io/otel/sdk/metric"
    "go.opentelemetry.io/otel/exporters/prometheus"
)

func initMeter() {
    exporter, err := prometheus.New()
    if err != nil {
        panic(err)
    }
    provider := sdkmetric.NewMeterProvider(
        sdkmetric.WithReader(exporter),
    )
    otel.SetMeterProvider(provider)
    http.Handle("/metrics", exporter)
    go http.ListenAndServe(":9464", nil)
}