Chapter 8 — Metrics Pipeline: Bridging OpenTelemetry and Prometheus

OpenTelemetry's metrics data model is intentionally richer than the Prometheus text exposition format. That richness — six instrument types, configurable aggregations via Views, exponential histograms, attributes-plus-resource-attributes — is precisely why the bridge is nontrivial.

1.1 The Six Core Instruments

Instruments are the API surface your code calls. They split along two axes: synchronous vs. observable (does the app push, or does the SDK pull a callback?) and monotonic vs. non-monotonic (can the value ever decrease?).

Think of synchronous instruments as ringing a bell on every event and observable instruments as a thermostat the SDK reads on a schedule. Both produce time series; the cost models differ.

1.2 Meter to Exporter — the OTel pipeline

Figure 8.1 above shows the SDK pipeline: an instrument emits a measurement; a View may rewrite or re-aggregate it; the aggregation builds a data point with a temporality stamp; an exporter pushes or exposes that point.

1.3 Views — the SDK's escape hatch

A View intercepts measurements before export. Views let you:

• Rename a metric (http.server.request.duration → request_latency).
• Drop high-cardinality attributes (e.g., strip user_id off a counter).
• Change the aggregation type (e.g., explicit-bucket → exponential histogram).
• Override bucket boundaries on a histogram.

Views are how you serve two backends from one instrument: an explicit-bucket histogram on the Prometheus scrape path, and an exponential histogram on the OTLP path.

// Go SDK: switch a histogram to exponential
sdkmetric.NewView(
    sdkmetric.Instrument{Name: "request_duration_seconds"},
    sdkmetric.Stream{
        Aggregation: sdkmetric.AggregationExponentialHistogram{
            MaxSize:  160,
            MaxScale: 10,
        },
    },
)

1.4 Exponential histograms

OTel's ExponentialHistogram uses base-2 buckets controlled by a scale s: buckets approximate [2^(i/2^s), 2^((i+1)/2^s)). Higher s = more buckets per power of two = more resolution.

Three properties make exponential histograms much better than fixed-bucket histograms:

1. Automatic dynamic range — no need to pre-pick bucket boundaries.
2. Bounded memory — when bucket count would exceed MaxSize, the aggregator downscales, merging neighbors. Memory stays constant; precision degrades gracefully.
3. Separate positive, negative, and zero buckets — supports signed observations.

Contrast with a classic histogram: when latency suddenly grows past your last bucket, every observation piles into the +Inf overflow bucket and your quantiles become meaningless.

Aggregation temporality is the single most common source of OTel-to-Prometheus bugs. It is the difference between a counter that grows forever and one that resets every export — and Prometheus' query language assumes the former.

2.1 Cumulative vs. delta — what each means

• Cumulative: each point is the total since a fixed start (usually process start).
• Delta: each point is the change since the previous export.

Temporality applies to sums (counters) and histograms. Gauges are instantaneous and ignore temporality. Crucially, temporality is a property of the exported time series, not the instrument: the same Counter can be exported as cumulative to one backend and delta to another.

2.2 What happens on restart and missed exports

2.3 The Collector as temporality translator

In real pipelines, you rarely want to pick one temporality and force every backend to live with it. The dominant 2025 pattern is to let the OpenTelemetry Collector convert temporality per exporter:

1. App SDK exports OTLP with AggregationTemporality = DELTA.
2. Collector receives delta points.
3. Collector forwards delta to an OTLP vendor backend that prefers deltas.
4. Collector accumulates delta into cumulative for prometheusremotewrite or the /metrics endpoint Prometheus scrapes.

The Prometheus exporters inside the Collector maintain per-series state to sum deltas into a running cumulative total. This is what makes the “delta-from-apps, cumulative-to-Prometheus” pattern work end to end.

2.4 Symptoms of getting it wrong

Misconfigured temporality produces distinctive symptoms in Prometheus:

• sum(rate(my_requests_total[5m])) returns negative values — your “counter” is actually delta, so each scrape sees a smaller value than the last.
• Raw series shows a counter that drops between scrapes — same root cause.
• increase() over a long window returns a value much smaller than the true event count — Prometheus interpreted drops as resets and discarded data.

The fix is always the same: ensure the exporter that feeds Prometheus emits cumulative, regardless of what the SDK and intermediate Collectors use.

Four practical bridges get OTel metrics into a Prometheus-based stack. Each trades coupling, push-vs-pull semantics, and number of moving parts.

3.1 Option 1 — SDK Prometheus exporter

Attach a Prometheus exporter directly to the OTel SDK inside your app. The SDK accumulates measurements internally and exposes them in Prometheus text format on an HTTP endpoint.

exporter, _ := prometheus.New()
provider := sdkmetric.NewMeterProvider(sdkmetric.WithReader(exporter))
otel.SetMeterProvider(provider)
http.Handle("/metrics", exporter)
go http.ListenAndServe(":9464", nil)

Pros: familiar; no Collector required. Cons: resource attributes flatten into labels (or are lost), exponential histograms are down-converted to explicit-bucket, every app couples to Prometheus' wire format.

3.2 Option 2 — Collector prometheus exporter (recommended)

App pushes OTLP to a Collector; the Collector hosts a Prometheus exporter on a port; Prometheus scrapes the Collector. You get OTLP push from apps, Prometheus pull at the storage boundary, and a Collector chokepoint to manage temporality, naming, and cardinality.

# Collector
exporters:
  prometheus:
    endpoint: "0.0.0.0:9464"
    namespace: otel
    const_labels:
      source: otel-collector
service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

3.3 Option 3 — prometheusremotewrite exporter

The Collector pushes metrics over the Prometheus remote-write protocol. The dominant pattern for shipping into Cortex, Mimir, Thanos Receive, and VictoriaMetrics. Cannot target vanilla Prometheus — vanilla Prometheus is a remote-write client, not a receiver.

This path can preserve OTel exponential histograms by mapping them to Prometheus native histograms over the wire, making it the highest-fidelity option in 2025.

3.4 Option 4 — Prometheus OTLP receiver

Recent Prometheus versions (2.47+, more complete in 3.x) include an OTLP receiver that accepts pushed OTLP metrics directly into the TSDB. Collapses the pipeline to two components, but you lose service-discovery and OTLP-specific semantics map to Prometheus equivalents with varying maturity.

otlp:
  http:
    endpoint: 0.0.0.0:4318
  grpc:
    endpoint: 0.0.0.0:4317

3.5 Choosing between the four

Once the wire path is sorted out, the next failure mode is semantic. OTel uses dotted, case-sensitive names plus explicit units. Prometheus uses underscored names with base-unit suffixes and the _total convention. A faithful bridge has to translate without quietly losing meaning.

4.1 Name conversion — the deterministic transform

1. Replace . with _ — http.server.request.duration → http_server_request_duration.
2. Replace other invalid characters (hyphens, slashes) with _.
3. Convert the value to the Prometheus base unit (e.g., ms → seconds), append the unit suffix.
4. If the instrument is a monotonic counter, append _total.

4.2 Unit suffixes

OTel uses UCUM unit codes; Prometheus uses base-unit suffixes. The exporter does the conversion if you set the OTel unit correctly.

4.3 Attributes → labels

OTel attributes (per-measurement) and resource attributes (per-SDK) both become Prometheus labels. Attribute keys go through the same dot-to-underscore conversion. The Prom exporter typically promotes service.name → job and service.instance.id → instance to mirror Prometheus' service-discovery model.

4.4 UTF-8 names in Prometheus 3.x

Prometheus 3.x supports UTF-8 metric and label names through the OpenMetrics/native protocols, which means the OTel dotted form can in principle be preserved end-to-end by quoting the metric name. In practice, dashboards, alerting rules, and recording rules predating 2024 still expect underscored names. Treat UTF-8 names as forward-looking; expect translation back to underscored names anywhere a tool was written before 2024.

4.5 Common pitfalls

• Forgetting to set the unit. Histogram values land in Prometheus without _seconds or _bytes — dashboards misinterpret them silently.
• High-cardinality resource attributes. host.id or k8s.pod.uid can produce one series per host/pod, times every metric. Use Views to drop attributes you do not need.
• Casing. Use http_status_code, not httpStatusCode.
• Reserved labels. Do not produce attributes starting with __ (Prometheus uses these internally).
• Bucket labels. The le label is added by the exporter — you cannot produce it as an OTel attribute.