Chapter 3 — Prometheus Architecture and Data Model

A single prometheus binary is best understood as a small distributed system that happens to run inside one process. Four cooperating subsystems share the binary:

• Retrieval — the scrape loop that pulls /metrics over HTTP on a schedule.
• Storage — the TSDB (head block in memory, WAL on disk, immutable 2-hour blocks).
• Query — the HTTP API and PromQL engine that answers /api/v1/query and /api/v1/query_range.
• Rules — the rule engine that evaluates recording and alerting rules and fires alerts to an external Alertmanager.

Figure 3.1: Prometheus server components and external integrations

The scrape loop state machine

For every target, Prometheus runs five steps on a timer: resolve address → HTTP GET /metrics → parse OpenMetrics → relabel → append to the head block. Whether or not the scrape succeeds, the loop also writes built-in meta-series: up, scrape_duration_seconds, and scrape_samples_scraped. These are the series you alert on when you want to alert on your monitoring itself.

Analogy: the scrape loop is a postal carrier walking the same route every 15 seconds. They do not wait for a letter; they pick up whatever is in the mailbox at that moment. If the house has been demolished, the carrier files a report (up = 0) and moves on. Service discovery is the address book that tells the carrier which houses exist this morning.

HTTP API endpoints worth memorizing

In pull, targets expose /metrics and Prometheus calls it. In push, targets stream samples to a collector. Pull buys Prometheus a surprisingly large bundle of properties that are hard to replicate in push systems:

• Implicit liveness. Cannot reach target ⇒ up=0 and existing series go stale — no "is this from a dead pod?" ambiguity.
• Server-side load control. Scrape interval, timeout, and concurrency are set on Prometheus; a misbehaving target cannot flood it.
• Debuggable with curl. Anyone with shell access can call /metrics and see exactly what Prometheus sees.
• No client buffering / retries. Targets do not need to know about Prometheus availability.
• Sharding by target. Scale horizontally by splitting the target list across servers.

Pull is awkward when targets live behind NAT, are very short-lived, or are conceptually event streams rather than stateful endpoints. Pull is therefore better for the steady state of long-lived services on trusted networks — which describes most of a Kubernetes cluster.

Animation A — The 15-second scrape pull cycle

Figure 3.2: Scrape loop sequence (pull model with implicit liveness)

Pushgateway — the narrow escape hatch

For short-lived batch jobs, the Pushgateway is a tiny HTTP server that accepts pushes, holds the current value in memory, and exposes them on its own /metrics — which Prometheus scrapes like any other target. The right shape: a nightly job pushes nightly_import_last_run_status, _duration_seconds, and _timestamp_seconds right before exit.

Anti-patterns to avoid:

• Do not use Pushgateway for long-running services — no built-in staleness; dead services look healthy forever.
• Do not push per-pod labels for ephemeral pods — each pod UUID becomes a permanent series.
• Do not treat it as an event stream — each push replaces the current value.
• Do not use it for SLO metrics — cached values hide outages.

For genuinely push-shaped workloads, the modern answer is usually the OpenTelemetry Collector (covered in chapter 5).

Federation

Federation lets one Prometheus pull a subset of series from another via /federate. The healthy hierarchy is leaf → aggregator → global, and you should pull only aggregated series (recording-rule outputs) through it. For raw data shipping, remote write is almost always the better tool.

A time series is uniquely identified by its metric name plus its label set. Everything else — help text, type, unit — is metadata. A single sample looks like:

http_requests_total{job="api", method="GET", status="200", path="/users"}  1873  1717520400000

That decomposes into:

• Metric name: http_requests_total. Conventions: _total for counters; _seconds, _bytes, _ratio for units.
• Label set: {job, method, status, path}. Each is a (name, value) pair. Change any value ⇒ new time series.
• Sample: a float value plus a Unix-millisecond timestamp.

This is the multi-dimensional idea: one metric name, many label dimensions, then slice at query time:

sum by (status) (rate(http_requests_total{job="api"}[5m]))

Analogy: metric name = spreadsheet, labels = columns, label values = cell contents, each unique row = a time series. Adding a high-cardinality column (user_id) is like adding a column with one row per user — cost is linear in distinct values.

OpenMetrics text exposition format

# HELP http_requests_total Total HTTP requests received.
# TYPE http_requests_total counter
http_requests_total{method="GET",status="200"} 1873
http_requests_total{method="GET",status="500"} 4
http_requests_total{method="POST",status="200"} 219

# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 4.2848256e+07

# HELP http_request_duration_seconds HTTP request latency.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.1"} 1500
http_request_duration_seconds_bucket{le="0.5"} 1860
http_request_duration_seconds_bucket{le="1.0"} 1872
http_request_duration_seconds_bucket{le="+Inf"} 1873
http_request_duration_seconds_sum 92.7
http_request_duration_seconds_count 1873
# EOF

Rules:

• # HELP and # TYPE are semantically meaningful comments.
• Each sample line: metric_name{labels} value [timestamp]. Timestamp is optional — scrape time is used if omitted.
• Histograms expand to multiple lines: one bucket per le boundary, plus _sum and _count — each bucket is its own series.
• OpenMetrics ends with a literal # EOF line.

Honor labels, target labels, and relabeling

By default, when a target's exposed labels conflict with the labels Prometheus attaches (most importantly job and instance), the target labels win. Setting honor_labels: true flips that — useful for the Pushgateway and federation.

Two flavors of relabeling:

• relabel_configs — runs before the scrape, against __meta_* SD labels. Selects which targets to scrape and rewrites their final labels.
• metric_relabel_configs — runs after the scrape, against each sample. Drops high-cardinality samples or renames labels.

scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # Keep only pods that opt in.
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      # Set the job label from the pod's annotation.
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_job]
        target_label: job
    metric_relabel_configs:
      # Drop a known cardinality-bomb metric.
      - source_labels: [__name__]
        regex: "go_gc_pauses_seconds_bucket"
        action: drop

Figure 3.3: Relabeling pipeline (target metadata to stored series)

Two rules of thumb: (1) test relabeling against a known target with --log.level=debug; (2) drop unbounded labels (user IDs, request IDs, raw URL paths) before they reach the TSDB.

Every sample follows a fixed path from scrape to retention:

1. WAL append. The sample (series ID + timestamp + value) is encoded as a CRC-checksummed record and written to the current WAL segment (~128 MB files: 00000001, 00000002, ...).
2. Head update. The TSDB looks up (or creates) the in-memory series for the label set and appends the sample to its current chunk.

The WAL is the durability story: if Prometheus crashes after step 1, the sample is replayed on restart from the most recent checkpoint.

Block layout

Every ~2 hours the head is cut into an immutable on-disk block:

01J1H1Q2K3V3Y2.../
  meta.json       # ULID, minTime, maxTime, compaction level, sources, stats
  index           # symbol table + postings lists (label=value -> series IDs)
  chunks/
    000001        # XOR-compressed chunks (mmapped at query time)
    000002
  tombstones      # optional, deletion intervals

The index is the crown jewel: a symbol table interns every label name and value, every series gets an integer ID, and postings lists hold the sorted series IDs for each label=value. A PromQL selector like {__name__="http_requests_total", job="api"} resolves by intersecting two postings lists — milliseconds, no table scan.

Chunks themselves use delta-of-delta encoding for timestamps and Gorilla-style XOR for floats — typically ~1-2 bytes per sample. Chunks are memory-mapped, which is why Prometheus often shows modest RSS but a large OS page cache.

Animation B — Block compaction over time

Figure 3.4: TSDB block lifecycle from scrape to retention

Crash recovery worked example

Suppose Prometheus has wal/checkpoint.10/ plus segments 00000011, 00000012, 00000013, and crashes mid-write on segment 13. On startup:

1. Existing immutable blocks load normally — no WAL needed.
2. The head is rebuilt from checkpoint.10.
3. Segments 11, 12, and 13 are replayed; recovery stops at the first record with a failing CRC.
4. Any incomplete writes are lost, but the head is consistent and queries work immediately.

Slow startup almost always means either checkpoints are not happening (frequent restarts, OOM kills) or the WAL grew during a long outage. The fix: keep Prometheus healthy long enough to checkpoint.

Remote write

Local TSDB is intentionally short-term (weeks). For longer retention or central aggregation, remote write POSTs Snappy-compressed protobuf batches to a configured URL:

remote_write:
  - url: https://mimir.example.com/api/v1/push
    basic_auth:
      username: tenant-42
      password_file: /etc/prometheus/mimir-token
    queue_config:
      capacity: 10000
      max_shards: 30
      min_backoff: 30ms
      max_backoff: 5s
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "go_.*"
        action: drop   # don't ship Go runtime metrics

Critical properties:

• Lossy under backpressure. If the remote endpoint cannot keep up, Prometheus's queue fills and eventually drops samples. Watch the prometheus_remote_storage_* metrics.
• Not a replay protocol. If you enable it on day 30, the backend sees day 30 onward — not the previous 30 days from local TSDB.
• write_relabel_configs can drop or rewrite labels on the way out — useful for stripping cardinality before it leaves your network.

Animation C — Remote write fanout

Thanos vs. Cortex vs. Mimir

Rules of thumb:

• Thanos — topology is "many independent Prometheus servers + give us LTS and a global view." Sidecar adds minimal moving parts; downsampling makes year-long dashboards fast.
• Mimir — you are building a multi-tenant metrics service with per-tenant quotas, isolation, and SLOs.
• Cortex — mostly if you already run it; for greenfield, Mimir is usually better.
• VictoriaMetrics — simpler single-binary alternative for teams without dedicated SREs.

Figure 3.5: Long-term storage architectures — Thanos vs. Mimir/Cortex