Chapter 12 — SLOs, Alerting, and Operational Excellence

A production service is a promise to its users. SLIs measure how well you keep it, SLOs are the targets you commit to, and the error budget is the contractually allowed slack between perfect and "good enough."

Choosing Good SLIs

A good SLI is a ratio between good events and valid events — e.g. "the fraction of HTTP requests that returned a non-5xx within 300 ms" — measured from the user's perspective. Ratios normalize naturally with traffic: 1% errors at 10 RPS is the same as 1% errors at 10,000 RPS.

Four families cover most services:

Analogy: SLIs are the dashboard of a car. Availability = "does the engine start?"; latency = "how fast can it accelerate?"; freshness = "how old is the GPS reading?"; correctness = "does the odometer match what we drove?"

Figure 12.1: SLI → SLO → Error Budget → Burn Rate

Error Budgets

Once you commit to an SLO, the arithmetic is trivial: error_budget = 1 - SLO. For 99.9% over 30 days: 0.001 × 43,200 minutes ≈ 43.2 minutes/month. That number — not the percentage — is the most useful artifact in your program: it converts an abstraction into a budget engineers and PMs can reason about. If you have burned 30 min this month you have 13 left, which immediately changes deployment risk decisions.

Burn Rate and Multi-Window Multi-Burn-Rate

The burn rate B = observed_error_fraction / error_budget answers the only question that matters: at the current pace, when do we run out? If B = 14.4, a 30-day budget is gone in ~2 days.

The naïve "alert when error rate > threshold" approach pages on transient blips and misses slow erosion. The Google SRE workbook's MWMBR pattern fixes both by requiring agreement across two windows:

# Recording rules — compute once, reuse everywhere
- record: slo:http_errors:ratio_rate5m
  expr: |
    sum by (service, env) (rate(http_requests_total{code=~"5.."}[5m]))
    /
    sum by (service, env) (rate(http_requests_total[5m]))

# Fast-burn MWMBR alert
- alert: SLOErrorBudgetBurnFast
  expr: |
    (
      slo:http_errors:ratio_rate5m{service="checkout"} > (14.4 * 0.001)
      and
      slo:http_errors:ratio_rate1h{service="checkout"} > (14.4 * 0.001)
    )
    and
    sum by (service, env) (rate(http_requests_total{service="checkout"}[5m])) > 1
  for: 2m
  labels: { severity: page, slo: availability-99.9-30d, team: payments }
  annotations:
    summary: "Checkout burning error budget at >14.4x in {{ $labels.env }}"
    runbook_url: "https://runbooks.example.com/checkout/slo-fast-burn"

Two production details: the sum(rate(...)) > 1 gate kills spurious 100% ratios in low-traffic windows; latency SLOs reuse the same machinery with a histogram le-bucket numerator.

Figure 12.2: MWMBR Alert Escalation

A well-tuned SLO is wasted if it lands in a flood of noise at 3 a.m. The alerting architecture — how alerts route, group, inhibit, and reach humans — decides whether on-call is sustainable.

Alertmanager: Grouping, Routing, Inhibition

• Grouping collapses related alerts into one notification. group_by: [alertname, service, severity, env] produces one notification per actual incident; grouping by pod or container floods you during every rolling deploy.
• Routing trees branch by severity, env, and team ownership. Critical prod → PagerDuty; warnings → Slack channel; info → low-priority or nowhere.
• Inhibition suppresses symptom alerts when a known root-cause alert is firing (e.g. a node-down alert silences every per-pod alert on that node).

route:
  receiver: 'default-slack'
  group_by: ['alertname', 'service', 'severity', 'env']
  group_wait: 60s
  group_interval: 10m
  repeat_interval: 4h
  routes:
    - matchers: [ severity="critical" ]
      routes:
        - matchers: [ env="prod" ]
          receiver: 'pagerduty-prod'
          continue: true
          routes:
            - matchers: [ team="payments" ]
              receiver: 'pagerduty-payments'

inhibit_rules:
  - source_matchers: [ severity="critical", alertname="KubernetesNodeDown" ]
    target_matchers: [ alertname=~"KubePodCrashLooping|KubePodNotReady|InstanceDown" ]
    equal: ['node', 'env']

For HA, run three Alertmanager replicas with gossip clustering so a restart doesn't double-fire every active alert.

On-Call Ergonomics

Every page must be actionable: the on-call engineer should know within 60 seconds what to do. The hard rule is no page without a runbook URL. A runbook entry answers:

1. How do I confirm this is real? (PromQL, dashboard, log query)
2. What is the most likely cause?
3. What are the safe remediation steps?
4. When do I escalate, and to whom?
5. What is the rollback criterion?

Anti-Patterns

• Paging on causes (CPU > 90%) instead of symptoms (users see errors). A CPU at 99% with happy users is not an incident; the same CPU at 60% during a brownout is. Anchor pages on the SLI ladder.
• Static thresholds on dynamic systems — "latency > 500 ms" pages every Black Friday. Burn-rate alerts are the cure.
• Alerts without owners — every rule must carry a team label.
• Per-pod alerts in Kubernetes — alert on Deployment/Service aggregates, not individual replicas.
• Unbounded silences — always set durations and require a ticket comment.
• No for: clause — without it every blip becomes a page.

Figure 12.3: Alertmanager Routing with Grouping and Inhibition

You have all the pieces. The job now is integration — a platform that holds together at scale, can roll out incrementally, and is itself observable.

Reference Architecture — Six Logical Layers

1. Instrumentation — OTel SDKs + auto-instrumentation, emitting OTLP for metrics, traces, logs, and profiles.
2. Collection — OTel Collector DaemonSets + fan-in Deployments; Prometheus via the Operator for scrape targets (kube-state-metrics, node-exporter, cAdvisor).
3. Storage — Prometheus TSDB (short-term) → Thanos/Mimir/Cortex (long-term); Tempo/Jaeger for traces; Loki/Elasticsearch for logs; Pyroscope/Parca for profiles.
4. Rules & alerting — recording + alerting rules, Alertmanager HA, runbook hosting.
5. Visualization — Grafana with datasources across every signal store.
6. Meta-observability — a small, segregated Prometheus + Alertmanager whose only job is monitoring the observability stack itself.

Figure 12.4: Reference Observability Platform on Kubernetes

Greenfield Rollout

Migration from Prometheus-only stacks follows the "wrap, don't replace" rule. Keep Prometheus where it works (scrape-based infra, alerting); add OTel Collectors as the entry point for new signals. The Collector's prometheusremotewrite exporter ships OTel metrics into Prometheus/Mimir, and the prometheus receiver scrapes existing exporters — a unified pipeline without a forklift migration.

Meta-Observability: Capacity Planning the Platform

Sizing rules of thumb:

• Prometheus memory: ~3 KB per active series in steady state; +25% headroom for query bursts.
• Prometheus disk: ~1.5 bytes/sample after compression × samples/sec × retention seconds.
• Tempo/Jaeger: head-based sampling at 1–10%, plus tail-based sampling for errors/slow traces.
• OTel Collector: 1 GB RAM per ~20k spans/sec with batching; CPU scales linearly with attribute count.
• Loki: log volume dominates. WARN+ in prod is sustainable; DEBUG in prod is not.

The fundamentals — SLOs, MWMBR, Alertmanager hygiene — have been stable for nearly a decade. The interesting changes are at the edges.

Profiles as a Fourth Signal

Three pillars (metrics, logs, traces) become four. Continuous profiling — always-on stack-trace sampling at 50–200 Hz with 1–3% overhead — is the fourth.

Example: metrics show p99 rising from 200 → 500 ms after deploy. Traces narrow it to CalculateDiscounts spans. Profiles reveal 40% of CPU is in a new calculate_rewards_v2 function, dominated by hashmap operations. Without profiles you have a suspect; with profiles you have the exact lines of code.

Grafana Pyroscope (Prometheus-like labels, multi-language SDKs, eBPF) and Parca / Polar Signals (Kubernetes-native eBPF DaemonSet) lead the OSS space. The maturing OpenTelemetry profiling signal adds profiles as a first-class OTLP signal sharing resource attributes and trace/span IDs.

AI-Assisted Root Cause Analysis

Vendors and OSS projects are building systems that:

• Correlate signals automatically when an SLO burn alert fires — matching traces, topology, profiles, recent commits, feature flags.
• Summarize hypotheses — an LLM drafts "what changed?" and "where is load concentrated?" for the on-call to edit.
• Suggest queries in natural language, grounded in the available metric and log schemas.

Two prerequisites: OTel semantic conventions for vocabulary, and structured runbooks for remediation playbooks. This is not a replacement for engineering judgment — it removes the manual query-writing tax during incidents.

Semantic Convention Convergence

The longest-running success of OTel is that vendors converge on the same names: service.name, http.response.status_code, db.system, k8s.pod.name, deployment.environment. Practical implications:

• Vendor lock-in shrinks — switching APMs becomes a Collector reconfiguration.
• Cross-team analytics work — fleet-wide SLO compliance from a single query.
• AI assistants become more useful — standard names mean cross-org generalization.

Strategic takeaway: invest in conformance now. Add CI checks that reject non-standard attribute names; require OTel resource attributes via the Collector's resourcedetection processor.

Figure 12.5: The Four Pillars and the Path Forward

You've crossed the bridge from "we collect telemetry" to "we operate to a contract." SLIs measure user-visible behavior as good/valid ratios. SLOs are targets on SLIs; their inverse is the error budget. MWMBR alerts page only when the budget is genuinely at risk. Alertmanager turns alerts into humane notifications via grouping, severity-aware routing, inhibition, runbook-linked annotations, and HA clustering. A reference architecture layers instrumentation, collection, storage, rules, visualization, and meta-observability. The next frontier is continuous profiling, AI-assisted RCA, and semantic-convention convergence — all of which plug into the OTel foundation you build today.