Chapter 10: The OpenTelemetry Collector in Depth

The Collector is not a single black box — it is a configurable pipeline engine built from four kinds of components, plus optional extensions. Every piece of telemetry flowing through takes the same conceptual journey: it enters through a receiver, traverses a chain of processors, and leaves through one or more exporters. Pipelines are declared per signal type (traces, metrics, logs), and the service block is what actually wires components together into runnable pipelines.

The four component types (plus extensions)

Connectors are the cleanest way to derive one signal from another — for example, generating RED metrics (Rate, Errors, Duration) from spans via a spanmetrics connector that exits a traces pipeline and re-enters a separate metrics pipeline.

Mermaid: Canonical pipeline anatomy (Figure 10.0)

A minimal three-signal config

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [loki]

A component defined in receivers, processors, or exporters but not referenced under service.pipelines is silently ignored. This is the most common source of "my config does nothing" surprises.

Processor order is decisive

Processors run in the order listed. This is one of the most consequential, and most commonly overlooked, properties of Collector configuration:

1. memory_limiter always first — back-pressure kicks in before later, more expensive processors waste CPU
2. Enrichment processors next (e.g., k8sattributes, resource) so downstream filters see full context
3. Filter / sampling next — drop unwanted data before transforms touch it
4. transform / scrubbing — reshape what is left
5. batch last — coalesce into large outbound batches just before the exporter

memory_limiter + batch — the mandatory pair

memory_limiter samples Collector memory on a fixed interval and, when usage crosses configured thresholds, refuses new data by returning errors to receivers. That refusal is what creates back-pressure: upstream senders see failures, retry, and slow down — instead of the Collector dying from an out-of-memory kill.

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 800       # ~80% of container memory limit
    spike_limit_mib: 200 # tolerance for short bursts
  batch:
    timeout: 5s
    send_batch_size: 512
    send_batch_max_size: 4096

Analogy: batch is a hotel shuttle that waits up to five minutes (or until full) before driving to the airport — far more efficient than calling a taxi for every guest. memory_limiter is the bouncer at the lobby door who turns guests away when the lobby is full, so the building never collapses.

transform with OTTL

For richer mutations — conditional logic, regex substitution, cross-field arithmetic — reach for the transform processor, which uses the OpenTelemetry Transformation Language (OTTL). OTTL statements look like set(target, value) where <boolean> and run inside a context (span, metric, datapoint, log, resource, or scope).

processors:
  transform:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          # Collapse user IDs in URL paths so cardinality stays bounded
          - replace_pattern(attributes["http.target"], "/users/[0-9]+", "/users/:id") where attributes["http.target"] != nil
          # Remove PII before exporting
          - delete_key(attributes, "user.email")
          - delete_key(attributes, "user.id")
          # Whitelist what is allowed to leave
          - keep_keys(attributes, ["http.method", "http.target", "http.status_code", "service.name"])
          # Mark anything from the checkout service
          - set(attributes["env"], "prod") where resource.attributes["service.name"] == "checkout-service"

error_mode: ignore matters: the default in some versions is propagate, which can fail an entire batch when a single statement errors. A close cousin, filter, uses OTTL conditions to drop data outright (e.g., dropping /healthz spans).

tail_sampling vs probabilistic_sampler

The probabilistic_sampler is cheap and stateless — it picks (say) 5% based on trace ID, but it cannot prefer error or slow traces. The tail_sampling processor is fundamentally different: it buffers all spans for a trace, keyed by trace ID, and decides keep/drop only after decision_wait seconds or all spans have arrived. Because it sees the whole trace, it can sample on end-to-end latency, final status, or attributes that appear only on a leaf span.

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 2000
    policies:
      - name: main
        type: composite
        composite:
          max_total_spans_per_second: 1000
          policy_order: [error-traces, slow-traces, premium-tenants, baseline]
          sub_policies:
            error-traces:    { type: status_code, status_code: { status_codes: [ERROR] } }
            slow-traces:     { type: latency,      latency:      { threshold_ms: 4000 } }
            premium-tenants: { type: string_attribute, string_attribute: { key: tenant.tier, values: ["gold","platinum"] } }
            baseline:        { type: probabilistic, probabilistic: { sampling_percentage: 1.0 } }

Crucial gotcha: tail sampling only works if SDKs export spans unsampled (always_on or parentbased(always_on)). If the SDK already dropped spans, no policy can resurrect them.

Mermaid: Tail sampling decision flow (Figure 10.2)

Head vs tail sampling at a glance

k8sattributes — Kubernetes enrichment

The k8sattributes processor watches the Kubernetes API and decorates telemetry with metadata about the sending pod (namespace, deployment, node, labels). It identifies the sender either by inbound connection IP or by an explicit k8s.pod.ip resource attribute. Run it on the agent (DaemonSet), never on a central gateway — the gateway only sees the agent's IP, not the application pod's. Limit extract.metadata to fields you actually query on; each one multiplies cardinality and API-server load.

Workhorse receivers

A common DaemonSet receiver block:

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
  hostmetrics:
    collection_interval: 30s
    scrapers: { cpu: {}, memory: {}, disk: {}, filesystem: {}, network: {}, load: {} }
  filelog:
    include: ["/var/log/pods/*/*/*.log"]
    start_at: end
    operators:
      - type: container

The prometheus receiver is worth a special mention: it accepts native Prometheus scrape config, so an existing prometheus.yml can be lifted into the Collector almost verbatim — a powerful migration path.

Workhorse exporters

exporters:
  otlp/tempo:
    endpoint: tempo-distributor.observability:4317
    tls: { insecure: true }
    sending_queue: { enabled: true, num_consumers: 10, queue_size: 2000 }
    retry_on_failure: { enabled: true, initial_interval: 5s, max_interval: 60s, max_elapsed_time: 0 }
  prometheusremotewrite:
    endpoint: http://mimir.observability:8080/api/v1/push
    resource_to_telemetry_conversion: { enabled: true }
  loki:
    endpoint: http://loki-gateway/loki/api/v1/push
  debug:
    verbosity: basic

resource_to_telemetry_conversion: true on prometheusremotewrite promotes OTLP resource attributes (like service.name, k8s.pod.name) into Prometheus labels so they become queryable in PromQL.

Connectors — the inter-pipeline glue

A connector behaves as an exporter on one pipeline and a receiver on another. The canonical example is spanmetrics: it consumes spans and emits aggregated RED metrics — derived signals produced once, near the source, rather than re-derived in each backend.

connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s]
    dimensions:
      - name: http.method
      - name: http.status_code

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [spanmetrics, otlp/tempo]      # spanmetrics is an EXPORTER here
    metrics/spans:
      receivers: [spanmetrics]                  # ... and a RECEIVER here
      processors: [batch]
      exporters: [prometheusremotewrite]

Other useful connectors include routing (split traffic by attribute) and forward (chain pipelines together).

Mermaid: Multi-signal pipeline topology (Figure 10.1)

Persistent queue and retry on failure

Most exporters support a sending_queue and retry_on_failure. By default the sending queue is in memory: fast, but lost on restart. Pairing it with the file_storage extension makes the queue durable, so an OOM kill or rolling deployment doesn't drop telemetry that has already been accepted from upstream.

extensions:
  file_storage:
    directory: /var/lib/otelcol/storage
    timeout: 1s

exporters:
  otlp/tempo:
    endpoint: tempo-distributor.observability:4317
    sending_queue:
      enabled: true
      storage: file_storage   # makes the queue durable across restarts
      num_consumers: 10
      queue_size: 2000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 60s
      max_elapsed_time: 0     # retry forever

Sizing tips:

• queue_size: 1,000-5,000 batches per exporter (each batch holds thousands of records)
• num_consumers: 5-10 typical; raise only if the backend can handle more parallelism
• retry_on_failure: exponential backoff with initial_interval: 5s, max_interval: 60s; set max_elapsed_time: 0 for critical data and let memory_limiter drop oldest when the queue fills

Mermaid: memory_limiter, batch, queue, retry interplay (Figure 10.4)

Extensions for operability

extensions:
  health_check: { endpoint: 0.0.0.0:13133 }
  pprof:        { endpoint: 0.0.0.0:1777 }
  zpages:       { endpoint: 0.0.0.0:55679 }

# pod spec:
livenessProbe:  { httpGet: { path: /, port: 13133 }, initialDelaySeconds: 10 }
readinessProbe: { httpGet: { path: /, port: 13133 }, periodSeconds: 5 }

zpages is especially useful during incidents — per-component counters and sampled recent traces come directly from the Collector's process, so you can answer "is data flowing? is anything dropping?" without leaving the cluster.

Mermaid: Two-tier agent + gateway topology (Figure 10.3)

Sizing, throughput, and memory tuning

The memory_limiter should target 70-80% of the container memory limit, with spike_limit_mib covering the largest plausible batch. Agents are not HPA-scaled (they scale with node count via DaemonSet); the gateway runs an HPA on CPU at 60-70% target utilization with minReplicas: 2 for graceful scaling and HA.

Tail-sampling capacity follows a simple rule of thumb: num_traces ≥ expected_new_traces_per_sec × decision_wait × 2. At 2,000 traces/sec and a 10s decision_wait, that is num_traces: 40000 minimum — round up to 50,000 for headroom.

Monitor the Collector with… itself: every Collector exposes its internal metrics on port 8888. Alert on:

• Memory near limit_mib for sustained periods
• Sustained growth of otelcol_exporter_queue_size
• Non-zero otelcol_processor_refused_* or otelcol_exporter_send_failed_* counters
• HPA at maxReplicas while queues continue to grow