Chapter 5 — OpenTelemetry Architecture: API, SDK, and Collector

OpenTelemetry's most important architectural choice is splitting instrumentation surface (API) from pipeline implementation (SDK), and separating both from the out-of-process telemetry agent (Collector). Each layer has a distinct audience, release cadence, and dependency footprint.

1.1 The API — a stable interface for instrumentation

The API defines interfaces like Tracer, Meter, and Logger (plus their providers). Crucially, it does not know about exporters, samplers, batching, OTLP, or any backend. Libraries depend only on the API. If no SDK is registered, the API returns no-op implementations — spans are created but never recorded; the cost is a few function calls.

1.2 The SDK — a configurable pipeline implementation

The SDK is what application developers wire up at startup. It owns four moving parts:

• Samplers — decide whether to record (AlwaysOn, ParentBased, TraceIdRatioBased).
• Processors — buffer, batch, enrich, or filter (BatchSpanProcessor).
• Exporters — serialize and send (OTLP, Jaeger, Zipkin, Prometheus).
• Resource — describes the producer (service.name, host.name).

1.3 Why the split matters

A library depending only on opentelemetry-api adds essentially zero weight and zero opinion about your backend — the same library can be used by an app that exports to Jaeger, to a SaaS vendor, or to nothing at all, without recompilation.

1.4 The Collector

A separate Go binary running outside your app. Its pipeline is fixed: receivers → processors → exporters. Moving logic out of the app enables decoupled deploys, large-scale batching, vendor portability, and centralized policy.

OpenTelemetry's promise of consistency across 12+ languages rests on three pillars:

1. Specification — defines API/SDK semantics per signal.
2. Semantic conventions — standardized attribute names (service.name, http.response.status_code, k8s.pod.name).
3. OTLP — the wire protocol with three transport variants.

2.1 Stability per signal, per language

Each language SIG implements the spec at its own pace. Traces stabilized first; metrics followed; logs are the most recent.

2.2 Semantic conventions: the lingua franca

If every team picks its own attribute names — http.statusCode vs http_status vs httpResponse.code — "vendor-neutral" telemetry becomes useless. Semantic conventions give backends like Grafana, Datadog, Honeycomb, and Tempo a consistent vocabulary, enabling out-of-the-box visualizations.

2.3 OTLP transport variants

A common mistake is mismatching protocol and port (e.g., http/protobuf with port 4317) — this yields "unexpected response" or "transport error" messages.

2.4 Partial success & retries

OTLP responses carry a partial_success field. Rejected items should NOT be retried — they are usually permanently bad. Retry only on UNAVAILABLE, DEADLINE_EXCEEDED, 5xx, or 429 with exponential backoff and jitter.

The same Collector binary, with different configuration, can be deployed as a sidecar, a DaemonSet per node, or a centralized fleet of pods. Most production Kubernetes environments combine more than one of these patterns.

3.1 Agent mode (sidecar or DaemonSet)

• Sidecar: one Collector per app pod. Useful for per-service pipelines or strict tenant isolation. Expensive at scale.
• DaemonSet: one Collector per node. Apps on that node send telemetry locally.

Agents excel at: node-local enrichment (k8s.pod.name), host-level signals (kubelet, cAdvisor, container logs), minimal latency, isolated failure domain.

3.2 Gateway mode (centralized Deployment)

Gateways excel at:

• Tail-based sampling — needs the full trace, which only a centralized view gives.
• Efficient batching — aggregating from many sources lets the gateway build large, compressible batches.
• Centralized auth — API keys, OAuth tokens, mTLS certs live in one place.
• Multi-tenant routing — per-team backends, per-tenant quotas.

For correct tail sampling across replicas you need trace-ID-aware load balancing so all spans of a trace land on the same pod (the loadbalancing exporter or L7 consistent hashing).

3.3 Trade-off summary

3.4 Hybrid — the recommended pattern

DaemonSet agent does cheap node-local work (enrichment, host signals, log tailing) and forwards OTLP to a Deployment gateway, which handles tail sampling, tenant routing, and backend auth. If the gateway is overloaded, agents buffer locally until pressure subsides.

3.5 The OpenTelemetry Operator

On Kubernetes, the Operator provides OpenTelemetryCollector CRDs with mode: daemonset, sidecar, or deployment, plus an Instrumentation CR that auto-injects SDKs for Java, Python, Node, .NET, and Go.

The Collector ships in multiple distributions — pre-built bundles of receivers, processors, exporters, and extensions.

4.1 otelcol vs otelcol-contrib

• otelcol (core): minimal, stable, includes OTLP, Prometheus, batch, memory_limiter, attributes, debug.
• otelcol-contrib: kitchen-sink — vendor exporters (Datadog, Splunk, AWS, GCP), specialized receivers (kubeletstats, filelog, hostmetrics), advanced processors (tail_sampling, transform, routing, k8sattributes).

Most real-world pipelines need at least one contrib component — otelcol-contrib is the practical default.

4.2 Custom builds with ocb

The OpenTelemetry Collector Builder lets you assemble exactly the components you need from a manifest. Motivations:

• Supply-chain hygiene — smaller binary, smaller attack surface, faster startup.
• Compliance — explicit Go-module inventory.
• Performance — fewer registered components.

dist:
  name: my-otelcol
  otelcol_version: 0.95.0
receivers:
  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.95.0
processors:
  - gomod: github.com/.../k8sattributesprocessor v0.95.0
  - gomod: github.com/.../tailsamplingprocessor v0.95.0
exporters:
  - gomod: go.opentelemetry.io/collector/exporter/otlpexporter v0.95.0

Then ocb --config manifest.yaml emits a single Go binary with exactly those components — nothing more.

4.3 Vendor distributions

AWS ADOT, Splunk OTel Collector, Datadog Agent (OTel mode), and Grafana Agent / Alloy bundle upstream Collector with vendor-tuned defaults. They still respect API/SDK/OTLP boundaries — applications continue emitting standard OTLP, only the Collector is vendor-flavored. Switching vendors is largely a Collector configuration change.

4.4 Choosing a distribution