Chapter 6 — Instrumentation: Manual, Automatic, and Zero-Code

Manual instrumentation puts the developer in direct control. You acquire a Tracer, Meter, or Logger from the SDK and explicitly start spans, record measurements, or write structured log events. It is the only way to express domain context: concepts like order_id, tenant_id, payment.status, and feature_flag.variant that no auto-instrumenter could ever guess.

Acquiring Tracers and Meters

The SDK exposes TracerProvider, MeterProvider, and LoggerProvider. From each you obtain a named, versioned instance scoped to your module — conventionally the package import path. The name becomes instrumentation.scope.name on every signal, letting backends filter by which code produced the data.

// Java
Tracer tracer = GlobalOpenTelemetry.getTracer("com.acme.payments", "1.4.0");
Meter  meter  = GlobalOpenTelemetry.getMeter("com.acme.payments");

# Python
tracer = trace.get_tracer("acme.payments", "1.4.0")
meter  = metrics.get_meter("acme.payments")

// Node.js
const tracer = trace.getTracer('acme-payments', '1.4.0');
const meter  = metrics.getMeter('acme-payments');

Creating Spans and Recording Attributes

A span is a named, timed operation with attributes, events, and a status. The idiomatic pattern wraps a unit of work so the span closes even on exceptions:

# Python: idiomatic context-manager span
with tracer.start_as_current_span("authorize_payment") as span:
    span.set_attribute("payment.method", "card")
    span.set_attribute("tenant.id", tenant_id)
    try:
        approved = gateway.authorize(amount)
        span.set_attribute("payment.outcome",
                           "approved" if approved else "declined")
    except Exception as exc:
        span.record_exception(exc)
        span.set_status(trace.StatusCode.ERROR, str(exc))
        raise

Use dot-namespaced keys (payment.method, not paymentMethod); follow semantic conventions where one exists; treat your custom namespace (acme.*) like a public API — once a dashboard depends on it, you cannot freely rename it.

Picking the Right Metric Instrument

Three rules: declare units (s, By, 1); let the SDK pick histogram buckets unless you really know the latency profile; remember that every attribute multiplies cardinality — an idea we revisit in Section 4.

Auto-instrumentation answers "How do I get traces from libraries I did not write?" The mechanism differs by runtime because each language exposes different hooks.

Bytecode Injection: Java and .NET

The Java agent attaches via the -javaagent flag, registers a premain with the Instrumentation API, and uses ByteBuddy to rewrite classes as the classloader loads them:

OTEL_SERVICE_NAME=checkout-service \
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \
OTEL_TRACES_EXPORTER=otlp \
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod,service.version=2.3.1 \
java -javaagent:/opt/otel/opentelemetry-javaagent.jar -jar /app/checkout.jar

The agent ships with modules for Servlet, Spring MVC/WebFlux, JAX-RS, gRPC, OkHttp, JDBC, R2DBC, Hibernate, Mongo, Cassandra, Kafka, RabbitMQ, JMS, and more. Because rewriting happens at class load, no source change is needed — but the agent must attach at JVM start and exotic classloaders sometimes need extra config. .NET uses a conceptually similar mechanism via a CLR profiler activated by CORECLR_ENABLE_PROFILING=1.

Monkey-Patching: Python and Node.js

Dynamic languages let you replace functions at runtime. Python's opentelemetry-instrument CLI bootstraps every installed opentelemetry-instrumentation-* package, which monkey-patches its target library at import:

pip install opentelemetry-distro opentelemetry-exporter-otlp \
            opentelemetry-instrumentation-requests \
            opentelemetry-instrumentation-psycopg2 \
            opentelemetry-instrumentation-flask

OTEL_SERVICE_NAME=orders-api \
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \
opentelemetry-instrument gunicorn orders.wsgi:application

The requests instrumentation replaces Session.request with a wrapper that opens a client span, records HTTP attributes, calls the original, captures the response, and ends the span. It must run before the first import of the patched library; otherwise the cached reference is the unpatched one. Node.js relies on a require hook via require-in-the-middle:

// tracing.js  — must be required first
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({}),
  instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();

// run: NODE_OPTIONS="--require ./tracing.js" node app.js

Java Agent Lifecycle (Figure 6.2)

Cross-Language Comparison

A shared environment-variable contract spans every language: OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER, OTEL_LOGS_EXPORTER, OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_TRACES_SAMPLER, OTEL_PROPAGATORS, OTEL_RESOURCE_ATTRIBUTES. One ConfigMap, one vocabulary, every workload.

Debugging axioms: No traces at all usually means an exporter is set to none, the protocol is mismatched (grpc vs. http/protobuf), or the bootstrap loaded too late. Duplicate spans almost always mean a library is being captured by both auto- and manual instrumentation — disable one.

"Zero-code" goes further than auto-instrumentation: the developer doesn't write tracing code and the build artifact isn't modified. Two distinct technologies live under this umbrella: eBPF agents that observe processes from the Linux kernel, and the OpenTelemetry Operator that injects auto-instrumentation agents into Kubernetes pods at admission time without changing container images.

eBPF-Based Auto-Instrumentation

eBPF lets you load safe, sandboxed bytecode into the Linux kernel and attach it to kernel events — syscalls, function entry/exit, tracepoints, network events — at runtime, without recompiling the kernel. An eBPF observability agent typically:

1. Attaches kprobes to network kernel functions (tcp_sendmsg, tcp_cleanup_rbuf, sys_enter_sendto/recvfrom) to observe every byte crossing TCP.
2. Attaches uprobes to user-space functions in shared libraries — SSL_read/SSL_write in libssl, Go's HTTP runtime, JVM JNI entries — to see data before encryption.
3. Writes structured records into eBPF maps drained at high frequency by a user-space agent.
4. Reconstructs requests — pairing sends/recvs, parsing HTTP headers, gRPC HTTP/2 framing — to produce L7 metrics and OTLP spans.

Because the hooks live in kernel and shared libraries, eBPF works for every language on the host — Go, Rust, Java, Python, Node, C++, even closed-source binaries — without touching their code. The output is typically the four golden signals plus distributed traces for common protocols.

OpenTelemetry Operator and the Instrumentation CRD

For Kubernetes workloads, the Operator offers a different flavor of zero-code: the cluster itself injects the SDK auto-instrumentation agents we saw in Section 2, without changing your container images.

# 1. The Instrumentation CRD: a reusable recipe per language
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: default-instrumentation
  namespace: production
spec:
  exporter:
    endpoint: http://otel-collector.observability:4317
  propagators: [tracecontext, baggage]
  sampler:
    type: parentbased_traceidratio
    argument: "0.1"
  resource:
    attributes:
      deployment.environment: prod
      service.namespace: payments
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
  nodejs:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest

# 2. The Deployment opts in via pod annotations
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "production/default-instrumentation"

When the annotated pod is created, the mutating webhook injects an init container that copies the agent JAR into a shared volume, then patches the application container with JAVA_TOOL_OPTIONS=-javaagent:/otel-auto-instrumentation/javaagent.jar and the appropriate OTEL_* environment variables. Python and Node.js use analogous mechanisms. Write the CRD once, label deployments by language, every new pod is born observable.

Limits of Zero-Code — Hybrid Is Production-Grade

Hybrid wins. Run eBPF for horizontal, polyglot baseline coverage; use the Operator to inject SDK auto-instrumentation on every K8s pod; add manual instrumentation on the critical business flows where you need tenant_id, feature_flag, and payment.outcome.

Instrumentation nobody can query is expensive noise. Semantic conventions are OpenTelemetry's contract that names things the same way everywhere, so a dashboard against http.response.status_code works whether the data came from a Java agent, a Python monkey-patch, a Beyla eBPF probe, or your own manual code.

Common Stable Attributes

A query like "p95 HTTP latency by route and status, last 30 minutes" is just groupby(http.route, http.response.status_code) of histogram(http.server.request.duration) — same panel, across the fleet, across vendors.

Resource Attributes — Identity of the Emitter

Where attributes describe a single signal, resource attributes describe the emitter. Set them once at SDK init via OTEL_RESOURCE_ATTRIBUTES:

OTEL_SERVICE_NAME=checkout-api
OTEL_RESOURCE_ATTRIBUTES=\
service.namespace=payments,\
service.version=2.3.1,\
service.instance.id=checkout-api-7d4f-x9w2,\
deployment.environment=prod,\
k8s.namespace.name=production,\
k8s.deployment.name=checkout-api,\
k8s.pod.name=checkout-api-7d4f-x9w2,\
cloud.provider=aws,\
cloud.region=us-east-1

The OpenTelemetry Operator can fill many of these from the pod's downward API — you should rarely set K8s resource attributes by hand.

Operator Workflow (Figure 6.4)

Cardinality — The Silent Killer

Every unique combination of attribute values produces a distinct time series. Cardinality destroys pricing, retention, query speed, and cluster stability. Before attaching an attribute, ask "How many distinct values can this take?"

Three mitigations:

1. Templates, not raw values. Use http.route=/orders/{id} on metrics; keep url.full for span-only debugging.
2. Drop or hash at the Collector. attributes, transform, and redaction processors can drop, truncate, or one-way-hash before export.
3. Separate metric and span schemas. A span can carry tenant.id and order.id; the derived metric should only carry tenant.tier and payment.method. Spans are sampled; metrics aggregate forever.

These same mitigations double as PII hygiene: url.full, url.query, client.address, and db.statement may contain personal data — redact at the SDK, allow-list at the Collector, hash where you need cardinality without identity.

Stability and Evolution

Conventions evolve under Experimental → Stable → Deprecated. The migration from http.method to http.request.method is a real example: both names exist for a period, the new is preferred, Collector transform processors normalize older signals so dashboards survive. Anchor dashboards on Stable attributes; treat Experimental as opt-in.