A distributed trace only works if every service knows which trace a request belongs to. Context propagation is the mechanism: the caller writes the trace ID and its own span ID into the request, and the receiver reads them and continues the same trace.
When propagation breaks, traces split into disconnected pieces and logs cannot be matched to requests. This guide explains what travels, how it travels, how to propagate through queues by hand, and how to get trace IDs into logs. Every snippet on this page was run: the Node.js ones with @opentelemetry/sdk-node 0.301, @opentelemetry/instrumentation-pino 0.71 and Pino 10.4, the Python ones with opentelemetry-sdk 1.41.1 and opentelemetry-instrumentation-logging 0.62b1.
Key takeaways
- The W3C
traceparentheader carries the trace ID, the parent span ID and the sampled flag. It is the default propagator in every SDK. - HTTP and gRPC instrumentation propagate for you. Queues and custom transports need
injectandextract. - Baggage travels to every downstream service, including third parties. Never put secrets or personal data in it.
- Logging instrumentation adds
trace_idandspan_idto each log line written inside a span. - A trace that splits in two points at one hop that does not propagate. Find that hop.
What context is
Inside one process, OpenTelemetry keeps a context: the active span, plus any baggage. When your code starts a span, the SDK makes it a child of whatever span is active. In Node.js that context follows async calls through AsyncLocalStorage; in Python it lives in contextvars.
Between processes there is no shared memory, so the context has to be written into the request. A propagator does that. It has two operations: inject writes the context into a carrier such as HTTP headers, and extract reads it back on the other side. The SDK's default propagators are set by OTEL_PROPAGATORS, which defaults to tracecontext,baggage: the W3C Trace Context headers plus the W3C Baggage header.
Other formats exist for older systems: b3 and b3multi for services instrumented with Zipkin libraries, and jaeger for the retired Jaeger clients, which the specification marks deprecated. If part of your system still uses one of them, list both, for example OTEL_PROPAGATORS=tracecontext,baggage,b3multi. The SDK writes every listed format on the way out. On the way in it runs every extractor in order, and a later format that finds its header replaces what an earlier one found, so put the format you trust most last.
The traceparent header
W3C Trace Context defines two HTTP headers. traceparent is the one that matters for joining spans:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^ ^^
| trace-id (32 hex) parent-id (16) flags
version 01 = sampled
tracestate: ot=th:c,vendor=opaque-value - Version:
00, the only version in the current standard. - Trace ID: 32 lowercase hex characters (16 bytes). The same in every span of the trace. All zeros is invalid.
- Parent ID: 16 hex characters (8 bytes), the span ID of the caller's span. The receiver's first span becomes its child.
- Flags: two hex characters. The lowest bit is sampled:
01means the caller recorded this trace,00means it did not. Parent-based samplers follow it, as the sampling guide explains. Trace Context Level 2, still a draft, adds a second flag (02), which tells the receiver that at least the trace ID's rightmost 7 bytes are random and safe to sample on.
tracestate is the second header: up to 32 comma-separated key=value entries where tracing systems keep their own data. OpenTelemetry uses the ot key, for example to record a sampling threshold (th). Propagators forward entries they do not understand unchanged.
A header that does not parse is ignored, and the receiver starts a new trace. That is the right failure mode, and also why a malformed header produces a split trace instead of an error.
Automatic propagation
For HTTP and gRPC you rarely touch any of this. The instrumentation for HTTP clients (such as fetch, undici, requests or httpx) injects traceparent into outgoing requests, and the instrumentation for servers and frameworks (such as Express, Fastify, Django or FastAPI) extracts it from incoming ones. Install both sides and traces connect.
The condition is that both sides are instrumented. A service with server instrumentation but no client instrumentation receives the trace and then starts a new one for every downstream call. The trace in your backend stops at that service, and the downstream service's spans show up as separate traces.
In the browser, the fetch and XMLHttpRequest instrumentations only add traceparent to cross-origin requests whose URLs you list in propagateTraceHeaderCorsUrls. The API must also allow the header: include traceparent and tracestate in Access-Control-Allow-Headers, or the browser blocks the request.
Queues and custom transports
Message queues are the common gap. Some client libraries have instrumentation that propagates through message headers; many do not, and a custom protocol never does. Propagate by hand: inject into whatever key-value headers the message supports when you publish, and extract when you consume.
orders.js
import { context, propagation, trace, SpanKind } from "@opentelemetry/api";
const tracer = trace.getTracer("orders");
// Producer: copy the current context into the message headers.
export function publish(body) {
return tracer.startActiveSpan("publish orders", { kind: SpanKind.PRODUCER }, (span) => {
const headers = {};
propagation.inject(context.active(), headers);
span.end();
return { body, headers };
});
}
// Consumer: continue the trace the message came from.
export function consume(message) {
const parent = propagation.extract(context.active(), message.headers);
tracer.startActiveSpan("process orders", { kind: SpanKind.CONSUMER }, parent, (span) => {
// handle the message
span.end();
});
} Span names follow the messaging conventions, {operation} {destination}. Running it with the Node.js SDK put this in the message headers, and the consumer span continued trace 03bf06ab…:
{"traceparent":"00-03bf06ab90d8fc69b27724424bd114d2-f195d892a7e752e5-01"} The Python equivalent, where queue and message stand in for your client library:
orders.py
from opentelemetry import trace
from opentelemetry.propagate import extract, inject
from opentelemetry.trace import SpanKind
tracer = trace.get_tracer("orders")
# Producer: copy the current context into the message headers.
with tracer.start_as_current_span("publish orders", kind=SpanKind.PRODUCER):
headers = {}
inject(headers)
queue.send(body, headers=headers)
# Consumer: continue the trace the message came from.
ctx = extract(message.headers)
with tracer.start_as_current_span("process orders", context=ctx, kind=SpanKind.CONSUMER):
handle(message) A consumer that processes a batch of messages from different traces cannot be a child of all of them. Start one span for the batch and add a span link to each message's context instead. Links connect related traces without forcing them into one tree.
Baggage
Baggage is a second piece of context: key-value pairs your code sets that travel to every downstream service in their own header.
baggage: tenant.id=acme,plan=enterprise Use it for values that downstream services need and cannot look up cheaply, such as a tenant ID set at the edge that a service four hops away wants on its spans. Three facts about it surprise people:
- It is not added to telemetry by itself. A value in baggage does not appear on spans, metrics or logs unless code copies it there. Several SDKs ship a baggage span processor that copies baggage onto each new span as attributes.
- It leaves your network. Instrumented HTTP clients send the baggage header on outgoing requests, including calls to third-party APIs. Never put tokens, emails or other personal data in it.
- It is not trusted. Any caller can send a baggage header. A service that makes decisions on baggage values from the public internet can be fed whatever an attacker likes. Strip or ignore incoming baggage at your edge.
Trace IDs in logs
Propagation also solves a logging problem. If every log line written during a request carries that request's trace ID, you can go from an error log to the full trace, and from a slow span to the log lines around it. The trace ID is the join key between the two.
There are two ways in. Log correlation adds trace_id and span_id to the log lines your logger already writes, to stdout or a file. Log export sends log records through the OpenTelemetry SDK over OTLP, where trace and span IDs are fields of every record. Many setups use both. The examples below show correlation, since it works with any log pipeline.
Node.js with Pino
Install the Pino instrumentation and register it with the SDK before your app loads:
npm install @opentelemetry/sdk-node @opentelemetry/instrumentation-pino pino instrumentation.cjs
const { NodeSDK } = require("@opentelemetry/sdk-node");
const { PinoInstrumentation } = require("@opentelemetry/instrumentation-pino");
const sdk = new NodeSDK({
instrumentations: [new PinoInstrumentation()],
});
sdk.start(); app.cjs
const { trace } = require("@opentelemetry/api");
const pino = require("pino");
const logger = pino();
const tracer = trace.getTracer("checkout");
tracer.startActiveSpan("charge-card", (span) => {
logger.info({ orderId: "ord_812" }, "charging card");
span.end();
}); OTEL_TRACES_EXPORTER=console node --require ./instrumentation.cjs app.cjs {"level":30,"time":1795291418372,"pid":12491,"hostname":"web-1","trace_id":"9bacbc55a63dfebdbc6f170321840877","span_id":"d211ecde634c5590","trace_flags":"01","orderId":"ord_812","msg":"charging card"}
Lines logged inside a span get trace_id, span_id and trace_flags; lines outside any span get none. The logKeys option renames the fields if your log pipeline expects other names. With a logs exporter configured, the same instrumentation also sends each record over OTLP. Two things kept the fields from appearing when we tested this:
- No tracer provider. With
OTEL_TRACES_EXPORTER=none, there was no real span to read, so no fields were added. Configure a real exporter, which you need anyway. - ES modules. An app using
importinstead ofrequirewas not patched until the OpenTelemetry module hook was registered first.instrumentation.mjshere is the same file asinstrumentation.cjs, written withimport:
register.mjs
// register.mjs, loaded with: node --import ./register.mjs app.mjs
import { register } from "node:module";
register("@opentelemetry/instrumentation/hook.mjs", import.meta.url);
await import("./instrumentation.mjs"); Python logging
The opentelemetry-instrumentation-logging package adds otelTraceID, otelSpanID, otelServiceName and otelTraceSampled to every LogRecord. Reference them in your format string:
pip install opentelemetry-sdk opentelemetry-instrumentation-logging app.py
import logging
from opentelemetry import trace
from opentelemetry.instrumentation.logging import LoggingInstrumentor
from opentelemetry.sdk.trace import TracerProvider
trace.set_tracer_provider(TracerProvider())
LoggingInstrumentor().instrument(
set_logging_format=True,
logging_format="%(levelname)s %(name)s trace_id=%(otelTraceID)s span_id=%(otelSpanID)s %(message)s",
)
logger = logging.getLogger("checkout")
tracer = trace.get_tracer("checkout")
logger.info("outside any span")
with tracer.start_as_current_span("charge-card"):
logger.info("charging card for order %s", "ord_812") INFO checkout trace_id=0 span_id=0 outside any span
INFO checkout trace_id=939f79e8bc99c8fd33baae1d5ba49344 span_id=ddb23a049b632335 charging card for order ord_812
Outside a span the IDs are 0. Two details from testing version 0.62b1. The fields are only added when correlation is on, through set_logging_format=True or OTEL_PYTHON_LOG_CORRELATION=true; without either, a format string that references them fails with KeyError: 'otelTraceID'. And correlation calls logging.basicConfig() itself, which only works once per process, so a later basicConfig(format=...) in your code is ignored. Pass your format as logging_format, or set OTEL_PYTHON_LOG_FORMAT, instead. Newer releases of the package may add options for this, so check its docs for the version you install. The example sets up a tracer provider with no exporter because it only shows the log fields; in your app, keep the exporter you already configure.
For JSON logs, add the same record attributes as fields in your JSON formatter. Using Loguru instead of the standard library? Its guide shows how to forward Loguru records with their trace context.
Fixing broken traces
A broken trace shows up as a trace whose root span is not the real entry point, or as two traces that should be one. Find the last span of the first piece and the first span of the second: the hop between them is where context was lost. Then check, in order:
- Missing instrumentation. The caller's HTTP client or the receiver's framework is not instrumented. Add it.
- Headers dropped in transit. A proxy, API gateway or service mesh strips unknown headers. Allow
traceparent,tracestateandbaggage. - A queue in the middle. Propagate through message headers as shown above.
- Lost in-process context. Work handed to a thread pool, a callback-based library or a manually created promise may run without the active context. Pass the context explicitly, or wrap the callback with
context.bind()in Node.js. - Different propagators. One side sends
b3and the other reads onlytracecontext. List both formats inOTEL_PROPAGATORSduring a migration. - Mixed sampling. A service ignoring the sampled flag records children of spans that were never kept. Use parent-based samplers.
Maple
From a log line to its trace
Maple links logs to traces by trace ID, so a log line with trace_id opens the request it came from.