OpenTelemetry 11 min read

OpenTelemetry sampling

Keep the traces that matter and drop the rest: head sampling in the SDK, tail sampling in the Collector, and what each one does to your metrics and your bill.

David Granzin By David Granzin

Sampling means recording or keeping only some traces. A service handling a thousand requests a second produces a thousand traces a second, most of them fast, successful and identical. Sampling keeps enough of them to understand normal behaviour and, done well, every trace you would want during an incident.

OpenTelemetry can sample in two places: in the SDK when a trace starts (head sampling) or in a Collector after the trace has finished (tail sampling). This guide covers both, with configs tested on Collector v0.165.0 and the JavaScript and Python SDKs.

Key takeaways

  • Use parentbased_traceidratio in every service, so only the root span decides and traces stay whole.
  • Head sampling is cheap but blind: it drops errors at the same rate as successes.
  • Tail sampling keeps every error and slow trace, but needs all spans of a trace on one Collector instance.
  • Sampling traces does not sample SDK metrics. Metrics derived from spans need the sampling weight, or computing before sampling.
  • Head sampling saves cost from the application onward. Tail sampling only saves what happens after the Collector.

Why sample

Trace volume grows with traffic and with the number of spans per request. A request that crosses five services with a database call in each produces at least ten spans. Storing every one costs ingest, storage and query time, and most of them answer no question anyone will ask.

Sampling has a price too. A dropped trace is gone: if a customer reports a failed checkout and that trace was sampled out, you cannot look at it. The aim is to drop the boring traces and keep the interesting ones, and the two approaches differ in how well they can tell them apart.

Sample traces, not spans. Dropping individual spans leaves traces with holes, which is worse than having fewer traces. Every approach in this guide decides per trace, using the trace ID so that different services and Collectors reach the same answer.

Head sampling

Head sampling decides when the first span of a trace is created. The SDK looks at the new trace ID, compares it with a threshold, and either records the trace or does not. An unsampled span still exists in memory so that context keeps flowing, but it records nothing and is never exported, so it costs almost nothing.

The SDK ships four building blocks:

SamplerDecision
AlwaysOnKeep every span
AlwaysOffKeep nothing
TraceIdRatioBased(0.1)Keep about 10% of traces, chosen by a fixed function of the trace ID
ParentBased(root)Root spans use the root sampler; every other span copies its parent's decision

Because TraceIdRatioBased decides from the trace ID, two services with the same ratio make the same decision for the same trace. Two services with different ratios do not, and that is where parent-based sampling comes in.

Parent-based sampling

When a request arrives with a traceparent header, the header's last field says whether the caller sampled the trace: 01 for sampled, 00 for not. A ParentBased sampler honours it. The service at the edge decides once, and every service downstream follows, so a trace is either recorded everywhere or nowhere.

Without it, each service decides on its own. A frontend at 10% and an API at 50% give you API spans whose parent span was never recorded: traces that start in the middle. A plain TraceIdRatioBased sampler in every service with the same ratio happens to agree, until somebody changes one ratio. Use the parent-based form.

The default sampler is parentbased_always_on: record every new trace and follow the parent otherwise. If you never configured sampling, you are keeping everything.

Configure the SDK

Every SDK reads two environment variables, which is the easiest way to set the same policy across languages:

export OTEL_TRACES_SAMPLER="parentbased_traceidratio"
export OTEL_TRACES_SAMPLER_ARG="0.1"   # keep 10% of new traces

The same in code, for Node.js and Python:

instrumentation.ts

import { NodeSDK } from "@opentelemetry/sdk-node";
import { ParentBasedSampler, TraceIdRatioBasedSampler } from "@opentelemetry/sdk-trace-base";

const sdk = new NodeSDK({
  // Roots: keep 10%. Children: do what the parent did.
  sampler: new ParentBasedSampler({ root: new TraceIdRatioBasedSampler(0.1) }),
});
sdk.start();

tracing.py

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased

# Roots: keep 10%. Children: do what the parent did.
provider = TracerProvider(sampler=ParentBased(root=TraceIdRatioBased(0.1)))
trace.set_tracer_provider(provider)

We ran the JavaScript version (SDK 2.12) against 10,000 traces of two spans each. It kept 984 traces, about the 10% asked for, and every kept child span belonged to a kept root. That is the behaviour you want: fewer traces, none of them broken.

Head-style sampling in the Collector

The probabilistic_sampler processor applies the same idea in a Collector: it hashes the trace ID and keeps a percentage. Use it when you cannot change the services' configuration, or when the Collector receives data from a source with no sampler.

processors:
  probabilistic_sampler:
    sampling_percentage: 10
    hash_seed: 22   # the same value on every Collector in this tier

Every Collector in the same tier must use the same hash_seed, or each makes a different decision for the same trace. The processor also records what it did: kept spans get a threshold in their tracestate, such as ot=th:c for 25%, which a backend can use to work out how many requests each kept span stands for. The JavaScript SDK's ratio sampler, in our test, wrote no threshold, so a backend sees those spans as unsampled.

The limit of every head approach is the same. The decision happens before the request runs, so a trace that will end in a 500 error is as likely to be dropped as one that succeeds. At 10% sampling, 90% of your errors have no trace.

Tail sampling

Tail sampling waits until a trace's spans have arrived, then decides with the whole trace in view. It can keep every trace with an error, every trace slower than two seconds, every trace for one important route, and a small share of everything else. The tail_sampling processor in the Collector does this.

The SDKs must send everything for this to work, so services run with the default parentbased_always_on sampler, or a generous head ratio, and the Collector does the dropping.

A tail sampling config

The processor section of a gateway Collector. Receivers and the otlp_http exporter are as in the Collector guide; the complete file passes otelcol-contrib validate on v0.165.0.

gateway tier 2: sampler

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    expected_new_traces_per_sec: 2000
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 2000
      - name: keep-checkout
        type: string_attribute
        string_attribute:
          key: http.route
          values: [/api/checkout]
      - name: sample-the-rest
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling]
      exporters: [otlp_http]

How it behaves:

  • Buffering. The processor holds each trace's spans in memory for decision_wait (default 30 seconds, 10 here), then evaluates the policies. Pick a value longer than your slowest normal trace.
  • Policies vote. The trace is kept if any policy says sample. A drop policy overrides every sample vote, which is how you exclude health checks. Here a trace is kept if it has an error, or took over 2 seconds, or hit /api/checkout, or lands in the random 10%.
  • Capacity. num_traces is how many traces fit in memory at once (default 50,000). A trace pushed out before its decision is dropped unsampled. Size it above your new traces per second times decision_wait: 2,000 traces a second for 10 seconds needs 20,000, so 100,000 leaves room for spikes.
  • Late spans. A span that arrives after its trace was decided follows that decision while the trace is still in the buffer. After that it is treated as a new trace. decision_cache keeps decisions for longer if your traces have stragglers.

Order matters

Put tail_sampling after processors that add context its policies match on, such as k8s_attributes, and after any redaction that deletes attributes, keeping in mind that a policy on a deleted attribute never matches.

Scaling it

One Collector instance can only decide on a trace if it received every span of it. With several gateway replicas behind an ordinary load balancer, spans of one trace land on different replicas, and each sees a fragment: the error is on one, the slow span on another, and both decide wrongly.

The fix is two tiers. The first tier runs the load_balancing exporter with routing_key: traceID, which sends all spans of a trace to the same second-tier instance. The second tier runs tail_sampling. The Collector guide has the first tier's config.

Routing depends on the set of sampler instances. When one is added, removed or restarted, some trace IDs move to a different instance, and traces in flight at that moment are judged on partial data. Scale the sampler tier deliberately, not on every traffic blip, and expect a few incomplete decisions around each change.

Watch the processor's own metrics. Traces dropped from the buffer early mean num_traces is too small; the processor's README suggests comparing a low percentile, such as p1, of its sampling_trace_removal_age metric with decision_wait. Memory grows with traffic, so give the sampler tier a memory_limiter and alerts like any other Collector.

What sampling does to metrics and cost

Metrics recorded with the metrics API, such as a request counter or a duration histogram, are not affected by trace sampling. Each request still increments the counter whether or not its trace was kept.

Metrics computed from spans are affected. If a dashboard counts server spans to show requests per second, 10% head sampling makes it read 10% of the real rate. Tail sampling distorts it in a less obvious way: a sampler that keeps every error but 10% of successes makes the error rate look roughly ten times higher than it is.

There are two ways to get correct numbers from sampled traces:

  • Weight each span. If a span records the probability it was kept with, a backend can count it as 1 ÷ probability requests. The OpenTelemetry convention for this is the th threshold in tracestate, written by the probabilistic_sampler processor. Policies that keep every error have no single probability, so weighting works best for random sampling.
  • Compute before sampling. The span_metrics connector turns every span into request, error and duration metrics before the sampler drops any. The metrics are exact and the stored traces are a sample.

gateway tier 2: exact span metrics

connectors:
  span_metrics:

service:
  pipelines:
    traces/all:            # every span, before sampling
      receivers: [otlp]
      exporters: [span_metrics]
    traces/sampled:        # what you store
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling]
      exporters: [otlp_http]
    metrics/spans:         # request, error and duration metrics from all spans
      receivers: [span_metrics]
      exporters: [otlp_http]

Cost follows where the drop happens. Head sampling saves at every step: the application does less work, less data crosses the network, and the backend ingests and stores less. Tail sampling saves only after the Collector: services still produce and send every span, and you now run a stateful Collector tier sized for full traffic. It pays off when backend ingest and storage are the expensive part, which they usually are.

Logs are a separate stream. Sampling traces does not sample logs, so a log line can carry the trace ID of a trace that was dropped. That is expected: the log is still useful on its own, it just has no trace to open. If you want logs to follow trace decisions, the probabilistic_sampler processor can sample logs by their trace ID with the same seed as your traces.

Which to use

SituationApproach
Low traffic, or you are just startingNo sampling. Keep everything until cost says otherwise.
High volume of similar requests, cost firstparentbased_traceidratio in every SDK
You must see every error and slow requestTail sampling in a gateway, SDKs sending everything
Very high volume and you need errorsA generous head ratio (say 25%) to cut volume, then tail sampling on what remains
You cannot change the servicesprobabilistic_sampler or tail_sampling in a Collector

Whatever you choose, make the services agree. Mixed samplers are the most common cause of broken traces after sampling is turned on, together with a service that does not forward the traceparent header. The context propagation guide covers the second one.

Maple

Sampled traces, unsampled numbers

Maple reads the th threshold in tracestate, or a SampleRate attribute, and weights each span at ingest, so throughput and error rates estimate the traffic before sampling.

Sources and further reading

FAQ

Common questions

What is the difference between head and tail sampling?
Head sampling decides whether to keep a trace when it starts, usually from the trace ID alone, so it is cheap but cannot know whether the trace will be slow or fail. Tail sampling decides after the trace's spans have arrived, so it can keep every error and slow trace, but it must buffer whole traces in a Collector.
What does parentbased_traceidratio mean?
It is an SDK sampler setting. For a new trace (a root span), it keeps the fraction of traces given in OTEL_TRACES_SAMPLER_ARG, decided from the trace ID. For any span with a parent, it follows the parent's decision, so a trace is either kept in every service or dropped in every service.
What is the default OpenTelemetry sampler?
parentbased_always_on. Every new trace is recorded, and child spans follow their parent's decision.
Does sampling affect metrics?
Metrics from the metrics SDK are not sampled. Metrics computed from spans are: a request rate counted from sampled spans is too low by the sampling factor unless each span carries its weight. Compute span metrics before sampling, for example with the span_metrics connector, or use a backend that reads the sampling weight.
Why are my traces missing spans after I enabled sampling?
Usually because services decide independently: one uses a plain ratio sampler or a different ratio, or a service in the middle does not propagate context. Use parent-based samplers everywhere so only the root decides, and make sure every service forwards the traceparent header.

See your first trace today.

Add the SDK, point your OTLP exporter at Maple, and traces arrive.

maple.dev: observability on OpenTelemetry