Metric cardinality is the number of distinct time series a metric produces. It is the main driver of metrics cost, and it is easy to raise by accident: add user.id to a request counter and the metric now has one series per user.
This guide explains how series multiply, how that turns into storage and money, how to find the attribute responsible, and how to fix it at the source, in the SDK and in the Collector. The SDK and Collector examples were run against current OpenTelemetry releases.
Key takeaways
- Every unique set of attribute values is a separate time series. Series counts multiply across attributes.
- Histograms multiply again by their bucket count.
- IDs, full URLs, IP addresses and error messages do not belong on metrics. Put them on spans and logs.
- Fix it where the metric is recorded: bounded attributes and SDK views beat cleanup downstream.
- A series with
otel.metric.overflow=truemeans a metric hit the SDK's cardinality limit.
What cardinality is
A time series is a metric name plus one specific set of attribute values (Prometheus calls them labels). These are three different series of the same histogram:
http.server.request.duration{http.request.method="GET", http.route="/orders/{id}", http.response.status_code="200"}
http.server.request.duration{http.request.method="GET", http.route="/orders/{id}", http.response.status_code="404"}
http.server.request.duration{http.request.method="POST", http.route="/orders", http.response.status_code="201"}
The metric's cardinality is how many such combinations actually occur. As an upper bound it is the product of each attribute's distinct values. Take http.server.request.duration with 5 methods, 40 routes and 10 status codes, reported by 20 instances whose service.instance.id differs:
| Attributes | Series |
|---|---|
| method × route × status | 5 × 40 × 10 = 2,000 |
| × 20 instances | 40,000 |
× user.id with 10,000 active users | 400,000,000 |
In practice not every combination occurs, since a route only returns some status codes. But the shape holds: adding an attribute multiplies, and an attribute with unbounded values multiplies without limit.
Histograms make it worse. A histogram series carries a count per bucket. The OpenTelemetry SDK's default explicit boundaries are 0, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000: 15 boundaries, 16 buckets. Stored the Prometheus way, that is 16 bucket series plus _sum and _count, so 18 series for every attribute combination. With those buckets, the 40,000 combinations above become 720,000 series. HTTP instrumentation usually sets its own boundaries in seconds (the semantic conventions advise 14), which changes the multiplier a little and the conclusion not at all.
How cardinality reaches your bill
Every series costs something at each stage:
- In the application. The SDK keeps one aggregation in memory per series and exports all of them each interval. The default export interval is 60 seconds.
- In transit. Each series becomes a data point in every export. 40,000 series at one point a minute is about 1.75 billion data points a month.
- In storage. Time series databases index each series. Memory use and query time grow with the number of active series, and queries that aggregate across a high-cardinality attribute read every one of them.
- On the invoice. Metrics products commonly bill per active series, per custom metric or per data point ingested. All three grow with cardinality.
Churn counts too. A series that appears for ten minutes and never again still had to be created, indexed and kept for the retention period. Attributes that change on every deploy, such as a pod name, create a new set of series per deploy, so a service deployed twenty times a day accumulates series even if its live count looks steady.
Common causes
| Attribute | Why it explodes | Use instead |
|---|---|---|
user.id, session.id, order or request IDs | One series per entity | Span attributes |
url.path, url.full | IDs inside paths: /orders/8812 | http.route: /orders/{id} |
client.address, network.peer.address | One series per IP | Nothing, or a coarse region |
exception.message, error text | Messages embed values | error.type |
db.query.text | One series per distinct statement | db.operation.name, db.collection.name |
k8s.pod.name, container.id | New values on every deploy and restart | k8s.deployment.name, or drop it from metrics |
| Tenant or customer ID | Grows with the business | Fine for a few hundred tenants; above that, a plan tier |
Auto-instrumentation libraries that follow the semantic conventions already use bounded attributes on their metrics. The usual source of trouble is custom metrics, where an attribute was added for one investigation and never removed, or a framework that reports the raw path because it does not know the route template.
Find the culprit
Start with the metric, then the attribute. In a Prometheus-compatible backend, count series per metric name. This query touches every series, so run it on a recent window and not on a dashboard:
# The 10 metric names with the most series
topk(10, count by (__name__) ({__name__=~".+"})) Then count each attribute's distinct values on the worst metric. With its default name translation, Prometheus stores url.path as url_path:
# How many distinct values one attribute has on one metric
count(count by (url_path) (http_server_request_duration_seconds_count))
Prometheus also shows the metric names and label names with the most series on its /tsdb-status page. Other backends have an equivalent: a cardinality explorer, a usage page, or a SQL table you can group by metric and attribute.
On the sending side, two signals help. A series with otel.metric.overflow=true means an SDK hit its cardinality limit, covered below. And a Collector's debug exporter with verbosity: detailed prints every data point's attributes, which shows quickly whether an unexpected attribute is present before anything reaches the backend.
Fix it
Fix cardinality as close to where the metric is recorded as you can. Each step downstream is less precise and costs more.
1. Record bounded attributes
The best fix is to not record the value. Replace the path with the route template, the error message with an error type, the user ID with nothing. If you need a per-user answer, a metric is the wrong tool; see where the detail goes.
2. Drop attributes with an SDK view
When the metric comes from a library you cannot change, a view in the metrics SDK rewrites it before aggregation. An allow-list of attribute keys keeps the ones you name and discards the rest, and the SDK aggregates the result correctly: points that only differed by a dropped attribute are summed.
Node.js
import { MeterProvider, createAllowListAttributesProcessor } from "@opentelemetry/sdk-metrics";
const meterProvider = new MeterProvider({
readers: [reader],
views: [
{
// Keep only these three attributes on this histogram.
instrumentName: "http.server.request.duration",
attributesProcessors: [
createAllowListAttributesProcessor(["http.request.method", "http.route", "http.response.status_code"]),
],
},
],
}); Python
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.view import View
provider = MeterProvider(
metric_readers=[reader],
views=[
# Keep only these three attributes on this histogram.
View(
instrument_name="http.server.request.duration",
attribute_keys={"http.request.method", "http.route", "http.response.status_code"},
),
],
)
We recorded 500 measurements with 500 different user.id values through each of these. Both SDKs exported a single data point with the three allowed attributes.
3. Keep the cardinality limit as a safety net
The OpenTelemetry metrics SDK caps how many attribute sets one metric can hold per collection: 2,000 by default. Past the limit, new combinations are folded into a single overflow series carrying otel.metric.overflow=true. No measurement is lost or counted twice, but the overflow series has no breakdown, so the metric stops being useful for whatever exceeded the limit. In the JavaScript SDK, a view can set a lower limit per instrument:
views: [
{ instrumentName: "orders.created", aggregationCardinalityLimit: 3 },
], With a limit of 3 and ten different customer.id values, the export held two real series and the overflow:
orders.created points 3 [{"customer.id":"c_0"},{"customer.id":"c_1"},{"otel.metric.overflow":true}] Alert on the overflow series. It is the earliest sign that a deploy added an unbounded attribute.
4. Re-aggregate in the Collector
For metrics you do not instrument, such as scraped Prometheus endpoints or a vendor agent, fix them in a Collector. Three processors cover most cases:
config.yaml
processors:
transform/route:
metric_statements:
- context: datapoint
statements:
- replace_pattern(attributes["url.path"], "/orders/[0-9]+", "/orders/{id}")
metrics_transform/drop-user:
transforms:
- include: http.server.request.count
action: update
operations:
- action: aggregate_labels
label_set: [http.request.method, url.path]
aggregation_type: sum
filter/debug-metrics:
metrics:
metric:
- 'IsMatch(name, "^debug\\.")' transformrewrites values: here, order IDs in paths become/orders/{id}.metrics_transformwithaggregate_labelskeeps only the listed attributes and sums points that become identical. In our test, three points for three users collapsed into one point with their total.filterdrops whole metrics nobody uses.
Do not just delete the attribute
The attributes processor can delete user.id from metric points, but it does not merge the points that are now identical. In our test, three points for three users came out as three points with the same attributes and a value of 5 each, where the right answer was one point of 15. Depending on the backend, such duplicates overwrite each other or are rejected. The same applies to transform on its own, which is why aggregate_labels runs after it. List the processors in the pipeline in the order shown.
The Collector receives the full cardinality before reducing it, so applications still pay for the memory and network of the original series. Treat Collector fixes as the place for metrics you cannot change at the source.
Where the detail goes
High cardinality is a problem for metrics, not for observability. Spans and logs are stored per event, so a user.id on a span costs a few bytes, while the same attribute on a metric multiplies the series count. A good split:
- Metrics answer "how many, how fast, how often" with bounded dimensions: route, method, status, region, version.
- Spans carry the specifics: user, tenant, order ID, query text. When a metric shows a slow route, the spans for that route tell you which users and which queries.
- Exemplars join the two: a metric point can carry the trace ID of one measurement, so a spike in a latency chart links to a trace from that spike.
For SLOs and alerts, keep the metric. For "which customer was affected", query the spans. Read the cardinality glossary entry for the short version, or go back to part 1 of this series to set up the Collector these fixes run in.
Maple
Metrics with the traces behind them
Send OTLP metrics, traces and logs to Maple and keep per-request detail on spans, where it costs bytes and not series.