Percentiles (p50, p95, p99)

David Granzin By David Granzin

A percentile is the value below which a given share of measurements fall: p99 latency is the response time that 99% of requests beat and the slowest 1% exceed.

An example

Ten requests to an endpoint took these times, sorted:

Request12345678910
ms4042454850525560651,900

The median (p50) is about 51 ms: half the requests were faster. The average is 236 ms, a number no request actually took, pulled up by one slow outlier. The slow request is the one a user complained about, and only the high percentiles show it. With ten samples the nearest-rank p99 is the slowest one (interpolating methods give a value between the two slowest); with real traffic of thousands of requests a minute, p99 is the time that the slowest 1% exceed.

p50, p95, p99: what each tells you

  • p50 is the typical experience. If it moves, most users are affected.
  • p95 is the slow edge of normal: one request in twenty is slower.
  • p99 is the tail. At 1,000 requests a minute, ten requests every minute are slower than p99. Users who make many requests per page view hit the tail often, so it matters more than "1%" suggests.
  • Max is one request. It is useful for spotting timeouts and useless as a trend.

When p99 rises and p50 does not, a subset of requests got slower: one route, one customer, one dependency. That calls for traces of the slow requests, as the API latency guide walks through.

How percentiles are computed

Storing every duration and sorting it is exact but expensive. Metrics systems record a histogram instead: counts of requests per latency bucket, such as 0 to 50 ms and 50 to 100 ms. The percentile is estimated from the bucket counts, assuming values are spread evenly inside each bucket. The estimate is only as good as the buckets: if p99 falls in a bucket from 1 to 2.5 seconds, the reported value is a guess inside that range. Exponential histograms use buckets that grow by a fixed ratio and rescale to fit the data, which bounds the relative error of the estimate.

Percentiles cannot be averaged. The mean of p99 across ten servers is not the p99 of all their requests. Merge the histograms first, then compute the percentile, which is why backends aggregate histogram buckets and not precomputed percentiles. For explicit-bucket histograms, merging only works when they share bucket boundaries, so keep boundaries the same across services that report the same metric. Exponential histograms can be merged across scales.

Percentiles in SLOs

Latency SLOs are usually written as a percentile or a threshold: "p99 under 500 ms", or "99% of requests faster than 500 ms". The second form turns directly into an error budget, because it counts requests over the threshold instead of estimating a percentile. It is exact when the threshold is a bucket boundary, so choose boundaries that include your SLO thresholds.

Sources

오늘 첫 트레이스를 확인하세요.

SDK를 추가하고 OTLP 익스포터를 Maple로 향하게 하면 트레이스가 도착합니다.

maple.dev: OpenTelemetry 위의 옵저버빌리티