Use Cases

The p99 moved. Find out which release moved it.

An SLO is only useful if a breach points somewhere. This one goes from a threshold crossing to the commit that caused it in three surfaces, without leaving the same time range.

signal checkout p99 380 ms · threshold 200 ms

The path

Threshold to commit, in three minutes

One SLO holds the whole path together: checkout p99 at or under 200 ms. Every surface below is filtered to the window where it stopped being true.

  1. alert T+0:00 · slocheckout p99 ≤ 200 ms

    The alert states the breach, not the symptom

    The rule is written against the SLO, so the notification carries the threshold, the observed value and the evaluation window. 380 ms against 200 ms, sustained across three windows.

    Alert rules listed with severity, signal type and current state
  2. dashboard T+0:40 · slocheckout p99 ≤ 200 ms

    The dashboard narrows it to one endpoint

    p50 has not moved. p99 has. That shape means a slow tail on one route rather than a service-wide slowdown, and the route breakdown names it: POST /checkout.

    The overview dashboard with request volume, error rate, latency and log volume charts
  3. releases T+3:15 · slocheckout p99 ≤ 200 ms

    Two releases, side by side

    Split the same endpoint by service.version and the step change has an owner: p99 120 ms on abc123, 380 ms on def456. The traces under def456 carry 50 db.query spans where the earlier ones carry 3.

    Two releases of one endpoint compared on p50, p95, p99 and error rate

outcome

Reverted def456 at T+9:20. p99 back under the threshold on the next evaluation window.

9 min
alert to cause
120 → 380 ms
p99, before and after
3 → 50
db spans per request

What it does

What made the path short

p99 > 200ms
The alert knows its own SLO
Rules are written against a percentile and a threshold, so a firing alert already contains the comparison. Nothing has to be looked up to know how far out it is.
service.version
Version is an attribute, so it is a filter
service.version rides on every span the SDK sends. Comparing two releases is a group-by, not an export into a spreadsheet.
select range
The percentile keeps its traces
p99 is computed from spans that are still there. Selecting the spike on the chart opens the requests inside it, so the number and the evidence are never separate artifacts.

Keep going

The same data, other jobs

Point OTLP at Maple.

One endpoint, one key. Traces, logs, metrics and sessions land on the same trace id from the first request.