# Maple Documentation

> Every Maple doc, concatenated. Source: https://maple.dev/docs

# Introduction

> Getting Started · https://maple.dev/docs/getting-started/introduction

What Maple is, what it includes, and how to send your first telemetry.

Maple is an observability platform built on OpenTelemetry. Your applications send traces, logs and metrics over OTLP, the protocol every OpenTelemetry SDK already speaks, and Maple stores them and lets you query, chart and alert on them. There is no proprietary agent to install.

<figure class="shot">
  <img src="/screenshots/docs/introduction-01-overview.webp" alt="Maple's Overview page for a production environment over the last 24 hours: an All clear alert strip, service health counts, totals for logs, traces, metrics and data size, and request volume and error rate charts." loading="lazy" />
  <figcaption>The Overview page: service health, signal totals, and request volume at a glance.</figcaption>
</figure>

## What's in Maple

- [Traces](/docs/explore/traces): search spans and open a trace to see every span in it.
- [Logs](/docs/explore/logs): search logs and jump from a log line to the trace that produced it.
- [Metrics](/docs/explore/metrics): browse and chart the metrics your services export.
- [Services](/docs/explore/services): throughput, error rate and latency per service.
- [Service map](/docs/explore/service-map): the calls between your services, built from trace data.
- [Errors](/docs/errors/overview): exceptions grouped into issues that you can assign and track to done.
- [Alerts](/docs/alerting/alert-rules): rules on any signal, with notifications to your destinations.
- [Dashboards](/docs/dashboards/build-dashboards): charts and tables built from queries.
- [Replays](/docs/session-replay/replays): recordings of browser sessions, linked to their traces.
- [Web analytics](/docs/product-events/web-analytics): page views and visitors from the browser SDK.
- [Agent Sessions](/docs/agent-sessions/overview): AI agent conversations, with every model call and tool call.
- Infrastructure: [Hosts](/docs/infrastructure/hosts), [Kubernetes](/docs/infrastructure/kubernetes), [Containers](/docs/infrastructure/docker), [Cloudflare](/docs/integrations/cloudflare), [Google Cloud](/docs/integrations/gcp) and [PlanetScale](/docs/integrations/planetscale).

## Getting started

For a copy-paste walkthrough, including a curl command that sends a test span without an SDK, follow the [Quickstart](/docs/getting-started/quickstart). The short version:

1. Sign up at [app.maple.dev](https://app.maple.dev). During onboarding you pick the region your organization lives in (US or EU).
2. Open **Settings → Ingestion** and copy an ingest key. Use the private key (`maple_sk_…`) for server applications and the public key (`maple_pk_…`) for browser code.
3. Point an OpenTelemetry SDK at the ingest endpoint for your region:

    | Region         | Ingest endpoint               |
    | -------------- | ----------------------------- |
    | United States  | `https://ingest.maple.dev`    |
    | European Union | `https://ingest.eu.maple.dev` |

    A key only works in the region it was created in. See [Regions](/docs/reference/regions).

4. Send a request to your application, then open **Explore → Traces** in Maple and check that a trace for it appears.

With the standard OpenTelemetry environment variables, step 3 looks like this:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"   # EU: https://ingest.eu.maple.dev
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
export OTEL_SERVICE_NAME="my-service"
```

If you use a coding agent, the [maple-onboard](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-onboard) skill instruments a repository for you, and [maple-audit](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-audit) checks an existing OpenTelemetry setup against Maple's conventions.

## Language guides

- [Effect SDK](/docs/sdks/effect): Maple's SDK for Effect applications
- [Node.js](/docs/guides/instrumentation-nodejs): Express, Fastify, Hono
- [Next.js](/docs/guides/instrumentation-nextjs): App Router, Pages Router, middleware
- [Python](/docs/guides/instrumentation-python): FastAPI, Django, Flask
- [Go](/docs/guides/instrumentation-go)
- [Rust](/docs/guides/instrumentation-rust): through the `tracing` crate
- [Java](/docs/guides/instrumentation-java): the OpenTelemetry Java agent
- [C# / .NET](/docs/guides/instrumentation-csharp)
- [Kotlin](/docs/guides/instrumentation-kotlin): Ktor and Spring Boot
- [Laravel](/docs/guides/instrumentation-laravel): Eloquent, queues, HTTP client

The attributes Maple reads for the service map, error grouping and throughput are listed in [OpenTelemetry conventions](/docs/concepts/otel-conventions).

## Next steps

- [Quickstart](/docs/getting-started/quickstart): your first trace in about five minutes.
- [Instrument your application](/docs/instrumentation): every language and framework guide.
- [OpenTelemetry conventions](/docs/concepts/otel-conventions): the attributes Maple reads.

---

# Quickstart

> Getting Started · https://maple.dev/docs/getting-started/quickstart

Send your first trace to Maple in about five minutes, from your own stack or with a single curl request.

import LanguageTabs from "../../../components/docs/LanguageTabs.astro"
import LanguageTab from "../../../components/docs/LanguageTab.astro"

You need a Maple account and either an app to instrument or a terminal with `curl`.

## 1. Sign up and pick a region

Sign up at [app.maple.dev](https://app.maple.dev). During onboarding you pick the region your organization lives in, US or EU. The region cannot be changed later, and each region has its own ingest endpoint:

| Region         | Ingest endpoint               |
| -------------- | ----------------------------- |
| United States  | `https://ingest.maple.dev`    |
| European Union | `https://ingest.eu.maple.dev` |

## 2. Copy your ingest key

Open **Settings → Ingestion** and copy the private key (`maple_sk_…`). It is for servers and scripts. The public key (`maple_pk_…`) is for browser code. Both can only send telemetry. See [Authentication](/docs/reference/authentication#ingest-keys).

## 3. Set the exporter variables

Every OpenTelemetry SDK reads these standard variables:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"   # EU: https://ingest.eu.maple.dev
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_SERVICE_NAME="quickstart"
```

The endpoint is the base URL. Exporters append `/v1/traces`, `/v1/logs` and `/v1/metrics` themselves.

## 4. Send a trace

Pick your stack. Each setup reads the variables from step 3, so run it in the same shell.

<LanguageTabs
  label="Language or framework"
  tabs={[
    { id: "node", label: "Node.js" },
    { id: "nextjs", label: "Next.js" },
    { id: "python", label: "Python" },
    { id: "go", label: "Go" },
    { id: "java", label: "Java" },
    { id: "csharp", label: ".NET" },
    { id: "laravel", label: "Laravel" },
    { id: "effect", label: "Effect" },
    { id: "curl", label: "curl" },
  ]}
>

<LanguageTab id="node">

Install the auto-instrumentations package and start your app through its `register` entry point. No code changes:

```bash
npm install @opentelemetry/api @opentelemetry/auto-instrumentations-node
node --require @opentelemetry/auto-instrumentations-node/register app.js
```

Send a request to your app. Each incoming HTTP request becomes a trace, with child spans for outgoing calls and database queries. For ES module apps and the full setup with logs and metrics, see the [Node.js guide](/docs/guides/instrumentation-nodejs).

</LanguageTab>

<LanguageTab id="nextjs">

Install `@vercel/otel`:

```bash
npm install @vercel/otel @opentelemetry/api
```

Create `instrumentation.ts` in the project root (or `src/instrumentation.ts` if your code lives in `src/`). With no exporter passed, `@vercel/otel` reads the endpoint and headers from the variables:

```typescript
// instrumentation.ts
import { registerOTel } from "@vercel/otel"

export function register() {
	registerOTel({ serviceName: "quickstart" })
}
```

Start the dev server with `npm run dev` and open a page. Each request becomes a trace with spans for rendering, route handlers and server `fetch()` calls. On Next.js 13.4 and 14, enable the instrumentation hook first; see the [Next.js guide](/docs/guides/instrumentation-nextjs), which also covers logs and metrics.

</LanguageTab>

<LanguageTab id="python">

Install the distro and the HTTP exporter, then let `opentelemetry-bootstrap` add the instrumentation packages for the libraries you already use:

```bash
pip install opentelemetry-distro opentelemetry-exporter-otlp-proto-http
opentelemetry-bootstrap -a install
```

Start your app through the `opentelemetry-instrument` wrapper. No code changes:

```bash
opentelemetry-instrument python app.py
# or: opentelemetry-instrument uvicorn main:app
```

Send a request to your app. FastAPI, Django, Flask and the common HTTP and database clients are traced automatically. Keep `OTEL_EXPORTER_OTLP_PROTOCOL` from step 3: the distro defaults to gRPC, which Maple does not accept. The wrapper does not work with `uvicorn --reload`. For logs, metrics and a setup in code, see the [Python guide](/docs/guides/instrumentation-python).

</LanguageTab>

<LanguageTab id="go">

Go has no zero-code agent, so add the SDK and wrap your HTTP handler:

```bash
go get go.opentelemetry.io/otel \
  go.opentelemetry.io/otel/sdk \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp \
  go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
```

```go
package main

import (
	"context"
	"log"
	"net/http"

	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

func main() {
	ctx := context.Background()

	// Reads OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS.
	// A setup error only disables tracing; it never stops the app.
	if exporter, err := otlptracehttp.New(ctx); err != nil {
		log.Printf("telemetry disabled: %v", err)
	} else {
		tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exporter))
		defer tp.Shutdown(ctx)
		otel.SetTracerProvider(tp)
	}

	mux := http.NewServeMux()
	mux.HandleFunc("GET /hello", func(w http.ResponseWriter, r *http.Request) {
		w.Write([]byte("hello"))
	})
	log.Fatal(http.ListenAndServe(":8080", otelhttp.NewHandler(mux, "server")))
}
```

Run it with `go run .` and call `curl localhost:8080/hello`. Each request becomes a server span, named after `OTEL_SERVICE_NAME`. For logs, metrics, gRPC and `database/sql`, see the [Go guide](/docs/guides/instrumentation-go).

</LanguageTab>

<LanguageTab id="java">

Download the OpenTelemetry Java agent and attach it when you start your app. No code changes:

```bash
curl -L -o opentelemetry-javaagent.jar \
  https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
java -javaagent:opentelemetry-javaagent.jar -jar app.jar
```

Send a request to your app. The agent instruments Spring, servlet containers, JDBC, HTTP clients and most other popular libraries, and exports traces, metrics and logs. It works the same for Kotlin and any other JVM app. See the [Java guide](/docs/guides/instrumentation-java) or the [Kotlin guide](/docs/guides/instrumentation-kotlin).

</LanguageTab>

<LanguageTab id="csharp">

Add the hosting extension, the OTLP exporter and ASP.NET Core instrumentation:

```bash
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
```

Register OpenTelemetry in `Program.cs`. `UseOtlpExporter()` reads the endpoint, protocol and headers from the variables:

```csharp
using OpenTelemetry;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing.AddAspNetCoreInstrumentation())
    .UseOtlpExporter();

var app = builder.Build();
app.MapGet("/", () => "Hello!");
app.Run();
```

Start it with `dotnet run` and send a request. Each request becomes a server span. For `HttpClient`, Entity Framework, logs and metrics, see the [.NET guide](/docs/guides/instrumentation-csharp).

</LanguageTab>

<LanguageTab id="laravel">

Install the Laravel OpenTelemetry package:

```bash
composer require keepsuit/laravel-opentelemetry
```

Laravel reads its configuration from `.env`, so put the values from step 3 there instead of exporting them:

```env
# .env
OTEL_SERVICE_NAME=quickstart
OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
```

Send a request to your app. Each request becomes a trace with spans for database queries, Redis commands and `Http` facade calls. If protobuf export fails in your environment, set `OTEL_EXPORTER_OTLP_PROTOCOL=http/json`. For queues, logs and metrics, see the [Laravel guide](/docs/guides/instrumentation-laravel).

</LanguageTab>

<LanguageTab id="effect">

Install Maple's Effect SDK:

```bash
npm install @maple-dev/effect-sdk effect
```

The SDK takes the key from `MAPLE_INGEST_KEY` rather than the OTLP headers variable, and uses `OTEL_EXPORTER_OTLP_ENDPOINT` from step 3 for the region:

```bash
export MAPLE_INGEST_KEY="YOUR_INGEST_KEY"
```

Provide the layer to your program:

```typescript
import { Maple } from "@maple-dev/effect-sdk"
import { Effect } from "effect"

const program = Effect.gen(function* () {
	yield* Effect.log("Hello from Effect!")
}).pipe(Effect.withSpan("hello-maple"))

Effect.runPromise(program.pipe(Effect.provide(Maple.layer({ serviceName: "quickstart" }))))
```

Run it once. The `hello-maple` span and its log line reach Maple together. For Effect 3, browsers and Cloudflare Workers, see the [Effect SDK](/docs/sdks/effect).

</LanguageTab>

<LanguageTab id="curl">

To check the key and endpoint before touching your app, post one span as OTLP JSON. No SDK needed:

```bash
NOW=$(date +%s)
curl -i "${OTEL_EXPORTER_OTLP_ENDPOINT}/v1/traces" \
  -H "Authorization: Bearer YOUR_INGEST_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "resourceSpans": [{
    "resource": { "attributes": [{ "key": "service.name", "value": { "stringValue": "quickstart" } }] },
    "scopeSpans": [{
      "spans": [{
        "traceId": "5b8efff798038103d269b633813fc60c",
        "spanId": "eee19b7ec3c1b174",
        "name": "GET /hello",
        "kind": 2,
        "startTimeUnixNano": "'"${NOW}"'000000000",
        "endTimeUnixNano": "'"${NOW}"'250000000",
        "status": { "code": 1 }
      }]
    }]
  }]
}'
```

A `200` response means Maple accepted the span. Change the `traceId` to send another trace with the same command.

</LanguageTab>

</LanguageTabs>

Rust, Kotlin and other stacks have their own guides under [Instrument your application](/docs/instrumentation). The variables from step 3 stay the same.

## 5. Verify

Open **Explore → Traces** and filter **Service** to `quickstart` (or your `OTEL_SERVICE_NAME`). The trace appears within about a minute. Click it to see its spans.

## Troubleshooting

- **`401` from the ingest endpoint.** The key is wrong, or it belongs to the other region. A US key is rejected by `ingest.eu.maple.dev` and the other way around. See [Regions](/docs/reference/regions).
- **Nothing shows up.** Maple accepts OTLP over HTTP only. An exporter set to `grpc` or pointed at port `4317` cannot reach it; use `http/protobuf` and the endpoint without a port. Set `OTEL_LOG_LEVEL=debug` to print export errors.
- **The trace is there, but under a name like `unknown_service`.** `OTEL_SERVICE_NAME` was not set in the shell (or `.env`) that started the app, so the SDK fell back to a default `service.name`.
- **Still nothing.** Check the time range on the Traces page. For the curl span, your machine's clock sets the timestamp.

More status codes and their meaning are in the [Ingest API reference](/docs/reference/ingest#status-codes).

## Next steps

- [Instrument your application](/docs/instrumentation): full guides per language, with logs and metrics.
- [Traces](/docs/explore/traces): search, filter and open traces.
- [Service map](/docs/explore/service-map): see calls between services once two of them are instrumented.
- [Alert rules](/docs/alerting/alert-rules): get notified when error rate or latency crosses a threshold.

---

# Use Maple with AI agents

> Getting Started · https://maple.dev/docs/getting-started/ai-agents

Connect Claude Code, Cursor, Windsurf or any MCP client to Maple, then let the agent debug production errors, chase latency, build dashboards, set up alerts and instrument your code.

Your coding agent can work with Maple directly. Connected to the [Maple MCP server](/docs/reference/mcp), it reads the same traces, logs, metrics and error issues you see in the dashboard, and it can act on them: triage an issue, build a dashboard, create an alert rule. Paired with the [GitHub integration](/docs/integrations/github), it can go from an exception to the line of code that raised it.

There are three ways agents fit in:

- **Query and operate Maple** through the MCP server. This page covers it.
- **Instrument your code** with the `maple-onboard` and `maple-audit` skills. See [Instrument with a coding agent](#instrument-with-a-coding-agent).
- **Monitor your own AI agents** in [Agent Sessions](/docs/agent-sessions/overview).

## Connect your agent

Open **Settings → MCP** in Maple. It shows the endpoint for your organization's region and a ready-made configuration for Claude Code, Cursor, Windsurf and other clients.

| Region | Endpoint                       |
| ------ | ------------------------------ |
| US     | `https://api.maple.dev/mcp`    |
| EU     | `https://api.eu.maple.dev/mcp` |

**With OAuth (recommended).** Add the endpoint URL to your client. On first use it opens Maple in your browser, you pick a workspace and approve access. No key to copy or rotate.

For Claude Code:

```bash
claude mcp add --transport http maple https://api.maple.dev/mcp
```

For Cursor, add this to `.cursor/mcp.json`:

```json
{
	"mcpServers": {
		"maple": {
			"type": "http",
			"url": "https://api.maple.dev/mcp"
		}
	}
}
```

**With an API key.** If your client does not support OAuth, create a key under **Settings → MCP** and send it as a Bearer token:

```json
{
	"mcpServers": {
		"maple": {
			"type": "http",
			"url": "https://api.maple.dev/mcp",
			"headers": { "Authorization": "Bearer maple_ak_…" }
		}
	}
}
```

Treat the key like a password. Keep it out of files you commit.

To check the connection, ask your agent: _"List my services in Maple."_ It should answer with your services and their error rates.

## What to ask

Plain questions work. The agent picks the tools. Some starting points:

- "What broke in the last hour?"
- "Why is `checkout` slower since yesterday's deploy?"
- "Show me the slowest traces for `POST /orders` today and explain where the time goes."
- "Which errors started after commit `a1b2c3d`?"
- "Build a dashboard for the payments service: throughput, error rate, p95 latency."
- "Alert me in Slack when the API error rate goes over 2% for 5 minutes."
- "Is my instrumentation missing anything?"

You get better answers when you name the **service**, the **time window** and the **environment**. Ask the agent to include trace IDs and the numbers it used, so you can open the evidence in Maple yourself.

## Common workflows

### Debug a production error

1. The agent finds the error with `find_errors` or `list_error_issues`.
2. `error_detail` gives sample occurrences, stack traces and related logs.
3. `inspect_trace` shows the full request: which service called which, and where it failed.
4. With the GitHub integration, `read_source_file` opens the code at the commit that was running.
5. The agent proposes a fix in your repository.

From the **Errors** page you can skip step 1: right-click an issue and choose **Copy agent prompt** to hand your agent the issue with its context.

### Work an error issue end to end

Agents can own issues the same way people do, using the lease described in [Errors](/docs/errors/overview):

1. `claim_error_issue` takes the lease, so two agents (or an agent and a teammate) do not work the same bug.
2. `comment_on_error_issue` records what the agent found.
3. `propose_fix` moves the issue to **In review**.
4. `link_pull_request` attaches the PR. When it merges, Maple watches for new occurrences and verifies the fix.

### Investigate a slowdown

`compare_periods` compares error rate, throughput and p95 between two windows and flags regressions per service. `find_slow_traces` and `get_service_top_operations` narrow it to the endpoint, and `inspect_trace` shows which span grew.

### Build dashboards and alerts

The agent can create dashboards (`create_dashboard`, `add_dashboard_widget`) and alert rules (`create_alert_rule`). A widget that saves is not always a widget that shows the right thing, so ask the agent to check its work with `inspect_chart_data`, which runs the query and reports whether the chart has data.

For heavier dashboard work, give your agent the [maple-dashboard-widgets](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-dashboard-widgets) skill. It documents every panel type, data source and unit.

### Answer anything else with SQL

When no tool fits, `describe_warehouse_tables` lists the tables and columns, and `run_sql` runs a read-only query scoped to your organization. See [SQL](/docs/reference/sql).

## Instrument with a coding agent

Open-source skills teach a coding agent how to set up OpenTelemetry for Maple:

- [maple-onboard](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-onboard) instruments every app and service in a repository: traces, logs and metrics, using the native OpenTelemetry SDK for each language.
- [maple-audit](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-audit) reviews an existing setup, reports gaps per service (missing service map edges, missing `service.version`, errors without exceptions) and fixes them.
- [maple-agent-tracing](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing) traces an AI agent so each conversation shows up as one [Agent Session](/docs/agent-sessions/overview). It detects the framework and installs a skill for that framework only. See [Trace your AI agent](/docs/agent-tracing).

Install them together with the per-language guides they read:

```bash
bunx skills add MapleTechLabs/maple/skills --skill '*'
```

Then ask: _"Set up Maple in this repo. My ingest endpoint is https://ingest.maple.dev and my ingest key is maple_pk\_…"_ or _"Audit my Maple instrumentation."_ EU organizations give `https://ingest.eu.maple.dev`. With the MCP server connected, the agent can confirm the first traces arrived and run `audit_setup` and `get_instrumentation_recommendations` against your live data.

## Stay in control

- **Read and write tools are separate.** Most tools only read. Tools that change data (triage, dashboards, alert rules) are marked in the [tool list](/docs/reference/mcp#tools). Most clients ask before each tool call; keep approval on for write tools.
- **Deletes need confirmation.** `delete_alert_rule` refuses to run without `confirm=true`.
- **Access is scoped to one organization**, the one the key belongs to or the one you picked at sign-in.
- **Rate limit:** 120 requests per 10 seconds per key or user. Over it, the server answers `429` and the agent should wait 10 seconds.

## Prefer a terminal?

The [`maple` CLI](/docs/reference/cli) outputs JSON by default, so an agent without MCP support can run it and pipe the result into `jq` or its own context.

## Troubleshooting

- **`401 Unauthorized`.** The key was revoked, or the endpoint is in the wrong region. EU organizations use `api.eu.maple.dev`.
- **Empty results.** Check the time window and the service name. Ask the agent to run `list_services` first to see what Maple has.
- **Source code tools return nothing.** They need the [GitHub App](/docs/integrations/github) installed with access to that repository.
- **The agent guesses instead of querying.** Tell it to use Maple: _"Use the maple MCP tools to check."_

## Related

- [MCP server reference](/docs/reference/mcp): endpoints, authentication and every tool
- [Errors](/docs/errors/overview): issue states, leases and fix verification
- [Agent Sessions](/docs/agent-sessions/overview): observe the AI agents you build
- [Authentication](/docs/reference/authentication): API keys and OAuth

---

# OpenTelemetry conventions

> Concepts · https://maple.dev/docs/concepts/otel-conventions

Maple's expected OpenTelemetry attributes, status codes, span kinds, and data model conventions.

Maple accepts telemetry over the OpenTelemetry Protocol (OTLP). This page describes the attributes and conventions Maple reads to build its dashboards, service map and analytics.

Maple stores every OTel attribute you send verbatim. A curated set gets special treatment: pre-extracted into indexed columns at ingest, exposed as short filter aliases, rendered as colored badges, used to draw the service map, or ranked higher in the log attribute chips. Most of these follow the [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/). If your SDK emits standard attributes, you usually don't need to do anything extra.

> **Audit with Claude Code:** `maple-audit` reviews an existing setup against these conventions per service, with severities, and fixes the gaps. See the [maple-audit skill](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-audit).

## Sending data

This page covers what to put on your telemetry. Endpoints, authentication headers, content types and the standard `OTEL_*` environment variables are in the [Ingest API reference](/docs/reference/ingest). To set up an SDK, start at [Instrument your application](/docs/instrumentation).

## Service identity

The bare minimum every span needs. `service.name` is the primary axis Maple groups by. Without it, spans go to a synthetic `unknown_service` bucket.

| Attribute             | Example                | What Maple does with it                                                                                     |
| --------------------- | ---------------------- | ----------------------------------------------------------------------------------------------------------- |
| `service.name`        | `api`, `ingest`, `web` | **Required.** Primary grouping for services list, service map, dashboards, alerts. Filter alias: `service`. |
| `service.version`     | `1.4.2`, `c0b92f68`    | Per-version slices on service overview. Hidden from log chips but always queryable.                         |
| `service.namespace`   | `payments`             | Logical grouping above `service.name`. Hidden from log chips.                                               |
| `service.instance.id` | UUID per process       | Distinguishes replicas of the same service. Hidden from log chips.                                          |

## Deployment and version tracking

Tag every span with these and you get per-environment and per-version slices across the services table, service map, and per-service overview.

| Attribute                     | Example                       | What Maple does with it                                                                             |
| ----------------------------- | ----------------------------- | --------------------------------------------------------------------------------------------------- |
| `deployment.environment.name` | `production`                  | Filterable everywhere; per-env throughput / latency / error rate; environment chips in span detail. |
| `deployment.environment`      | `production`                  | Legacy alias, treated as the same value. Either spelling is accepted.                               |
| `vcs.ref.head.revision`       | `c0b92f68`                    | Git commit SHA. Enables release markers on charts and per-version metrics.                          |
| `vcs.repository.url.full`     | `https://github.com/acme/api` | Canonical repo URL. Links telemetry to source.                                                      |

`vcs.repository.url.full` and `vcs.ref.head.revision` are the OpenTelemetry semantic-convention keys. Use them exactly as named. Legacy spellings like `deployment.commit_sha`, `git.repo`, or `app.repo_url` are not read.

Set resource attributes via environment variable:

```bash
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production,vcs.repository.url.full=https://github.com/acme/api,vcs.ref.head.revision=abc123"
```

In the search bar, `env`, `environment`, and `commit_sha` are short aliases. See [Filter aliases](#filter-aliases) below.

## Span status codes

Maple stores span status codes as title-case strings. Maple's ingest converts the OTLP status enum on the way in:

| OTLP value | Stored as | Meaning                         |
| ---------- | --------- | ------------------------------- |
| `0`        | `"Unset"` | Default. No explicit status set |
| `1`        | `"Ok"`    | Explicitly marked successful    |
| `2`        | `"Error"` | Span encountered an error       |

Set status through your SDK's status API. You don't need to convert anything yourself.

Title case matters when you filter or write queries. `StatusCode = 'Error'` matches. Uppercase (`ERROR`) or lowercase (`error`) variants match zero rows.

Only spans with status `Error` appear in error analytics.

## Span kinds

| Kind         | Description                          | How Maple Uses It                                                                                         |
| ------------ | ------------------------------------ | --------------------------------------------------------------------------------------------------------- |
| `"Server"`   | Incoming request handler             | Throughput and error rate calculations. Callee side of a service map edge. Renders path-only HTTP routes. |
| `"Client"`   | Outgoing request to another service  | Caller side of a service map edge; database nodes; external dependencies. Renders host+path HTTP routes.  |
| `"Producer"` | Async message producer               | Caller side of a service map edge; messaging dependencies.                                                |
| `"Consumer"` | Async message consumer               | Throughput calculations. Callee side of a `Producer` edge.                                                |
| `"Internal"` | Default, synchronous in-process work | Trace detail view.                                                                                        |

`Client` spans get a small outgoing-arrow icon in HTTP labels and render their route as `host+path` so the destination is visible. `Server` spans render path-only. The [service map](#service-map) and a service's dependencies are built only from `Client` and `Producer` spans. A network call left on `Internal` does not appear on either.

## HTTP attributes

The most heavily instrumented namespace. Maple extracts three fields into indexed columns at write time, then renders method, route, and status code in trace rows.

### Fast columns

Filtering on these scans a small column instead of doing a per-row map lookup. The legacy and current OTel semconv names map to the same column. The first non-empty source wins, so you don't need to migrate just for fast filtering.

| Attribute(s)                                    | Indexed column   |
| ----------------------------------------------- | ---------------- |
| `http.method`, `http.request.method`            | `HttpMethod`     |
| `http.route`, `url.path`, `http.target`         | `HttpRoute`      |
| `http.status_code`, `http.response.status_code` | `HttpStatusCode` |

### Method color pills

`http.method` / `http.request.method` drives a colored pill on the span row:

| Method  | Color     |
| ------- | --------- |
| GET     | Blue      |
| POST    | Orange    |
| PUT     | Green     |
| PATCH   | Gray      |
| DELETE  | Red       |
| HEAD    | Gray      |
| OPTIONS | Dark gray |

### Status badge tiers

`http.status_code` / `http.response.status_code` is rendered as a colored badge in the trace list:

| Status range | Tone         |
| ------------ | ------------ |
| 5xx          | Error (red)  |
| 4xx          | Warn (amber) |
| 3xx          | Blue         |
| 1xx–2xx      | Info (green) |

In log chips, the same key scores 95, just below `exception.*`. See [Attribute prominence scoring](#appendix-attribute-prominence-scoring).

### Route extraction and fallback chain

For full HTTP info, Maple tries each source in order until one matches:

- **Method:** `http.method` → `http.request.method` → span name (e.g. `http.server GET /path`, or bare `GET /path`).
- **Route on server spans:** `http.route` → `http.target` → `url.path`.
- **Route on client spans:** `http.route` → parsed `url.full` / `http.url` (host+path) → `server.address` / `net.peer.name` combined with `url.path` / `http.target`.
- **Status:** `http.status_code` → `http.response.status_code`.

If none of the route attributes are set, Maple falls back to the route in the span name.

So `url.full` (e.g. `https://api.stripe.com/v1/charges`) on a `Client` span is enough for route rendering. Emitting `http.route` is still preferred, because it's a semantic path (`/api/users/:id`) instead of a high-cardinality URL.

## Service map

The [service map](/docs/explore/service-map) draws a node for each service and each database, and an edge for the calls between them. Calls to external HTTP hosts, message queues and RPC services are not drawn on the map. They are listed on the service's **Dependencies** tab. Four rules make sure your spans show up correctly.

### 1. Service-to-service edges

Maple draws an edge when a `Client` or `Producer` span in one service has a child `Server` or `Consumer` span in another service, in the same trace. Two things make that work:

- The callee is instrumented, so it records its own `Server` or `Consumer` span.
- The caller propagates trace context (the `traceparent` header), so the callee's span becomes a child of the client span.

Instrumented HTTP and RPC clients do both for you.

```text
api:    GET /v1/users  (span.kind=Client, span_id=a1)
users:  GET /v1/users  (span.kind=Server, parent_span_id=a1)
                            └──> draws an edge api → users
```

`peer.service` does not draw edges. If the callee is not instrumented, or the caller drops `traceparent`, no edge is drawn. The call shows up at most as an external dependency on the caller's **Dependencies** tab (see rule 3).

### 2. Database nodes

Set `db.system.name` and `db.namespace` on database `Client` spans. The legacy `db.system` spelling is also accepted. Maple keys each database node on the pair, so services that call the same database share one node.

```text
SELECT * FROM users  (span.kind=Client, db.system.name=postgresql, db.namespace=users_db)
                            └──> draws an edge api → postgresql users_db
```

Without `db.system.name`, the call is treated as an external HTTP or other dependency, not a database. Without `db.namespace`, Maple falls back to the legacy `db.name`, then to the host (`server.address`). With none of these set, every database behind that driver collapses into one node.

### 3. External dependencies

Other outbound `Client` and `Producer` spans become external dependencies. They are listed on the calling service's **Dependencies** tab, not drawn on the map. Maple names each one from these attributes:

| Call type   | Attributes                                       | Dependency name                            |
| ----------- | ------------------------------------------------ | ------------------------------------------ |
| Messaging   | `messaging.system`, `messaging.destination.name` | Destination name, or the system if missing |
| RPC         | `rpc.system`, `rpc.service`                      | `rpc.service`, or the system if missing    |
| HTTP, other | `server.address`                                 | Host                                       |

HTTP client instrumentation sets `server.address` automatically.

### 4. Pick canonical names

Keep `db.system.name`, `messaging.system`, and `rpc.system` values spelled the same across services. If one service emits `db.system.name=postgresql` and another emits `PostgreSQL`, they become separate database nodes. Use the OpenTelemetry well-known values where one exists.

## Database queries

Besides the service map, these drive the log chips and the AI error-debug prompt context.

| Attribute                                   | What Maple does with it                                                                                     |
| ------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| `db.system.name` (legacy `db.system`)       | Scored 70 in log chips; with `db.namespace`, keys the service map database node; toned `info` in log chips. |
| `db.namespace`                              | Names the database node on the service map.                                                                 |
| `db.query.text` (legacy `db.statement`)     | Scored 70; rendered in span detail; included as context in the error-debug prompt.                          |
| `db.operation.name` (legacy `db.operation`) | Scored 70 (e.g. `"SELECT"`, `"INSERT"`).                                                                    |

## Caching

Maple detects a cache span when `cache.system` or `cache.result` is present. When detected, the trace UI renders a hit/miss badge and an operation pill (GET / SET / DELETE) on the span row.

| Attribute                | Example                | What Maple does with it                                                       |
| ------------------------ | ---------------------- | ----------------------------------------------------------------------------- |
| `cache.system`           | `redis`, `memcached`   | Identifies the cache backend; presence triggers cache-span detection.         |
| `cache.result`           | `hit` \| `miss`        | Drives the hit/miss badge color; presence also triggers cache-span detection. |
| `cache.name`             | `user-sessions`        | Logical cache name shown in span detail.                                      |
| `cache.operation`        | `GET`, `SET`, `DELETE` | Drives the operation pill color.                                              |
| `cache.lookup_performed` | `true` \| `false`      | Whether a lookup was actually executed (string, not bool).                    |

## Errors and exceptions

Drives the error banner in the log detail panel and the highest-priority chip on every log row.

| Attribute           | What Maple does with it                                              |
| ------------------- | -------------------------------------------------------------------- |
| `exception.message` | Banner body. Falls back to `error.message`, then to the log body.    |
| `exception.type`    | Monospace badge beside the banner title. Falls back to `error.type`. |
| `error.message`     | Same as `exception.message` (legacy fallback).                       |
| `error.type`        | Same as `exception.type` (legacy fallback).                          |

Any attribute matching `exception.*` scores 100 (top of the log chips) and is toned `error` (red).

If the message runs past 3 lines or 160 characters, the banner collapses by default and shows a "Show more" toggle.

## RPC

For gRPC and other RPC frameworks.

| Attribute              | What Maple does with it                                                      |
| ---------------------- | ---------------------------------------------------------------------------- |
| `rpc.system`           | Names RPC dependencies when `rpc.service` is missing.                        |
| `rpc.service`          | Scored 68 in log chips; toned `info`. Names RPC dependencies.                |
| `rpc.method`           | Scored 68; toned `info`.                                                     |
| `rpc.grpc.status_code` | Scored 90 (just below HTTP status). Non-zero values are toned `error` (red). |

## User identity

Promotes user/customer context to the log chips so it's visible at a glance on every log row.

| Attribute                                             | What Maple does with it                      |
| ----------------------------------------------------- | -------------------------------------------- |
| `user.id`, `enduser.id`, `customer.id`, `customer_id` | All scored 66 in log chips (equal priority). |

## Logs

### Severity levels

`SeverityText` drives the per-row text color in the log list and the trace detail timeline.

| SeverityText | SeverityNumber | Color theme    |
| ------------ | -------------- | -------------- |
| `TRACE`      | 1-4            | severity-trace |
| `DEBUG`      | 5-8            | severity-debug |
| `INFO`       | 9-12           | severity-info  |
| `WARN`       | 13-16          | severity-warn  |
| `ERROR`      | 17-20          | severity-error |
| `FATAL`      | 21-24          | severity-fatal |

`ERROR` and `FATAL` severities also show the error banner at the top of the log detail panel.

### Trace correlation

Logs are automatically correlated with traces when `TraceId` and `SpanId` fields are present. Most OTel SDKs inject these fields when a span is active.

## Kubernetes and infrastructure

Kubernetes resource attributes power the service map's pod-count badges and the Infrastructure pages. The `maple-k8s-infra` Helm chart sets most of these for you via the OTel operator and the `k8sattributes` processor.

### Workload identity (joins to `service.name`)

| Attribute              | What Maple does with it                                                                                |
| ---------------------- | ------------------------------------------------------------------------------------------------------ |
| `k8s.deployment.name`  | Primary workload identity. With `k8s.namespace.name`, joins to `service.name` for infrastructure data. |
| `k8s.statefulset.name` | Same join, for stateful workloads.                                                                     |
| `k8s.daemonset.name`   | Same join, for DaemonSets.                                                                             |
| `k8s.job.name`         | Job filter and pod detail on the Infrastructure pages. Not part of the service join.                   |
| `k8s.cluster.name`     | Cluster shown on the Infrastructure pages. Not part of the service join.                               |

### Promoted to log chips

Resource attributes normally stay out of log chips. These are the exceptions, along with `deployment.environment`.

| Attribute            | What Maple does with it          |
| -------------------- | -------------------------------- |
| `k8s.pod.name`       | Promoted to log attribute chips. |
| `k8s.namespace.name` | Promoted to log attribute chips. |
| `cloud.region`       | Promoted to log attribute chips. |

### Node detail metadata

| Attribute             | What Maple does with it                                                       |
| --------------------- | ----------------------------------------------------------------------------- |
| `k8s.node.name`       | Required to match node metrics from kubelet; node-list and node-detail views. |
| `k8s.node.uid`        | Display in node metadata panel.                                               |
| `k8s.pod.uid`         | Used to count distinct pods per workload.                                     |
| `k8s.kubelet.version` | Display in node metadata panel.                                               |
| `container.runtime`   | Display in K8s node metadata (containerd, cri-o, etc.).                       |

## Cloud and platform badges

These set the platform badge and runtime icon next to a service on the service map. SDKs running on common platforms detect most of them automatically. The keys are listed here so self-instrumenters can match.

| Attribute              | Example values                                    | What Maple does with it                                                                                                  |
| ---------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| `cloud.provider`       | `cloudflare`                                      | `cloudflare` sets the Cloudflare badge.                                                                                  |
| `cloud.platform`       | `cloudflare.workers`, `aws_lambda`                | `cloudflare.workers` sets the Cloudflare badge; `aws_lambda` sets the Lambda badge.                                      |
| `cloud.region`         | `us-west-2`, `iad1`                               | Promoted to log chips (see Kubernetes section above).                                                                    |
| `process.runtime.name` | `nodejs`, `bun`, `deno`, `workerd`, `rust`, `jvm` | Runtime mark on the service map.                                                                                         |
| `faas.name`            | Lambda function name, Worker script name          | Any value sets the Lambda badge unless the service is on Cloudflare. Matches Cloudflare Worker scripts to their service. |
| `faas.version`         | Function version / revision                       | Shown as the script version on Cloudflare Workers.                                                                       |

A service with `k8s.pod.name` or `k8s.deployment.name` gets the Kubernetes badge. Cloudflare takes precedence over Lambda, and Lambda over Kubernetes.

`nodejs`, `bun`, `deno`, `workerd`, `rust`, `python` (or `cpython`), `ruby`, and the JVM (`jvm`, `java`, or the OTel-canonical `OpenJDK Runtime Environment`) render as their logo next to the service name. `go`, `dotnet`, `php`, and anything unrecognized render as a short text chip, because their logos are wordmarks that are unreadable at icon size. A runtime the platform badge already implies (`workerd` on a Cloudflare service) is not shown a second time.

Common aliases are folded together (`node`/`nodejs`, `go`/`golang`). Keep the value consistent across services on the same runtime anyway. An unlisted variant falls through to the text chip, and one fleet ends up wearing two different marks.

## Filter aliases

In Maple's WHERE-clause search bar (trace list, log search, dashboard widgets), you can type a short alias and it resolves to the canonical attribute:

| Alias                | Resolves to                                       |
| -------------------- | ------------------------------------------------- |
| `service`            | `service.name`                                    |
| `span`               | `span.name`                                       |
| `environment`, `env` | `deployment.environment`                          |
| `commit_sha`         | `vcs.ref.head.revision`                           |
| `root.only`          | `root_only` (synthetic boolean, root spans only)  |
| `errors_only`        | `has_error` (synthetic boolean, error spans only) |

So `env = "production"` and `deployment.environment = "production"` mean the same thing. Pick whichever is shorter.

## Reserved namespace

`maple_*` is reserved for Maple platform internals (org routing, ingest auth keys). Do not use this prefix for your own attributes. The UI hides anything starting with `maple_` from log attribute chips.

## Attributes Maple hides from log chips

These are stored on the row but skipped from the log attribute chips because they're noisy or already shown elsewhere (service column, etc.):

- `service.name`, `service.namespace`, `service.instance.id`, `service.version`
- `telemetry.sdk.*`
- `process.runtime.*`, `process.executable.*`
- `os.*`
- `host.arch`, `host.name`
- `maple_*`

The data is still queryable. You can filter or group by these in the search bar. They just don't appear in the row's attribute chips.

## Appendix: attribute prominence scoring

Each log row shows every attribute that isn't hidden as a chip, ordered by score. Higher score comes first.

| Score | Attributes                                                                                          |
| ----- | --------------------------------------------------------------------------------------------------- |
| 100   | `error`, `exception`, `exception.*` (anything)                                                      |
| 95    | `http.status_code`, `http.response.status_code`                                                     |
| 90    | `rpc.grpc.status_code`                                                                              |
| 80    | `http.method`, `http.request.method`                                                                |
| 70    | `db.system.name`, `db.system`, `db.query.text`, `db.statement`, `db.operation.name`, `db.operation` |
| 68    | `rpc.service`, `rpc.method`                                                                         |
| 66    | `user.id`, `enduser.id`, `customer.id`, `customer_id`                                               |
| 60    | `duration_ms`, `latency_ms`, `http.duration`                                                        |
| 55    | `http.url`, `http.route`, `url.path`                                                                |
| 40    | Other `http.*`, `url.*`                                                                             |
| 38    | Other `db.*`                                                                                        |
| 36    | Other `rpc.*`                                                                                       |
| 34    | `messaging.*`                                                                                       |
| 32    | Other `user.*`, `enduser.*`                                                                         |
| 25    | Anything with a dot (`namespace.key`)                                                               |
| 20    | Bare keys (no namespace)                                                                            |

Resource attributes appear in chips only if they're in the promoted set (`deployment.environment`, `deployment.environment.name`, `k8s.pod.name`, `k8s.namespace.name`, `cloud.region`). A promoted resource attribute scores 10 lower than the same key on the log itself.

## Metrics

Maple accepts OTLP metrics at `/v1/metrics`: sums (counters), gauges, histograms, and exponential histograms. Summary data points are not supported and are dropped at ingest.

For exact RED (rate, error, duration) metrics alongside sampled traces, derive metrics from every span with the Collector's SpanMetrics connector before sampling. See [Exact counts with SpanMetrics](/docs/concepts/sampling-throughput#exact-counts-with-spanmetrics).

## Data retention

How long each signal is kept depends on your plan. See [Retention](/docs/reference/retention).

---

# Sampling and throughput estimation

> Concepts · https://maple.dev/docs/concepts/sampling-throughput

How Maple weights sampled spans at ingest so throughput, error rate and service map call counts reflect the traffic you actually served.

When you sample traces, the spans Maple receives are a fraction of the requests your services handled. Maple gives every span a weight at ingest and sums those weights, so request rates, error rates and service map call counts estimate the traffic before sampling.

## What is trace sampling?

Sampling reduces span volume by exporting only a subset of traces.

- **Head sampling** decides at the start of a trace, for example "keep 10% of traces". The OpenTelemetry SDK or Collector makes the choice before any spans are exported.
- **Tail sampling** decides after the trace completes, usually based on latency, errors or other attributes.

Both cut storage and processing cost. Both also mean the raw span count no longer equals the number of requests.

## How each span gets its weight

At ingest, Maple stores a `SampleRate` on every span: the number of requests that span stands for. It resolves the value from three sources, in this order:

1. **`SampleRate` span attribute.** If the span carries a `SampleRate` attribute of 1 or more (for example, set by your Collector), Maple uses it as is.
2. **W3C threshold in `tracestate`.** When OpenTelemetry probability sampling is active, the span's `tracestate` carries an `ot` entry with a `th` (rejection threshold) key, such as `ot=th:e668`. Maple converts the threshold to a weight of `1 / acceptance probability`. The weight is capped at 10000 (an acceptance probability of 0.01%).
3. **Default of 1.** A span with neither is treated as unsampled.

No configuration is needed. If your SDK or Collector sets `th` in `tracestate`, or sets a `SampleRate` attribute, Maple picks it up.

For example, `th:e668` means about 90% of traces were rejected, so about 10% were kept and each kept span has a `SampleRate` of about 10.

## How throughput is calculated

Throughput is the sum of the weights, divided by the length of the window:

```
estimated requests = sum(SampleRate)
throughput         = estimated requests / window seconds
```

With 10% sampling (`SampleRate = 10`) and 500 sampled entry-point spans over 60 seconds, the estimate is 5,000 requests, or about 83 requests per second.

Because the weight is stored on each span, **mixed sampling rates add up correctly**. Take 100 spans in one bucket: 99 kept at 50% (weight 2) and 1 kept at 1/8192 (weight 8192). The estimate is `99 × 2 + 1 × 8192 = 8390` requests. The same holds when different services, or different deployments of one service, sample at different rates.

## How error rate is calculated

Error rate uses the same weights. Each error span counts for its `SampleRate`, and so does each span in the denominator:

```
error rate = sumIf(SampleRate, StatusCode = 'Error') / sum(SampleRate)
```

A tail sampler that keeps every error but only 1% of successful requests would make a raw error count look far too high. Weighting each span by its own `SampleRate` corrects for that, as long as the sampler reports the weight it used.

Service map edges use the same weighting for call counts between services.

## UI indicators

When a service has sampled spans in the selected window:

- **Tilde prefix (`~`).** Throughput is prefixed with `~` to mark it as an estimate. This appears in the services table and on service map nodes and edges.
- **Traced rate.** The services table shows the rate of spans actually received under the estimate, for example `~8.3 traced`.
- **Tooltip.** Hovering the throughput cell shows the sampling rate and the extrapolation factor, for example "Estimated from 10% sampled traces (x10 extrapolation)".

## Limitations

**The estimate is only as good as the weight.** If a sampler drops spans without recording `th` in `tracestate` or a `SampleRate` attribute, Maple counts the surviving spans at weight 1 and underestimates throughput.

## Exact counts with SpanMetrics

For exact RED metrics (rate, errors, duration) next to sampled traces, use the OpenTelemetry Collector [SpanMetrics Connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/spanmetricsconnector). It derives metrics from every span before sampling, so the counts do not depend on the sampling configuration.

A typical Collector pipeline:

```yaml
connectors:
    spanmetrics:
        namespace: span.metrics

service:
    pipelines:
        traces:
            receivers: [otlp]
            processors: [batch]
            exporters: [otlp, spanmetrics] # fork to both export + metrics
        metrics:
            receivers: [spanmetrics]
            exporters: [otlp]
```

The metrics pipeline sees every request while the traces pipeline samples as aggressively as you like.

---

# Traces without a root span

> Concepts · https://maple.dev/docs/concepts/missing-root-spans

Why traces reach Maple without their root span, how Setup health measures it, and how to fix each cause.

The root span is the one span in a trace with no parent. When it never reaches Maple, the rest of the trace still arrives, but:

- the trace is missing from the trace list, its filters and trace search, which list traces by their root span;
- alert rules on error rate, latency and throughput don't count it, because they are computed over root spans;
- its duration and its waterfall start at the first span that did arrive.

## How Setup health measures it

**Settings → Setup health** runs the check **Traces have a root span** (`TRC-02`). It takes every trace whose first span started between 75 and 15 minutes ago and counts the ones whose root span never arrived. A span is exported when it ends, and the root usually ends last, so the 15 minutes give it time to arrive.

| Traces without a root span | Result   |
| -------------------------- | -------- |
| None or one                | Passes   |
| Two or more, up to 0.1%    | Info     |
| More than 0.1%             | Warning  |
| More than 1%               | Critical |

Warning and critical need at least 50 traces in the window. Traces whose root was dropped by sampling don't count: those with a `th:` value in `tracestate`, and, when Maple's ingest sampling is on, traces made only of the error and slow spans it keeps from traces it drops.

The finding lists the services where these traces start, each with its own share. The `audit_setup` [MCP tool](/docs/reference/mcp) returns the same result.

## Find the cause

The earliest span that did arrive points at the cause. The [agent prompt](#investigate-with-an-agent) below finds example traces and reads them for you.

| Earliest span that arrived                                           | Cause                                                                                 | Fix                                                               |
| -------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
| An Internal or Client span in a short-lived process                  | The root span was never exported                                                      | [Flush before the process stops](#flush-before-the-process-stops) |
| A Server span with a parent ID                                       | A proxy, gateway, load balancer or platform in front sends `traceparent` but no spans | [Context from upstream](#context-from-upstream)                   |
| Any span, in a trace that runs for many minutes                      | The root span is still open                                                           | [Long-running root spans](#long-running-root-spans)               |
| A span whose parent belongs to a service that exports somewhere else | The caller sends its spans to another Maple organization or another backend           | Send every service to the same organization                       |

## Flush before the process stops

The OpenTelemetry SDK exports spans in batches. Whatever is still in the batch when the process exits or the runtime freezes it is lost, and the root span, which ends last, is usually among them.

- In serverless functions, flush the tracer provider at the end of every invocation with `await provider.forceFlush()`.
- In CLI commands, scripts and jobs, call `await sdk.shutdown()` before the process exits, on the error path too.
- In servers, call `sdk.shutdown()` on `SIGTERM`, so a deploy or scale-down doesn't drop the roots of requests in flight.

The [instrumentation guides](/docs/instrumentation) show the setup for each language.

## Context from upstream

When a proxy, gateway or load balancer adds or forwards `traceparent` without exporting a span of its own, your first service continues a trace whose root lives in that proxy. Then nearly every trace that passes through it has no root.

- If the proxy can export OpenTelemetry, send its spans to Maple, and it becomes the root.
- Otherwise, remove the incoming `traceparent` and `tracestate` headers at the proxy, so the first service starts the trace itself.

Cloud Run rewrites `traceparent` on every request it forwards: the parent becomes a span inside Google's infrastructure that Maple never receives, so traces that enter a Cloud Run service have no root. [Instrument a Cloud Run service](/docs/integrations/gcp-opentelemetry#instrument-a-cloud-run-service) covers the setup and this limit.

## Long-running root spans

A root span that stays open for more than about 15 minutes, such as a WebSocket session or a batch job, counts as missing until it ends. Start a new trace for each message or step, and connect it to the session with a span link.

## Investigate with an agent

Connect the [Maple MCP server](/docs/getting-started/ai-agents) to a coding agent such as Claude Code, Codex or Cursor, and paste this prompt:

```text
Investigate why traces in my Maple organization arrive without a root span, using the Maple MCP tools. Guide: https://maple.dev/docs/concepts/missing-root-spans

1. Call audit_setup and read the TRC-02 finding: the share of traces without a root span and the services where they start.

2. Find example traces with run_sql, over a 30-minute window that ends 15 minutes ago:

SELECT ServiceName AS firstService, SpanKind AS firstSpanKind, SpanName AS firstSpan,
       count() AS traces, any(TraceId) AS exampleTrace
FROM (
  SELECT TraceId,
         argMin(ServiceName, Timestamp) AS ServiceName,
         argMin(SpanKind, Timestamp) AS SpanKind,
         argMin(SpanName, Timestamp) AS SpanName
  FROM traces
  WHERE $__orgFilter AND $__timeFilter(Timestamp)
  GROUP BY TraceId
  HAVING countIf(ParentSpanId = '') = 0
     AND min(Timestamp) >= $__startTime + INTERVAL 10 MINUTE
)
GROUP BY firstService, firstSpanKind, firstSpan
ORDER BY traces DESC
LIMIT 20

3. For the top rows, call inspect_trace on exampleTrace and inspect_span on its earliest span. Note the span kind, the parent span ID, the resource attributes (cloud.platform, faas.*, telemetry.sdk.*) and how long the trace runs.

4. Classify each row:
   - Internal or Client earliest span in a short-lived process: the root span is never exported, because the process exits or is frozen before the SDK flushes.
   - Server earliest span with a parent ID: a proxy, gateway, load balancer or platform such as Cloud Run sends traceparent without exporting spans.
   - The trace runs for many minutes: the root span is still open.
   - The parent belongs to a service that exports to another Maple organization or another backend.

5. If the source code is available, locally or through search_source_code and read_source_file, find where each affected service sets up OpenTelemetry. Check that it flushes at the end of each request, invocation or command, and shuts the SDK down on SIGTERM.

6. Report per service: the cause, the trace IDs that show it, and the code or configuration change that fixes it. Don't change any code until I confirm.
```

---

# Traces

> Explore · https://maple.dev/docs/explore/traces

Search traces by service, route, status, duration, and any span or resource attribute, then open a trace to inspect its spans, logs, and session replay.

The **Traces** page lists the traces Maple received in the selected time range. Filter the list with sidebar facets or a query, then open a trace to see every span on a waterfall, timeline, or flow view. From a trace you can jump to its logs and to the session replay that produced it.

## What data it needs

Traces come from any OpenTelemetry SDK exporting spans over OTLP. The list and facets read these fields:

- `service.name` resource attribute for **Service**, and `deployment.environment` and `service.namespace` for **Environment** and **Namespace**.
- Span name for **Root Span**.
- Span status (`Ok`, `Error`, `Unset`) for **Has Error** and the **Status** column.
- `http.request.method` and `http.response.status_code` (or their older `http.method` and `http.status_code` spellings) for **HTTP Method** and **Status Code**.

See [OpenTelemetry conventions](/docs/concepts/otel-conventions) for the full list of attributes Maple reads.

## The trace list

The table has these columns: **Trace ID**, **Root Span** (with start time), **Services**, **Spans**, **Duration**, and **Status**. **Status** shows the HTTP status code when the root span has one, otherwise the span status. Sort by clicking **Duration**. The default order is newest first. More rows load as you scroll.

<figure class="shot">
  <img src="/screenshots/docs/traces-01-list.webp" alt="The Traces list with Trace ID, Root Span, Services, Spans, Duration and Status columns, and a filter sidebar with Has Error, Root Traces Only, environment, namespace, service and root span facets." loading="lazy" />
  <figcaption>The trace list with its facet sidebar. Counts next to each facet value cover the selected time range.</figcaption>
</figure>

The default time range is the last 12 hours. Pick a preset from **Last 5 minutes** to **Last 1 month**, or set a custom range.

Click a row to open the trace in a side sheet. Use **Previous trace** and **Next trace** (or the `K` and `J` keys) to step through the list, and **Open trace** to go to the full page. Cmd-click or Ctrl-click opens the full page directly.

## Filter traces

### Sidebar facets

| Facet                      | What it does                                                                                                                                       |
| -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Has Error**              | Only traces with an error span.                                                                                                                    |
| **Root Traces Only**       | Lists one row per trace, keyed on its entry span (a root span, or a Server or Consumer span). On by default. Turn it off to list individual spans. |
| **Hide Single-Span Noise** | Hides single-span traces that are not entry points, such as orphaned client spans. On by default, shown while **Root Traces Only** is on.          |
| **Environment**            | Filter by `deployment.environment`.                                                                                                                |
| **Namespace**              | Filter by `service.namespace`.                                                                                                                     |
| **Service**                | Filter by `service.name`.                                                                                                                          |
| **Root Span**              | Filter by root span name.                                                                                                                          |
| **Duration**               | A minimum and maximum duration range.                                                                                                              |
| **HTTP Method**            | Filter by request method.                                                                                                                          |
| **Status Code**            | Filter by HTTP response status code.                                                                                                               |

Every value facet can include or exclude a value.

### Query syntax

Click **Advanced Filter** (or press `F`) to write a query. Press Ctrl+Space for autocomplete and Cmd+Enter to apply.

```
service.name = "checkout" AND attr.http.route != "/health"
```

Join clauses with `AND`. Supported keys and operators:

| Key                                                                        | Operators                                                                     |
| -------------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
| `service.name`, `span.name`, `deployment.environment`, `service.namespace` | `=`, `!=`, `contains`                                                         |
| `http.method`, `http.status_code`                                          | `=`, `!=`                                                                     |
| `has_error`, `root_only`, `min_duration_ms`, `max_duration_ms`             | `=`                                                                           |
| `attr.<key>` (span attribute), `resource.<key>` (resource attribute)       | `=`, `!=`, `contains`, `!contains`, `exists`, `!exists`, `>`, `>=`, `<`, `<=` |

You can use up to 5 filters per attribute map (span or resource). The dialog shows a warning for any clause the list cannot apply.

## Inspect a trace

The trace page header shows the environment, the commit SHA (from `vcs.ref.head.revision`), the error state, and the HTTP status. If the trace produced [product events](/docs/product-events/overview), they appear in a **Product events** panel.

Three views show the spans:

- **Timeline** (default): spans on a time axis, with **Search spans…** and a color-by picker.
- **Waterfall**: the span tree with a duration bar per span.
- **Flow**: spans as a graph of nodes and edges, with repeated child spans combined into one card with a count.

<figure class="shot">
  <img src="/screenshots/docs/traces-02-timeline.webp" alt="A 148-span trace for GET /v2/widget_summary on the Timeline view: a duration header, a service breakdown bar, and nested spans for API key lookups, warehouse queries and SQL statements drawn on a time axis." loading="lazy" />
  <figcaption>A trace on the Timeline view, with nested database and warehouse spans.</figcaption>
</figure>

Click a span to open the span panel. Press Esc to close it. The selected span is kept in the URL, so you can share a link to it.

The span panel shows the duration and status, and an error section when the status is `Error`. It has these tabs:

- **Details**: **Start Time**, **Span ID**, **Parent Span ID**, then **Span Attributes** and **Resource Attributes**. Each attribute row has **Filter by** and **Exclude** actions that open the trace list with that attribute filter applied.
- **Logs**: logs emitted inside this span. Click one to open its detail.
- **Infrastructure**: host, container, or Kubernetes details, shown when the span carries those resource attributes.

## Jump to logs and replays

- **View Logs** in the trace header opens the [Logs](/docs/explore/logs) page scoped to the trace ID. The button shows a count and appears only when the trace has logs. The time window covers the trace duration plus 5 minutes on either side.
- **View Session Replay** opens the [replay](/docs/session-replay/replays) of the browser session that made the request. It appears when a recorded session is linked to this trace ID by the [browser SDK](/docs/session-replay/browser-sdk).

## Query traces from an assistant

The [MCP server](/docs/reference/mcp) exposes the same data:

- `search_traces`: filter by service, error, duration, HTTP method, span name, or one attribute key and value.
- `inspect_trace`: the span tree and logs for one trace ID.
- `inspect_span`: every attribute of one span.
- `find_slow_traces`: the slowest traces with p50 and p95 context.

## Troubleshooting

- **"No traces yet".** Maple has never received a span for this organization. Configure an OpenTelemetry SDK with your ingest key and the [ingest endpoint](/docs/reference/ingest). See [Instrumentation](/docs/instrumentation).
- **"No traces in this time range".** Maple has data, just not in this window. Click **Widen time range**.
- **"No traces match these filters".** Your exclusions removed everything. Click **Clear filters**.
- **A trace you expect is missing.** **Root Traces Only** and **Hide Single-Span Noise** are on by default. Turn off **Hide Single-Span Noise** to see single-span traces, or **Root Traces Only** to search individual spans.
- **"Trace not found".** The trace has expired past your [retention](/docs/reference/retention) or has not been ingested yet.

## Next steps

- [Logs](/docs/explore/logs): search log lines and jump back to their traces.
- [Services](/docs/explore/services): latency, throughput, and errors per service.
- [OpenTelemetry conventions](/docs/concepts/otel-conventions): attributes that power filters and badges.

---

# Logs

> Explore · https://maple.dev/docs/explore/logs

Search log messages by text or trace ID, filter by severity, service, and environment, and move between a log line and the trace it belongs to.

The **Logs** page shows the log records Maple received in the selected time range. A volume chart stacks log counts by severity. Below it, a table lists each record. Search by message text or trace ID, filter with the sidebar, and open any record to see its attributes and the trace it came from.

## What data it needs

Logs arrive over OTLP, usually from an OpenTelemetry log bridge attached to your existing logger. Writing to stdout alone does not send anything to Maple. The page reads:

- `service.name`, `deployment.environment`, and `service.namespace` resource attributes for the **Service**, **Environment**, and **Namespace** filters.
- Severity text (for example `INFO`, `WARN`, `ERROR`) for **Severity** and the chart colors.
- Trace ID and span ID on the log record for trace correlation. OpenTelemetry log bridges set these automatically when a log is written inside an active span.

See [OpenTelemetry conventions](/docs/concepts/otel-conventions#logs) for severity levels and trace correlation.

## The log list

The default time range is the last 12 hours. The chart at the top shows log volume by severity. Drag across it to zoom the time range to that window.

<figure class="shot">
  <img src="/screenshots/docs/logs-01-explorer.webp" alt="The Logs page filtered to the maple-chat service: a log volume chart by severity, a Service is maple-chat filter chip, and rows with time, service and message, next to a sidebar of severity, environment, namespace and service filters." loading="lazy" />
  <figcaption>Logs filtered to one service. Drag across the volume chart to zoom into a window.</figcaption>
</figure>

The table has **Time**, **Service**, and **Message** columns. The toolbar above it has three controls:

- **Columns**: pin attributes as extra columns. The picker lists attribute keys seen in the selected time range. Pinned columns are saved in the URL.
- **Wrap long log lines**.
- **Toggle comfortable row density**.

Click a row to expand it inline. Click **Open detail** to open the full record.

## Search and filter

### Search box

The **Search** box accepts three kinds of input:

| Input                            | What it does                                             |
| -------------------------------- | -------------------------------------------------------- |
| Any text                         | Matches log messages containing the text, ignoring case. |
| A 32-character hex trace ID      | Shows only logs from that trace.                         |
| A W3C `traceparent` header value | Shows only logs from the trace it names.                 |

A trace ID becomes a removable chip. To search for a trace ID as plain text in messages, wrap it in double quotes. There is no `key:value` syntax. Use the sidebar to filter on fields.

### Sidebar filters

| Filter          | What it does                                                                                                    |
| --------------- | --------------------------------------------------------------------------------------------------------------- |
| **Severity**    | Filter by severity. Values come from your data, such as `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, and `FATAL`. |
| **Environment** | Filter by `deployment.environment`.                                                                             |
| **Namespace**   | Filter by `service.namespace`.                                                                                  |
| **Service**     | Filter by `service.name`.                                                                                       |

Each filter can include or exclude a value.

## Log details

The detail sheet opens with the message, a meta strip, and an **Error** or **Fatal** banner for those severities. It has these tabs:

- **Attributes**: **Log Attributes** and **Resource Attributes**, with a **Search attributes...** box.
- **Trace**: a **Trace Timeline** of every log in the same trace (up to 200), with span boundaries marked. Shown when the log has a trace ID.
- **Raw**: the record as a **JSON Payload**.
- **Infrastructure**: host, container, or Kubernetes details, shown when the log carries those resource attributes.

The meta strip shows the timestamp, a `trace:` link, and the span ID. It also has **Open in full page**, **Shareable link**, and **Log JSON** (copy) actions.

## Move between logs and traces

- **From a log to its trace**: click the `trace:` link in the meta strip. The trace view opens at the log's timestamp.
- **From a trace to its logs**: on a trace, click **View Logs**, or open a span's **Logs** tab. See [Traces](/docs/explore/traces#jump-to-logs-and-replays).

## Query logs from an assistant

The [MCP server](/docs/reference/mcp) exposes two log tools:

- `search_logs`: filter by service, severity, message text, trace ID, or span ID.
- `mine_log_patterns`: groups recent matching log messages into templates such as `GET /api/users/<*> 200 in <*>ms`, with a count and a service and severity breakdown per template. It samples up to 50,000 recent logs (10,000 by default). Pattern mining is available through MCP only; the Logs page has no patterns view.

## Troubleshooting

- **"No logs yet".** Maple has never received a log record for this organization. Add an OpenTelemetry log bridge to your logger and export to the [ingest endpoint](/docs/reference/ingest). See [Instrumentation](/docs/instrumentation).
- **"No logs in this time range".** Maple has logs, just not in this window. Widen the time range.
- **"No logs match these filters".** Your exclusions removed everything. Clear the filters.
- **Logs have no `trace:` link.** The record has no trace ID. Emit the log inside an active span, with the OpenTelemetry log bridge installed, so the SDK attaches the IDs.
- **A trace-scoped search returns nothing.** The trace's logs may fall outside the selected time range. Widen it, or open the logs from the trace page, which sets the window for you.

## Next steps

- [Traces](/docs/explore/traces): inspect the requests your logs belong to.
- [Errors and issues](/docs/errors/overview): how error spans become issues.
- [Alert rules](/docs/alerting/alert-rules): alert on a query over your telemetry.

---

# Metrics

> Explore · https://maple.dev/docs/explore/metrics

Browse every OpenTelemetry metric Maple has received, then chart one with an aggregation, a filter, and a breakdown by attribute.

The **Metrics** page lists every metric Maple received in the selected time range, with its type, emitting service, and data point count. Open a metric to chart it: choose an aggregation, filter by attribute, break it down by a dimension, and add the chart to a dashboard or turn it into an alert.

## What data it needs

Metrics arrive over OTLP from an OpenTelemetry SDK or collector, or from a [Prometheus scrape target](/docs/integrations/prometheus). Maple stores four OTLP metric types:

| Type                  | Badge         | Examples                                        |
| --------------------- | ------------- | ----------------------------------------------- |
| Sum                   | **Sum**       | Request counters, bytes sent. Monotonic or not. |
| Gauge                 | **Gauge**     | Memory in use, queue depth.                     |
| Histogram             | **Histogram** | Request duration with explicit buckets.         |
| Exponential histogram | **Exp Hist**  | Request duration with exponential buckets.      |

OTLP Summary metrics are not stored. The Prometheus scraper converts summaries into sums and a quantile gauge before ingest.

The **Service** column and the service filter read the `service.name` resource attribute. Data point attributes become the dimensions you can filter and group by.

## Browse metrics

The default time range is the last 24 hours.

Four cards at the top count metrics by type: **Sum Metrics**, **Gauge Metrics**, **Histogram**, and **Exp Histogram**. Each shows its data point count and number of unique metrics. Click a card to filter the list to that type. Click it again to clear the filter.

<figure class="shot">
  <img src="/screenshots/docs/metrics-01-catalog.webp" alt="The Metrics page: four cards counting sum, gauge, histogram and exponential histogram metrics, a search box, and a grid of sparkline cards, one per metric, each labelled with its type and the services that emit it." loading="lazy" />
  <figcaption>The metric catalog in grid view. Click a type card to filter the list.</figcaption>
</figure>

Type in **Search metrics...** to filter by metric name. Switch between **Grid view** (a sparkline card per metric, the default) and **Table view**. The table, headed **Available Metrics**, has these columns: **Metric Name** (with its description), **Type**, **Service**, **Points**, and **Last Seen**. Click **Load more** to page through long lists.

Click a metric to open it.

## Chart a metric

The metric page has a chart, a breakdown panel, and a side panel of metadata.

Query controls above the chart:

| Control       | What it does                                                                                                                                                          |
| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Aggregate** | How data points combine per time bucket. Monotonic sums offer `rate` (the default), `increase`, and `sum`. Other types offer `avg`, `sum`, `min`, `max`, and `count`. |
| **Where**     | A filter on attributes, with autocomplete. For example `http.route = "/api/users"`.                                                                                   |
| **Group by**  | **Everything (no breakdown)**, `service.name`, or any data point attribute as `attr.<key>`.                                                                           |
| **Every**     | The bucket size in seconds. **Auto** picks one from the time range.                                                                                                   |

The chart header shows the query, such as `rate(http.server.requests) by service.name`, and the metric's unit.

The **Top values** panel ranks values of one attribute. Choose it with **Break down by...** (default `service.name`). Click a bar to add that value to **Where**.

The side panel shows the type, unit, and whether a sum is monotonic. It also shows **Datapoints in range**, **First seen**, **Last seen**, how many services emit the metric, and each attribute key with its usage count.

## Save and share

- **Add to dashboard** adds the chart to an existing dashboard, or creates one with **Create & add**.
- **Create alert** opens a new [alert rule](/docs/alerting/alert-rules) with this query filled in.
- **Copy link** copies a URL that restores the aggregation, filter, grouping, and step.

## Query metrics from an assistant

The [MCP server](/docs/reference/mcp) exposes the same data:

- `list_metrics`: search metrics by name, service, or type, with unit, monotonicity, and volume.
- `query_data` with `source=metrics`: a time series or breakdown for one metric, with the same aggregations and grouping by service or attribute.

## Troubleshooting

- **"No metrics found".** No metric matched the search, type, or time range. Clear the search and widen the range.
- **"No data for this metric in the selected range".** The metric exists but had no data points in this window. Widen the range, or check that the emitting service is still running.
- **A summary metric is missing.** OTLP summaries are dropped at ingest. Export histograms instead.
- **"Not enough datapoints for a preview".** The metric has too few data points in range to draw a sparkline. Open the metric to see its values.
- **Rates look wrong.** `rate` and `increase` apply to monotonic cumulative sums. For a gauge, use `avg` or `max`.

## Next steps

- [Build dashboards](/docs/dashboards/build-dashboards): combine metric charts with trace and log queries.
- [Prometheus scraping](/docs/integrations/prometheus): pull metrics from exporters.
- [Uptime monitoring](/docs/alerting/uptime-monitoring): alert on HTTP check metrics from the OpenTelemetry Collector.
- [Services](/docs/explore/services): latency and error metrics derived from traces.

---

# Services

> Explore · https://maple.dev/docs/explore/services

Latency percentiles, error rate, throughput, and Apdex for every service that sends traces, with a detail page per service for its endpoints, operations, and dependencies.

The **Services** page lists every service that sent spans in the selected time range, with latency percentiles, error rate, and throughput. Open a service to see its charts over time, its HTTP endpoints, its operations, the services and databases it calls, and its open error issues.

## What data it needs

Services are built from traces. Each distinct `service.name` resource attribute is one service. The metrics come from entry spans: root spans and spans with kind `Server` or `Consumer`.

- **Error rate** counts spans with status `Error`.
- **Environment** and **Namespace** read `deployment.environment` and `service.namespace`.
- **Commit SHA** and **Last deploy** read `vcs.ref.head.revision`.

See [OpenTelemetry conventions](/docs/concepts/otel-conventions#service-identity) for these attributes.

## The services list

The default time range is the last 12 hours.

The table has these columns: **Service**, **P50**, **P95**, **P99**, **Error Rate**, **Throughput**, and **Last deploy**. **Error Rate** and **Throughput** include a sparkline. Rows are grouped by environment, or by namespace when your services set one. Switch grouping with **Group services by**.

<figure class="shot">
  <img src="/screenshots/docs/services-01-list.webp" alt="The Services list grouped by namespace and environment, with P50, P95 and P99 latency, error rate and throughput sparklines, and the last deploy for each service. A filter sidebar on the left lists health, environment, namespace and commit SHA." loading="lazy" />
  <figcaption>Services grouped by namespace and environment, each with latency percentiles, error rate, throughput and last deploy.</figcaption>
</figure>

A colored dot marks a service as degraded or unhealthy. Health comes from open alert incidents (**Alert firing**) and anomalies (**Anomaly**) on that service. A critical cause makes a service unhealthy. Any other cause makes it degraded. The footer counts services and how many are unhealthy or degraded.

### Filters

The sidebar filters by **Health**, **Environment**, **Namespace**, and **Commit SHA**. Each filter can include or exclude a value.

Click a row to open the service, scoped to that row's environment.

## Service detail

The header has an environment switcher (**All environments** or one environment) and a **Create Alert** button. The page has four tabs.

### Overview

Four charts: **Latency**, **Throughput** (requests per second), **Apdex**, and **Error Rate**. Deploys appear as commit markers on every chart.

<figure class="shot">
  <img src="/screenshots/docs/services-02-detail.webp" alt="The overview tab of the maple-api service: Latency, Throughput, Apdex and Error Rate charts over 12 hours with commit markers along the top, and a Top operations list below." loading="lazy" />
  <figcaption>A service's Overview tab. The dashed markers on each chart are deploys.</figcaption>
</figure>

**Apdex** scores requests from 0 to 1 against a 500 ms target (T):

- Satisfied: no error and duration under T.
- Tolerating: no error and duration from T up to 4T.
- Frustrated: slower than 4T, or an error.

Apdex = (satisfied + 0.5 × tolerating) / total. See [Apdex alerts](/docs/alerting/apdex-alerts) to alert on it.

Panels below the charts:

- **Top operations**, with **View all** to open the **Operations** tab.
- **Open issues**: [error issues](/docs/errors/overview) for this service from the latest 90 days.
- **Recent deploys**.
- **Ingest this window**: logs, spans, and metrics this service sent in the time range.
- **Kubernetes**: workloads running the service, with CPU and memory, when Kubernetes resource attributes are present.
- **Talks to**: the services and databases it calls.

### API

HTTP endpoints served by this service, with **Req/s**, **Errors**, **p50**, **p95**, and **p99**. Summary stats show the endpoint count, total requests per second, errors, and the worst p99. Requests that did not match a route are split into **Unrouted paths** and **Scanner probes**, hidden until you click **Show anyway**. Click an endpoint to open matching traces.

### Operations

Every span name the service handles, with **Calls /s**, **Errors**, **p50**, and **p95**. Click a row to open matching traces.

### Dependencies

Everything this service calls, with **Calls /s**, **Errors**, and **p95**. Each row has a type badge: **Service**, **Database**, **HTTP**, **Queue**, or **RPC**. The header highlights the **Busiest**, **Most errors**, and **Slowest p95** dependency. Click a row to open the traces for those calls.

Dependencies come from spans with kind `Client` or `Producer`. See [Service map](/docs/explore/service-map) for how Maple identifies the target of a call.

## Query services from an assistant

The [MCP server](/docs/reference/mcp) exposes the same data:

- `list_services`: throughput, error rate, and P95 per service, filterable by environment.
- `diagnose_service`: health metrics, Apdex, top errors, and recent traces and logs for one service.
- `get_service_top_operations`: a service's operations ranked by a chosen metric (call count by default).

## Troubleshooting

- **"No services yet".** Maple has never received a span. Services come from traces, so set up an OpenTelemetry SDK first. See [Instrumentation](/docs/instrumentation).
- **"No services in this time range".** Maple has traces, just not in this window. Widen the range.
- **"No services match these filters".** Your filters excluded everything. Clear them.
- **A service is missing.** Only services with entry spans (root, `Server`, or `Consumer`) get a row. A service that only emits `Client`, `Producer`, or `Internal` spans with a parent does not.
- **A service appears under the wrong name.** Set `service.name` explicitly in your SDK. Without it, many SDKs report `unknown_service`.
- **"No HTTP endpoints in this window" on the API tab.** An endpoint needs a `Server` span that carries a route. Most HTTP framework instrumentations set `http.route`. See [HTTP attributes](/docs/concepts/otel-conventions#http-attributes).

## Next steps

- [Service map](/docs/explore/service-map): how services and databases connect.
- [Traces](/docs/explore/traces): the requests behind each number.
- [Apdex alerts](/docs/alerting/apdex-alerts): alert when user-facing latency degrades.

---

# Service map

> Explore · https://maple.dev/docs/explore/service-map

A live graph of your services and the databases they call, drawn from trace context, with call volume, error rate, and latency on every node and edge.

The **Service Map** draws your services and the databases they call as a graph. Maple builds it from your traces: an edge appears when one service calls another with trace context propagated, or when a service calls a database. Each node shows throughput, error rate, and latency. Each edge shows call volume and errors. Click a node to see its details.

## What creates nodes and edges

### Service to service

Maple draws an edge from service A to service B when a `Client` or `Producer` span in A has a child `Server` or `Consumer` span in B, in the same trace. Calls within one service do not draw an edge.

Both sides must be instrumented, and the caller must propagate trace context (the `traceparent` header) so the callee's span becomes a child of the client span. Instrumented HTTP and RPC clients do this for you. `peer.service` does not create edges.

### Service to database

A `Client` or `Producer` span with `db.system.name` (or the older `db.system`) creates a database node. Maple splits nodes by database name, taken from `db.namespace`, then `db.name`, then `server.address`, then `net.peer.name`. Services that call the same database share one node.

Database nodes are labeled by category from the `db.system.name` value:

| Category | Systems                                                   |
| -------- | --------------------------------------------------------- |
| Cache    | `redis`, `memcached`, `hazelcast`                         |
| Queue    | `kafka`, `rabbitmq`, `pulsar`, `nats`, `activemq`, `sqs`  |
| Search   | `elasticsearch`, `opensearch`, `solr`                     |
| Database | Everything else, such as `postgresql`, `mysql`, `mongodb` |

When the [PlanetScale integration](/docs/integrations/planetscale) is connected, PlanetScale databases are marked on their nodes. Databases reached through Cloudflare Hyperdrive are grouped into one **Hyperdrive** node.

### Calls the map does not draw

Outbound HTTP calls to hosts you do not instrument, and messaging or RPC calls without `db.system.name`, do not appear on the map. They appear on the **Dependencies** tab of the calling [service](/docs/explore/services#dependencies), keyed by `server.address`, `messaging.system`, or `rpc.system`.

See [OpenTelemetry conventions](/docs/concepts/otel-conventions#service-map) for the attributes to set.

## Read the map

Service nodes show requests per second (**req/s** or **calls/s**), error percentage (**err%**), and average latency (**avg**). A badge shows the runtime platform, such as **Kubernetes**, **Cloudflare Workers**, **AWS Lambda**, or **Web (browser)**. With Kubernetes infrastructure monitoring set up, nodes also show a pod count.

<figure class="shot">
  <img src="/screenshots/docs/service-map-01-map.webp" alt="The Service Map in 2D: services grouped by namespace, each node showing requests per second, error percentage and average latency, with animated edges to databases such as Postgres, ClickHouse and Tinybird on the right." loading="lazy" />
  <figcaption>The Service Map. Databases and external services sit to the right of the services that call them.</figcaption>
</figure>

Node color follows error rate. The legend shows **Healthy**, **Degraded** (above 1%), and **Error** (above 5%).

Each edge shows its call count and, when above zero, its error percentage. Counts come from traced requests. When your SDK samples traces, counts are prefixed with `~` and the real rate may be higher. See [Sampling and throughput](/docs/concepts/sampling-throughput).

The map shows at most 200 edges.

## Controls

| Control                                 | What it does                                                                                                                                                                   |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **2D** / **3D**                         | Switch between a flat layout (the default) and a 3D view.                                                                                                                      |
| Environment                             | Show one `deployment.environment`, or **All Environments**. Defaults to `production` when it exists.                                                                           |
| Time range                              | Defaults to the last 12 hours.                                                                                                                                                 |
| **Focus a service…**                    | Center the map on one service and its neighbors. Pick **1 hop** or **2 hops**. **Hide rest** hides everything outside the focus instead of dimming it. **Clear focus** resets. |
| Traffic threshold                       | **All traffic**, or hide edges below 0.1%, 1%, or 5% of the busiest edge. A button shows how many services and edges are hidden.                                               |
| **Color nodes by**                      | **Service**, **Health**, or **Platform**.                                                                                                                                      |
| **Re-sort**                             | Discard manual positions and auto-arrange.                                                                                                                                     |
| **Zoom In**, **Zoom Out**, **Fit View** | Canvas zoom. You can also scroll to zoom and drag nodes to arrange them.                                                                                                       |

The focused service, hop count, and hide mode are kept in the URL, so you can share a focused view.

## Node details

Click a node to open its side panel. Click the canvas to close it. Click a collapsed namespace to expand it.

A service panel has two tabs:

- **Service**: **Throughput**, **Error Rate**, **Avg Latency**, and **P95 Latency**, plus **Dependencies** and **Called By** lists. **View service** opens the [service page](/docs/explore/services).
- **Infrastructure**: the Kubernetes workloads running the service. See [Kubernetes](/docs/infrastructure/kubernetes) to set it up.

A database panel shows **Queries**, **Throughput**, **Error Rate**, **P50 Latency**, **P95 Latency**, and **Avg Latency**.

## Query the map from an assistant

The [MCP server](/docs/reference/mcp) tool `service_map` returns the edges with call counts, error rates, and latency, filtered by service and environment.

## Troubleshooting

- **"No service map yet".** Maple has not seen a cross-service call or a database call. Instrument at least two services that call each other, or a service with database instrumentation. If your services are active, widen the time range.
- **Two services are not connected.** The callee is not instrumented, or the caller drops the `traceparent` header (common across proxies, queues, and custom HTTP clients). Open a trace for the call and check that the callee's `Server` span has the client span as its parent.
- **Every database collapses into one node.** Set `db.namespace` on database spans, or at least `server.address`.
- **The same database appears twice.** Services spell `db.system.name` differently (for example `postgresql` and `PostgreSQL`). Use the OpenTelemetry well-known values.
- **The Infrastructure tab says "No Kubernetes workloads found".** The service's spans lack Kubernetes workload attributes. See [Kubernetes](/docs/infrastructure/kubernetes).

## Next steps

- [OpenTelemetry conventions](/docs/concepts/otel-conventions#service-map): attributes that shape the map.
- [Services](/docs/explore/services): per-service metrics and all outbound dependencies.
- [Traces](/docs/explore/traces): inspect the calls behind an edge.

---

# Errors and issues

> Errors · https://maple.dev/docs/errors/overview

How Maple groups error spans into issues, how an issue moves from triage to done, and how to claim, prioritize, and get notified about them.

Maple turns error spans into **issues**. Every error span is fingerprinted, and all occurrences with the same fingerprint belong to one issue. An issue is the durable record of a bug: it has a workflow state, a severity, a lease for whoever is working on it, and a history. The **Errors** page lists your issues. Each issue page shows its stack, its occurrences, and the traces behind them.

An issue can flare up many times. Each flare-up is an **incident** under the issue. The issue is the bug; incidents record when it was active.

## What counts as an error

An error is a span with status `Error`. Log severity does not create errors.

Maple reads the exception from the span in this order:

1. The first `exception` span event: `exception.type`, `exception.message`, and `exception.stacktrace`.
2. The span's status message, when there is no exception event.
3. The span attributes `exception.type`, `exception.message`, and `exception.stacktrace`, then `error.type` and `error.message`.

If none of these are set, the error is labeled **Unknown Error**. Record exceptions on the span (most SDKs have a `recordException` call) to get a useful title and stack.

HTTP 4xx spans are skipped unless they carry real exception data. A span with status `Error` and a 400 to 499 status code is ignored when it has no exception event, no `exception.type` attribute, and an `error.type` that is empty or equal to the status code. 5xx spans are always kept.

See [Errors and exceptions](/docs/concepts/otel-conventions#errors-and-exceptions) for the attributes to set.

## How errors are grouped

The fingerprint combines five things:

- The organization.
- The service (`service.name`).
- The exception type.
- The top 3 stack frames, with line numbers, URL origins, bundle hashes, and memory addresses removed.
- A message signature: the first 120 characters of the message after emails, URLs, quoted values, IDs, and numbers are replaced with placeholders. JSON messages are normalized by key.

The environment is not part of the fingerprint, so the same bug in staging and production is one issue.

A fingerprint becomes an issue after 3 occurrences. Until then it notifies nobody. A fingerprint that stays below 3 occurrences for 24 hours is dropped.

Issues come from three sources. The **Source** filter names them:

- **Exceptions**: fingerprinted errors from spans.
- **Alert rules**: opened by an [alert rule](/docs/alerting/alert-rules), one per rule and group.
- **Integrations**: raised by a connected system, such as a [PlanetScale](/docs/integrations/planetscale) event.

## Issue states

| State           | Set by        | Meaning                                                |
| --------------- | ------------- | ------------------------------------------------------ |
| **Triage**      | Maple, or you | New and not yet looked at.                             |
| **Todo**        | You           | Accepted, not started.                                 |
| **In progress** | You           | Someone holds the lease and is working on it.          |
| **In review**   | You           | A fix is proposed or a pull request is attached.       |
| **Verifying**   | Maple         | The fix merged. Maple is watching for new occurrences. |
| **Done**        | You, or Maple | Fixed.                                                 |
| **Regressed**   | Maple         | It was done, and it fired again from a newer build.    |
| **Won't fix**   | You           | Suppressed, forever or until a snooze time.            |
| **Cancelled**   | You           | Closed for good. Cannot be reopened.                   |

Maple sets **Regressed** and **Verifying** itself. You cannot pick them.

### Regressions

A **Done** issue becomes **Regressed** when a new occurrence arrives more than 1 hour after it was resolved, from a `service.version` that was not running when it was resolved. Occurrences from the old build only update counts. They open no incident and send no notification. If your services report no `service.version`, any occurrence after the 1-hour grace period is a regression.

**Regressed** is distinct from **Triage** so the next person can see the bug was fixed once before.

### Won't fix and snooze

**Won't fix** suppresses an issue. With a snooze time, the issue returns to **Triage** when it expires. Set a snooze time with the `transition_error_issue` MCP tool (`snooze_until`). The issue page shows **Snoozed until**.

### Fix verification

Attach a GitHub pull request to an issue from the **Pull requests** panel (**Attach**). This requires the [GitHub integration](/docs/integrations/github). When the pull request merges, the issue moves to **Verifying** and a **Fix verification** card shows progress:

- **Watching**: Maple is counting occurrences after the merge. The window is sized to see about 20 occurrences at the pre-merge rate, within a range set by severity: 30 minutes to 1 day for critical, 1 to 3 days for high, 4 hours to 7 days for medium, and 12 hours to 14 days for low or unset.
- **Fix holds**: no occurrences from builds that postdate the merge. Issues with low, medium, or no severity close as **Done** automatically. High and critical issues show the verdict and wait for you to close them.
- **Still broken**: the error kept firing. The issue returns to **In progress** or **In review**.
- **Inconclusive**: the window ended without a clear answer. After a first inconclusive check, Maple watches for longer.
- **Stopped**: verification was abandoned.

### Archiving

**Done** issues resolved more than 14 days ago are archived. Archived issues are deleted after 90 days.

## Claim an issue

An issue has one lease at a time, so two people (or two agents) do not fix the same bug. In the issue sidebar under **Lease**:

- **Claim** takes the lease and moves a **Triage**, **Regressed**, or **Todo** issue to **In progress**.
- **Extend** renews it.
- **Release** gives it up. An **In progress** issue returns to **Todo**.

A lease lasts 30 minutes by default. Any action by the holder renews it. When it expires, an issue still **In progress** returns to **Todo**. Moving an issue to **In progress** also takes the lease. Claiming fails while someone else holds it.

## Severity

Severity is **Critical**, **High**, **Medium**, **Low**, or not set. Set it from the **Severity** select in the issue sidebar, or for several issues at once from the bulk bar. A severity you set by hand overrides one set by an agent or an alert rule.

To route issues to destinations by severity, open **Settings → Automation** and turn on **Severity escalation**. Pick destinations per severity and click **Save policy**. Escalation fires once per issue and severity level, and only upward. When a manual severity change would notify destinations, Maple asks you to confirm with **Change severity and notify**. Only organization admins can change this policy.

## Notifications

The error notification policy decides when Maple sends an issue to your [notification destinations](/docs/alerting/notification-destinations). Defaults:

| Event                     | Default |
| ------------------------- | ------- |
| First seen (new incident) | On      |
| Regression                | On      |
| Incident resolved         | Off     |

A minimum occurrence count (default 1) applies to first-seen and regression notifications. No destinations are selected by default, so nothing is sent until you add one.

The Errors page has no screen for this policy. Organization admins change it with the `update_error_notification_policy` MCP tool, which sets the destinations, the first-seen, regression, and resolve toggles, and the minimum count.

An incident opens when an issue fires with no open incident. It resolves after 30 minutes without new occurrences, or when the issue is marked **Done**.

## The Errors page

The page opens with a stat strip: error count, share of all spans, and affected services and traces.

<figure class="shot">
  <img src="/screenshots/docs/errors-01-list.webp" alt="The Errors page on the Open tab: a stat strip with error count, share of spans, services and traces, then a list of issues with a 24-hour trend, event count, service, status and last seen time, next to service, environment and source filters." loading="lazy" />
  <figcaption>The Errors page. Each row is an issue, with its trend over the last 24 hours.</figcaption>
</figure>

Tabs filter by state:

| Tab          | States                                          |
| ------------ | ----------------------------------------------- |
| **Open**     | Triage, Regressed, Todo, In progress, In review |
| **Triage**   | Triage, Regressed                               |
| **Active**   | Todo, In progress, In review                    |
| **Resolved** | Done, Cancelled, Won't fix                      |
| **All**      | Every state                                     |

Columns: **Error**, **Trend · 24h**, **Events**, **Service**, **Status**, and **Last seen**. The trend and event counts cover the last 24 hours. The list covers all time.

- Sort by **Most recent** (default), **Most errors**, or **Severity**.
- Filter by severity, and in the sidebar by **Regressed only**, **Service**, **Environment**, and **Source**.
- Select rows to change **Severity** or **Move to** a new state in bulk.
- Right-click a row for **Change status**, **Open in new tab**, **Copy link**, **Copy ID**, and **Copy agent prompt**.

The list loads 50 issues at a time. Click **Load more** for the next page.

## The issue page

The header shows the service, the error title, severity, state, and an **Open incident** marker while one is open. It has three tabs:

- **Overview**: the **Culprit** (top stack frame) and **Fingerprint**; facts such as **Events · all time**, **First seen**, **Last seen**, and **Regressions**; an occurrence chart; and an **Incidents** table with each incident's status, reason (**First seen**, **Regression**, or **Manual**), and event count.
- **Occurrences**: **Latest occurrences**, with time, service, message, and a link to each trace. Shown for issues from spans.
- **Activity**: a timeline of state changes, claims, and comments. Add a comment with **Comment**. Markdown is supported.

<figure class="shot">
  <img src="/screenshots/docs/errors-02-issue.webp" alt="An issue page for @maple/cli/ServerError: the error message, culprit and fingerprint, event counts with first and last seen, an occurrences chart, a linked investigation, and a details panel with status, severity, priority, scope and pull requests." loading="lazy" />
  <figcaption>An issue page. The right panel holds state, severity, lease and linked pull requests.</figcaption>
</figure>

The sidebar holds **Details** (status, severity, assignee), **Scope** (service, environment, issue ID), **Lease**, **Pull requests**, and **Fix verification**.

**Copy agent prompt** copies a prompt with the issue context for a coding agent.

## Work with issues from an assistant

The [MCP server](/docs/reference/mcp) covers the whole workflow:

| Tool                                       | What it does                                                           |
| ------------------------------------------ | ---------------------------------------------------------------------- |
| `list_error_issues`                        | List issues by state, severity, source, service, or last seen time.    |
| `find_errors`                              | Error types with their fingerprints and counts.                        |
| `error_detail`                             | Sample occurrences and logs for one fingerprint.                       |
| `list_error_issue_events`                  | An issue's activity history.                                           |
| `list_error_incidents`                     | Incidents under an issue.                                              |
| `claim_error_issue`, `release_error_issue` | Take or give up the lease.                                             |
| `transition_error_issue`                   | Change state, with an optional `snooze_until` for Won't fix.           |
| `set_issue_severity`                       | Set severity. Never overrides a severity set by hand.                  |
| `comment_on_error_issue`                   | Add a comment.                                                         |
| `propose_fix`                              | Claim the issue, move it to In review, and optionally attach a PR URL. |
| `link_pull_request`                        | Attach a GitHub pull request. Its merge starts fix verification.       |
| `update_error_notification_policy`         | Change the notification policy (admins only).                          |

## Troubleshooting

- **Errors show in traces but no issue appears.** A fingerprint needs 3 occurrences to become an issue. Also check that the span status is `Error`, and that a 4xx span carries an exception.
- **Many issues titled "Unknown Error".** Your spans set status `Error` without an exception. Record the exception on the span.
- **One bug is split across many issues.** The message differs in a way the normalizer does not catch, or the stack differs in its top 3 frames. Put variable data in attributes instead of the exception message.
- **A fixed issue keeps regressing.** Old instances are still running. Set `service.version` on your services so occurrences from pre-fix builds do not count as regressions.
- **No notifications arrive.** The policy has no destinations by default. Add one with `update_error_notification_policy`.

## Next steps

- [Traces](/docs/explore/traces): inspect the requests behind an occurrence.
- [Notification destinations](/docs/alerting/notification-destinations): set up Slack, email, and webhooks.
- [GitHub](/docs/integrations/github): attach pull requests and verify fixes.
- [MCP server](/docs/reference/mcp): let a coding agent triage and fix issues.

---

# Browser SDK

> Session Replay · https://maple.dev/docs/session-replay/browser-sdk

Instrument a website with OpenTelemetry tracing, logs, Web Vitals, error capture and session replay using the @maple-dev/browser SDK.

`@maple-dev/browser` adds OpenTelemetry tracing, logs, Web Vitals, error capture and session replay to a website in one package. Everything it sends is OpenTelemetry (OTLP traces and logs), apart from the replay recording itself. Every span and every replay event carries the same `session.id`, so a trace links to the replay that produced it, and a replay links to its traces.

<div class="flex flex-wrap gap-2 mb-8 not-prose">
    <span class="text-[10px] uppercase tracking-wider px-2 py-1 border border-border text-fg-muted">Browsers</span>
    <span class="text-[10px] uppercase tracking-wider px-2 py-1 border border-border text-fg-muted">Beta</span>
</div>

> **Using Effect?** The browser entry point of `@maple-dev/effect-sdk` has the same replay engine built in, with the recorder in a lazily loaded chunk. See [Session Replay & Sessions](/docs/sdks/effect-client#session-replay-and-sessions). Run replay from one SDK, not both.

## Install

```bash
npm install @maple-dev/browser
```

## Quick start

Call `MapleBrowser.init` once, as early as possible in your app's entrypoint:

```ts
import { MapleBrowser } from "@maple-dev/browser"

MapleBrowser.init({
	ingestKey: "maple_pk_...", // public ingest key
	serviceName: "acme-web",
	region: "eu", // only for EU organizations; omit for the US region
})
```

Use a **public** ingest key (`maple_pk_…`) from **Settings → Ingestion**. It is safe to ship in browser code. Ingest keys belong to one region, so set `region: "eu"` if your organization is in the EU region.

That single call:

- starts OpenTelemetry browser tracing, auto-instrumenting `fetch` and exporting to Maple's ingest (`POST /v1/traces`);
- records uncaught errors and unhandled promise rejections as error spans;
- records the session with rrweb, in chunks of about 5 seconds or 100 KB, gzipped with the browser's `CompressionStream` and uploaded to `POST /v1/sessionReplays/blob`;
- writes session metadata at start (`active`) and on page hide (`ended`), including the trace ids observed during the session.

The SDK is best-effort. A telemetry network failure never throws into your app.

`init()` returns a handle, `{ sessionId, shutdown }`, for reading the active session id and tearing telemetry down. See [Sessions](#sessions).

## Configuration

Every field accepted by `MapleBrowser.init`:

| Option                                 | Type                                                     | Default                                   | Description                                                                                                                                                                |
| -------------------------------------- | -------------------------------------------------------- | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `serviceName`                          | `string`                                                 | none                                      | **Required.** Service name reported on traces and stored on replay sessions.                                                                                               |
| `ingestKey`                            | `string`                                                 | none                                      | Public ingest key (`maple_pk_...`), sent as `Authorization: Bearer`. Leave it unset only when `endpoint` points at your own proxy that adds the key.                       |
| `region`                               | `"us"` \| `"eu"`                                         | `"us"`                                    | Region of your Maple organization. `"us"` sends to `https://ingest.maple.dev`, `"eu"` to `https://ingest.eu.maple.dev`. Ignored when `endpoint` is set.                    |
| `endpoint`                             | `string`                                                 | from `region`                             | Ingest base URL. Overrides `region`. Use it for a proxy.                                                                                                                   |
| `serviceNamespace`                     | `string`                                                 | none                                      | Logical group this service belongs to, sent as the `service.namespace` resource attribute on traces.                                                                       |
| `serviceVersion`                       | `string`                                                 | none                                      | Service version or commit SHA, attached to traces.                                                                                                                         |
| `environment`                          | `string`                                                 | none                                      | Deployment environment, for example `"production"`.                                                                                                                        |
| `user`                                 | `object`                                                 | none                                      | End-user identity: `id`, `email`, `username`, `groupId`, `groupName`, `traits`. Attached to the session and to browser spans. See [Identifying users](#identifying-users). |
| `userId`                               | `string`                                                 | none                                      | **Deprecated.** A bare user id. Use `user` instead. When both are set, `user` wins.                                                                                        |
| `tracing.enabled`                      | `boolean`                                                | `true`                                    | Enable OpenTelemetry browser tracing.                                                                                                                                      |
| `tracing.instrumentFetch`              | `boolean`                                                | `true`                                    | Create spans for `fetch()` calls. Set `false` when another tracer (such as the Effect client SDK) already instruments requests, to avoid duplicate network spans.          |
| `tracing.instrumentXhr`                | `boolean`                                                | `true`                                    | Create spans for `XMLHttpRequest` calls (axios and older clients). Turn it off for the same reason as `instrumentFetch`.                                                   |
| `tracing.captureErrors`                | `boolean`                                                | `true`                                    | Record uncaught errors and unhandled promise rejections as error spans. Turn it off only when another tool owns the page's global error handlers.                          |
| `tracing.propagateTraceHeaderCorsUrls` | `Array<string \| RegExp>`                                | `[]`                                      | Cross-origin URLs whose `fetch()` and XHR requests carry the `traceparent` header. See [Connect browser and backend traces](#connect-browser-and-backend-traces).          |
| `tracing.sampleRate`                   | `number`                                                 | `1`                                       | Fraction of sessions whose traces are exported, `0` to `1`. Error spans are always exported. See [Sampling](#sampling).                                                    |
| `tracing.captureHeaders`               | `{ request?, response? }`                                | none                                      | Header names to record on `fetch`/XHR spans. See [Request and response detail](#request-and-response-detail).                                                              |
| `tracing.longFrames`                   | `boolean`                                                | `false`                                   | Span main-thread frames of 100ms or more. See [Jank](#jank).                                                                                                               |
| `tracing.slowInteractions`             | `boolean`                                                | `false`                                   | Span interactions of 200ms or more. See [Jank](#jank).                                                                                                                     |
| `errors`                               | `object`                                                 | see [Filtering errors](#filtering-errors) | `ignore`, `denyUrls`, `allowUrls`, `beforeCapture`, `defaultFilters` and `captureHttpStatus`.                                                                              |
| `breadcrumbs`                          | `boolean`                                                | `true`                                    | Keep the last clicks, inputs, navigations and console lines, and send them with the next error. See [Breadcrumbs](#breadcrumbs).                                           |
| `webVitals`                            | `boolean`                                                | `true`                                    | Report Core Web Vitals. See [Web Vitals](#web-vitals).                                                                                                                     |
| `logs.captureConsole`                  | `Array<"debug" \| "log" \| "info" \| "warn" \| "error">` | `[]`                                      | Console levels sent as logs as they happen. See [Logs](#logs).                                                                                                             |
| `reporting.csp`                        | `boolean`                                                | `true`                                    | Content Security Policy violations as logs. See [Browser reports](#browser-reports).                                                                                       |
| `reporting.browserReports`             | `boolean`                                                | `false`                                   | Browser deprecation and intervention reports as logs.                                                                                                                      |
| `transport.offline`                    | `boolean`                                                | `false`                                   | Keep batches that could not be sent and send them later. See [Offline](#offline).                                                                                          |
| `replay.enabled`                       | `boolean`                                                | `true`                                    | Enable session recording.                                                                                                                                                  |
| `replay.sampleRate`                    | `number`                                                 | `1`                                       | Fraction of sessions to record, `0` to `1`. See [Sampling](#sampling).                                                                                                     |
| `replay.onErrorSampleRate`             | `number`                                                 | `0`                                       | Fraction of the sessions not recorded that keep the last minute in memory and upload it only if an error happens. See [Replay on error](#replay-on-error).                 |
| `replay.canvasFps`                     | `number`                                                 | off                                       | Record `<canvas>` content at this many frames per second. Never with `privacy.maskAllText`.                                                                                |
| `replay.networkBodies`                 | `{ urls, maxLength? }`                                   | none                                      | Keep request and response bodies of these URLs in the replay. See [Request and response detail](#request-and-response-detail).                                             |
| `privacy.maskAllInputs`                | `boolean`                                                | `true`                                    | Mask all `<input>` values in the recording.                                                                                                                                |
| `privacy.maskAllText`                  | `boolean`                                                | `false`                                   | Mask all text in the recording, and omit captured click-target text from session events.                                                                                   |
| `privacy.sanitizeUrl`                  | `(url: string) => string`                                | none                                      | Rewrite every URL before it leaves the page. See [Redacting URLs](#redacting-urls).                                                                                        |
| `privacy.persistVisitorId`             | `boolean`                                                | `true`                                    | Store a persistent visitor id (localStorage and cookie) so unique and returning visitors can be counted. Turning it off also deletes any id already stored.                |
| `privacy.persistence`                  | `"browser" \| "memory"`                                  | `"browser"`                               | `"memory"` keeps everything in memory: no cookies, Web Storage or IndexedDB, and no visitor id. See [Memory-only mode](#memory-only-mode).                                 |
| `privacy.crossSubdomainCookie`         | `boolean`                                                | `true`                                    | Scope the visitor-id cookie to the registered domain so sibling subdomains share it. See [Linking a marketing site to your app](#linking-a-marketing-site-to-your-app).    |
| `privacy.cookieDomain`                 | `string`                                                 | probed                                    | Explicit cookie `Domain=` (no leading dot). `""` forces a host-only cookie.                                                                                                |
| `privacy.requireConsent`               | `boolean`                                                | `false`                                   | Capture nothing until `MapleBrowser.setConsent(true)`. See [Consent](#consent).                                                                                            |
| `privacy.captureUserEmail`             | `boolean`                                                | `true`                                    | Store the email passed to `identify()`.                                                                                                                                    |
| `privacy.respectDoNotTrack`            | `boolean`                                                | `false`                                   | Treat `navigator.doNotTrack` like Global Privacy Control (suppresses the persistent visitor id).                                                                           |

A fully-specified call:

```ts
MapleBrowser.init({
	ingestKey: "maple_pk_...",
	serviceName: "acme-web",
	environment: "production",
	serviceVersion: "1.4.2",
	user: currentUser ? { id: currentUser.id, email: currentUser.email } : undefined,
	tracing: {
		enabled: true,
		instrumentFetch: true,
		captureErrors: true,
		propagateTraceHeaderCorsUrls: [/^https:\/\/api\.example\.com\//],
	},
	replay: { enabled: true, sampleRate: 1.0 },
	privacy: { maskAllInputs: true, maskAllText: false },
})
```

## Sessions

Every span and replay event the SDK emits carries one **`session.id`** (a `crypto.randomUUID()` v4), created on the first `MapleBrowser.init` call. That shared id is what links a trace to the replay that produced it.

### Storage and continuity

The session is stored in `sessionStorage` under the key `maple.session`, so it **survives reloads within a tab**. `sessionStorage` is per tab, so **each tab or window gets its own session**. When `sessionStorage` is unavailable (for example in some private-browsing modes), the SDK keeps the session in memory for the life of the page.

The replay chunk a page is still sending when it unloads is kept in `sessionStorage` under `maple.replay.pending` and sent by the next page of the same session in that tab, if that page starts recording within ten minutes. Revoking consent removes it; with `privacy.requireConsent`, it is sent once consent is granted on that next page, unless it was revoked in between.

Client-side route changes in a single-page app do **not** start a new session. Session boundaries are purely time-based.

### Rotation

A new `session.id` is created when either limit is crossed, whichever comes first:

- **30 minutes idle.** No recorded activity for half an hour rotates the session.
- **24 hours old.** A hard cap on a session's lifetime regardless of activity, so a tab left open for days does not become one giant replay.

While replay is recording, each uploaded chunk marks the session active and pushes back the idle deadline, so a session that keeps recording stays whole.

### Start and end metadata

The SDK writes a small session-metadata row at two points:

- an **`active`** row when recording starts (and again on each reload);
- an **`ended`** row when the page is hidden or unloaded. It fires on `visibilitychange` to `hidden` (the reliable "leaving" signal on mobile) and on `pagehide` (tab close or navigation on desktop).

The `ended` row carries the session duration, the click count, and the **trace ids observed during the session**. Maple uses these to link traces and replays, and to fill the user and session columns on the Replays page. The unload write uses `keepalive`, so it survives the page going away.

### Accessing the session id

`init()` returns a handle whose `sessionId` is the active session's id. Use it to correlate Maple sessions with your own backend logs:

```ts
const { sessionId } = MapleBrowser.init({
	ingestKey: "maple_pk_...",
	serviceName: "acme-web",
})

// for example, forward it on your own requests
fetch("/api/checkout", { headers: { "x-maple-session": sessionId } })
```

`init()` is idempotent: calling it again returns the same live handle. On the server (SSR, no `window`) it returns a no-op handle with an empty `sessionId`.

### Teardown

Call `shutdown()` to upload the final replay chunk and stop tracing and replay. After it resolves, telemetry is stopped and a later `init()` may start a new session. Use it when a single-page app unmounts its telemetry client:

```ts
const maple = MapleBrowser.init({ ingestKey: "maple_pk_...", serviceName: "acme-web" })

// later, on teardown
await maple.shutdown()
```

## Identifying users

Pass `user` to `init()` so replays and traces are tied to a known user. It fills the user columns on the Replays page, and browser-created spans include `user.id`:

```ts
MapleBrowser.init({
	ingestKey: "maple_pk_...",
	serviceName: "acme-web",
	user: {
		id: "user_123",
		email: "ada@acme.com",
		username: "ada",
		groupId: "org_42",
		groupName: "Acme",
		traits: { plan: "pro", signup_month: "2026-01" },
	},
})
```

`groupId` and `groupName` are the company or team the Replays page can group by. `traits` are capped at 24 keys, 64-character keys and 256-character values. The identity is never written to browser storage.

If you do not know the user at init time (for example, the SDK starts before login resolves), leave `user` out and the session starts anonymous. Call `MapleBrowser.identify()` once you know who the user is. It takes the same object, or a bare user id:

```ts
// after the user signs in
MapleBrowser.identify({ id: user.id, email: user.email, groupId: org.id, groupName: org.name })

// after the user signs out
MapleBrowser.identify(null)
```

Each call **replaces** the identity. It does not merge, so a signed-out user's email never carries over to whoever signs in next on a shared device. Future session rows and spans read the latest identity.

`userId` still works as a bare user id, but it is deprecated. Use `user`.

## Capturing errors

With `tracing.captureErrors` on (the default), uncaught errors and unhandled promise rejections are recorded as error spans with status `Error`, which feed the [Errors](/docs/errors/overview) page.

An error your app catches never reaches those global handlers. Report it with `captureException`, for example from a framework error boundary:

```ts
try {
	await submitOrder()
} catch (error) {
	MapleBrowser.captureException(error, {
		name: "checkout.submit_failed", // span name, default "exception"
		attributes: { "order.step": "payment" },
	})
	showErrorToast()
}
```

The same error object is recorded once, even if your code reports it and then rethrows it, or [`traced`](#navigation-and-data-loading-spans) already recorded it. `captureException` accepts any thrown value: strings and plain objects are turned into an `Error`. Calls before `init()` do nothing.

A cross-origin script that throws shows up in the browser as a bare "Script error." with no details, and the SDK skips it. Add the `crossorigin` attribute to the script tag to get the real error.

`error.cause` chains and the errors inside an `AggregateError` (up to five) are added to the stack trace as `Caused by:` blocks, after the error's own frames.

### Filtering errors

Filters run before an error is recorded, so a dropped error costs nothing and never opens an issue. They apply to the global handlers and to `captureException`.

```ts
MapleBrowser.init({
	// ...
	errors: {
		ignore: ["ChunkLoadError", /^AbortError: /], // matched against "Name: message"
		denyUrls: [/widgets\.example\.net/], // matched against the top frame's script URL
		allowUrls: [/^https:\/\/app\.example\.com\//], // errors with no stack frames are kept
		beforeCapture: (error, { source, originalError }) => !error.message.includes("401"),
	},
})
```

Errors thrown by browser extensions and the harmless `ResizeObserver loop` notices are dropped by default. Set `errors.defaultFilters: false` to keep them.

### Failed HTTP requests

`fetch` and `XMLHttpRequest` spans follow the OpenTelemetry HTTP conventions for client requests: a response with a 4xx or 5xx status marks the span as an error, with `error.type` set to the status code (`"404"`, `"503"`), so it opens an issue. Issues group by status code.

To count fewer statuses, narrow the list (the default is `[[400, 599]]`). A status left out is not an error:

```ts
errors: {
	captureHttpStatus: [[500, 599], 429]
} // a 404 from a search box is expected here
```

A request that gets no response at all (offline, DNS, CORS, or a timeout from `AbortSignal.timeout()`) is always an error. A request your code aborts with its own `AbortController` is not.

### Breadcrumbs

The SDK keeps the last 50 clicks, inputs, navigations and console lines in memory. Nothing is sent until an error is recorded. Then the trail is sent as OpenTelemetry log records linked to the error, so you see what the user did right before it on the error's trace. Each breadcrumb is sent once. Clicks and inputs are recorded as a short selector (`button#save`), never with input values. Turn it off with `breadcrumbs: false`.

## Connect browser and backend traces

For a request to the same origin as the page, the `fetch()` span sends a W3C `traceparent` header, and your backend's span joins the same trace. For a request to another origin, such as `https://api.example.com` from `https://app.example.com`, the header is not sent unless you list the URL:

```ts
MapleBrowser.init({
	ingestKey: "maple_pk_...",
	serviceName: "acme-web",
	tracing: {
		propagateTraceHeaderCorsUrls: [/^https:\/\/api\.example\.com\//],
	},
})
```

Your API must also allow the header in its CORS configuration. Add `traceparent` to `Access-Control-Allow-Headers` in the preflight response. Without it, the browser blocks the request.

Your backend must be instrumented with OpenTelemetry and read `traceparent`, which every OpenTelemetry HTTP server instrumentation does.

## Logs

`MapleBrowser.logger` writes OpenTelemetry log records, each linked to the span that was active when it was written and to the session:

```ts
MapleBrowser.logger.info("checkout started", { "cart.items": 3 })
MapleBrowser.logger.error("payment declined", { "payment.provider": "card" })
```

To send console output as logs, list the levels: `logs: { captureConsole: ["warn", "error"] }`. Calls before `init()` are queued.

## Web Vitals

LCP, CLS, INP, FCP and TTFB are reported as OpenTelemetry log events named `browser.web_vital`, with the `browser.web_vital.name`, `value`, `delta`, `rating`, `id` and `navigation_type` attributes from the OpenTelemetry browser conventions, plus `url.path`. Each is linked to the page load's trace when you use [navigation spans](#navigation-and-data-loading-spans). CLS, INP and LCP are final when the page is hidden, so they arrive then. Turn them off with `webVitals: false`.

## Jank

Two opt-in options show where the main thread got stuck:

```ts
tracing: { longFrames: true, slowInteractions: true }
```

- `longFrames` records every frame of 100ms or more as a `longAnimationFrame` span, with the script that ran longest (file, function and what invoked it, such as `BUTTON#save.onclick`). Browsers without the Long Animation Frames API record `longtask` spans instead.
- `slowInteractions` records every interaction of 200ms or more as an `interaction click` (or `keydown`, ...) span, split into input delay, processing and presentation time, with the element that was used.

Both nest under the open navigation span, and include what happened before the SDK finished loading.

## Request and response detail

List the headers to record on `fetch` and XHR spans:

```ts
tracing: { captureHeaders: { request: ["x-request-id"], response: ["x-cache", "server-timing"] } }
```

They are stored as `http.request.header.<name>` and `http.response.header.<name>`. `authorization`, `proxy-authorization`, `cookie` and `set-cookie` are never recorded, even when listed. XHR spans get response headers only, and a cross-origin response only exposes the headers its server lists in `Access-Control-Expose-Headers`.

Request and response bodies can be kept in the session replay's network events, for the URLs you list only:

```ts
replay: {
	networkBodies: {
		urls: [/^https:\/\/api\.example\.com\/checkout/]
	}
}
```

Only text and JSON bodies are kept, cut to `maxLength` characters (1,000 by default, which is also the most Maple stores), and nothing is kept with `privacy.maskAllText`. Bodies can contain personal data, so list only endpoints whose payloads you are allowed to record.

## Browser reports

Content Security Policy violations are sent as `maple.browser.csp_violation` warning logs, with the directive, the blocked URL and, when known, the script location. Set `reporting.browserReports: true` to also get the browser's deprecation and intervention reports. These are logs, not errors, so they never open an issue. Each kind of report is sent once per page.

## Navigation and data-loading spans

Three calls turn one click into one trace: a navigation span, the data-loading spans under it, the `fetch()` spans those make, and the backend spans behind them. Call them from your router's hooks. The [frontend guides](/docs/frontend) show where for each framework.

```ts
// the router starts a navigation
MapleBrowser.startNavigation(location.pathname)

// a route's data loading
const project = await MapleBrowser.traced("loader /projects/:id", () => fetchProject(id), {
	isFailure: (error) => !isRedirect(error), // control-flow throws aren't failures
})

// the new route is ready: pass its template, not the concrete URL
MapleBrowser.endNavigation("/projects/:id")
```

- `startNavigation(path)` opens a `pageload` span on the first call and a `navigate` span on every later one, with the path as `url.path`. A navigation still open is ended with `app.navigation.interrupted: true`. The page load joins the server render's trace when the document response has a `Server-Timing: traceparent;desc="…"` entry or the page a `<meta name="traceparent">` tag.
- The page load starts at the browser's navigation start, and once the page has loaded it gets child spans for fetching the HTML (`documentFetch`, with `dns`, `connect`, `request` and `response` under it), `domProcessing` and `loadEvent`.
- `endNavigation(route?)` renames the span to `navigate <route>` (or `pageload <route>`) and ends it. It does nothing when no navigation is open.
- `traced(name, fn, options?)` runs `fn` in a span under the open navigation and returns its result unchanged. A throw is recorded on the span, marks it `Error`, and is rethrown; `isFailure` returning `false` leaves the span `Ok`. An error `traced` recorded isn't reported again by `captureException` or the global handlers.

Using React Router or TanStack Router? `instrumentReactRouter` and `instrumentTanStackRouter` from `@maple-dev/browser/react` make these calls for you. See [React](#react).

The browser has no async context: only requests `fn` starts before its first `await` nest under its span. Start independent requests together, with `Promise.all`.

All three do nothing on the server, before `init()`, with tracing disabled or before consent is granted; `traced` then only runs `fn`. A page load that happened before consent isn't traced later: the next navigation is a `navigate` span. Leaving the page or calling `shutdown()` ends an open navigation as interrupted, so it still exports.

## Custom events

`track(name, props)` records a product event against the current session. It appears inline in the session transcript next to the clicks and network calls around it, and counts as a [product event](/docs/product-events/overview).

```ts
MapleBrowser.track("checkout_completed", { plan: "pro", seats: 12 })
```

Names are capped at 128 characters. Props are capped at 32 keys, 64-character keys, 1024-character values and 8 KB in total. Values are converted to strings (`Date` to ISO, objects to JSON; `null`, `undefined` and functions are dropped). Calls before `init()` finishes are queued, and `track()` never throws.

Every session event, page views and `track()` calls alike, carries the person it belongs to: the visitor id, plus the `id` and `groupId` from `identify()`. That is what lets a funnel follow one person from an anonymous marketing visit through sign-in, and lets browser events line up with the same user's [server-side events](/docs/sdks/effect-server#server-side-track). Identity is resolved when the batch is sent, so an `identify()` shortly after `init()` still lands on the first page view. The visitor id is empty when the visitor cookie is off (consent not granted, Global Privacy Control, `persistVisitorId: false`). Events from older SDK builds arrive with no identity.

## Linking a marketing site to your app

The visitor id is stored in **both** localStorage and a cookie scoped to your registered domain, so `example.com` and `app.example.com` resolve to the same visitor. Initialize the SDK on both and an anonymous visit links to the signed-in sessions it later becomes. On a replay, the link next to **Visitor ID** lists every session from that visitor, which shows the whole journey.

The **session** id is not shared: each origin keeps its own session, and the visitor id is the join key between them.

The SDK finds the cookie domain by probing, without a public-suffix list. Override it when the default is wrong:

```ts
MapleBrowser.init({
	// …
	privacy: {
		crossSubdomainCookie: true, // default; false keeps the cookie host-only
		cookieDomain: "example.com", // explicit override
	},
})
```

## Consent

Capture is on by default. Set `privacy.requireConsent` to hold everything until the user agrees, then call `setConsent()`:

```ts
MapleBrowser.init({ ingestKey, serviceName, privacy: { requireConsent: true } })

// once the banner is accepted
MapleBrowser.setConsent(true)
```

Revoking stops capture without uploading what is buffered, and a later grant starts a new session. Global Privacy Control is honored regardless of `requireConsent`. It suppresses the persistent visitor id (the one cross-session identifier the SDK stores) and leaves session-scoped capture alone. `doNotTrack` is ignored unless you set `privacy.respectDoNotTrack`. `privacy.persistVisitorId: false` turns the visitor id off entirely and deletes any already stored. `privacy.captureUserEmail: false` keeps the email from `identify()` out of Maple.

### Memory-only mode

For sites that must not store anything on the device, set `privacy.persistence: "memory"`:

```ts
MapleBrowser.init({ ingestKey, serviceName, privacy: { persistence: "memory" } })
```

The SDK then writes no cookies, localStorage, sessionStorage or IndexedDB. Sessions, replays, traces, errors and `track()` events still work within a page load. What you give up:

- **Each page load is a new session.** A reload or a full-page navigation starts a new session and a new replay. Single-page apps keep one session across client-side navigation.
- **No visitor id.** Unique, new and returning visitors are not counted, and an anonymous visit cannot be linked to the account it becomes. `identify()` still attaches the user to the current session.
- **Nothing is kept for later.** `transport.offline` stores no failed batches, and the last replay chunk of a closing page is sent only if the upload finishes before the page unloads.

Turning it on also deletes the visitor id and visit cookies from an earlier configuration. The setting only tightens: if two SDKs on a page disagree, memory-only wins.

## Privacy and masking

`maskAllInputs` is **on by default**, so every `<input>` value is masked before it leaves the browser. Set `maskAllText: true` to also mask all rendered text.

To block specific elements or subtrees, use rrweb's attributes:

- `data-rr-block` attribute, or the `.rr-block` class: block an element and its subtree. It is recorded as a placeholder.
- `.rr-ignore` class: ignore input events on an element.

```html
<div class="rr-block">
	<!-- never captured in the replay -->
	<CreditCardForm />
</div>
```

### Redacting URLs

The SDK already replaces the values of credential-shaped query and fragment parameters (`token`, `code`, `access_token`, `password` and similar) in every URL it sends. To redact more, pass `privacy.sanitizeUrl`. It runs after the built-in redaction, on session entry and exit URLs, event rows, network events, replay metadata and span attributes:

```ts
MapleBrowser.init({
	ingestKey: "maple_pk_...",
	serviceName: "acme-web",
	privacy: {
		// hide invite codes in paths like /invite/abc123
		sanitizeUrl: (url) => url.replace(/\/invite\/[^/?#]+/, "/invite/:code"),
	},
})
```

## Sampling

To record only a fraction of sessions, set `replay.sampleRate` between `0` and `1`. For example, `0.1` records about 10% of sessions. Tracing is not affected.

```ts
MapleBrowser.init({
	ingestKey: "maple_pk_...",
	serviceName: "acme-web",
	replay: { sampleRate: 0.1 },
})
```

A value outside `0` to `1` is clamped, with a console warning.

`tracing.sampleRate` samples traces the same way. The decision is made once per session, so a sampled session keeps all of its traces and its replay never links to a missing one. Errors reported as their own spans (uncaught errors, unhandled rejections, `captureException` and failures inside `traced()`) are always sent. Failed request spans follow the session's sampling like any other span. Sampled traces carry their sampling rate so request counts in Maple stay accurate.

### Replay on error

`replay.onErrorSampleRate` covers the sessions `replay.sampleRate` leaves out. Those sessions record into memory only, keeping roughly the last minute, and upload nothing. When an error is recorded, the buffered minute is uploaded and the rest of the session is recorded normally, including its later page loads.

```ts
replay: { sampleRate: 0.05, onErrorSampleRate: 1 } // 5% of sessions, plus every session with an error
```

These sessions download the recorder like recorded ones, and keep up to 4 MB in memory.

## Offline

The SDK retries a failed send for a few seconds. With `transport: { offline: true }`, a batch of spans or logs that still could not be sent is kept in IndexedDB and sent again when the browser is back online or on the next page load. At most 100 batches are kept, for up to 24 hours, and revoking consent deletes them.

## Session size limit

A single session records at most **1 GiB** of decompressed replay data. Past that, the recording is cut at a chunk boundary: earlier chunks stay playable, later ones are dropped, and the session is still listed with its metadata and linked traces.

Normal sessions are far below this. The limit stops a runaway recording, which in practice means canvas capture, unmasked media, or a page whose DOM changes continuously. `replay.sampleRate` does not help here, because it drops whole sessions. Mask or block the noisy subtree instead.

## Framework examples

### Plain HTML

```html
<script type="module">
	import { MapleBrowser } from "https://esm.sh/@maple-dev/browser"

	MapleBrowser.init({
		ingestKey: "maple_pk_...",
		serviceName: "acme-web",
	})
</script>
```

### React and Vite

Initialize at the top of your client entry point so it runs once before the app renders:

```ts
// src/maple.ts
import { MapleBrowser } from "@maple-dev/browser"

MapleBrowser.init({
	ingestKey: import.meta.env.VITE_MAPLE_INGEST_KEY,
	serviceName: "acme-web",
	environment: import.meta.env.MODE,
})
```

```ts
// src/main.tsx
import "./maple" // import first, before rendering
import { createRoot } from "react-dom/client"
import { App } from "./App"

createRoot(document.getElementById("root")!).render(<App />)
```

### React

`@maple-dev/browser/react` adds an error boundary, a handler for React 19's root error options, and router integrations that record navigations for you:

```tsx
import { createBrowserRouter, RouterProvider } from "react-router"
import { instrumentReactRouter, mapleReactErrorHandler, MapleErrorBoundary } from "@maple-dev/browser/react"

const router = createBrowserRouter(routes)
instrumentReactRouter(router) // or instrumentTanStackRouter(router)

createRoot(document.getElementById("root")!, {
	onCaughtError: mapleReactErrorHandler(),
	onUncaughtError: mapleReactErrorHandler(),
}).render(
	<MapleErrorBoundary fallback={({ reset }) => <button onClick={reset}>Try again</button>}>
		<RouterProvider router={router} />
	</MapleErrorBoundary>,
)
```

- `MapleErrorBoundary` reports a render error once, with the component stack, and renders `fallback`.
- `instrumentReactRouter` works with data routers (`createBrowserRouter`). A navigation starts when the router starts loading and ends when its loaders finish, named after the route (`navigate /projects/:id`).
- `instrumentTanStackRouter` does the same, named after the route's full path (`navigate /projects/$projectId`). Changing only search params is not a navigation.

With a router integration, don't call `startNavigation` and `endNavigation` yourself.

### Next.js

Next.js exposes `NEXT_PUBLIC_*` variables through `process.env`, not `import.meta.env`. Initialize from a client component and render it in the root layout:

```tsx
// app/maple.tsx
"use client"

import { MapleBrowser } from "@maple-dev/browser"

// init() is a no-op during server rendering, so module scope is safe here.
MapleBrowser.init({
	ingestKey: process.env.NEXT_PUBLIC_MAPLE_INGEST_KEY!,
	serviceName: "acme-web",
	environment: process.env.NODE_ENV,
})

export function Maple() {
	return null
}
```

```tsx
// app/layout.tsx
import { Maple } from "./maple"

export default function RootLayout({ children }: { children: React.ReactNode }) {
	return (
		<html lang="en">
			<body>
				<Maple />
				{children}
			</body>
		</html>
	)
}
```

## Verify

Load a page with the SDK installed, click around for a few seconds, then leave the tab.

1. Open **Traces** and filter by your `serviceName`. The page's `fetch()` spans appear within about a minute.
2. Open **Replays**. The session appears in the list. Open it to play the recording.
3. If you set `propagateTraceHeaderCorsUrls`, open a `fetch()` span in **Traces**. Your backend's span is in the same trace.

## Troubleshooting

- **Nothing arrives, and requests to the ingest endpoint return `401`.** The ingest key is wrong, or it belongs to the other region. EU keys need `region: "eu"`.
- **No replay, but traces arrive.** Check `replay.enabled` and `replay.sampleRate`, and whether `requireConsent` is on without a `setConsent(true)` call.
- **Cross-origin API calls fail after adding `propagateTraceHeaderCorsUrls`.** The API's CORS preflight does not allow `traceparent`. Add it to `Access-Control-Allow-Headers`.
- **Browser and backend spans are in separate traces.** The API origin is not in `propagateTraceHeaderCorsUrls`, or the backend does not read `traceparent`.
- **Each error appears twice.** Another tool on the page also handles uncaught errors and reports them to Maple. Set `tracing.captureErrors: false` so only one of them does.
- **Duplicate network spans.** Another tracer, such as the Effect client SDK, also instruments `fetch`. Set `tracing.instrumentFetch: false`.

## Notes

- Replay recordings are stored as compressed blobs. Only small, queryable metadata is indexed, and playback streams the blobs through signed URLs.
- The SDK is browser-only and best-effort. Telemetry network failures never surface to your application.

## Next steps

- [Replays](/docs/session-replay/replays): find and play back sessions.
- [Product events](/docs/product-events/overview): funnels on `track()` and server-side events.
- [Web analytics](/docs/product-events/web-analytics): visitors, pages and referrers from the same SDK.

---

# Replays

> Session Replay · https://maple.dev/docs/session-replay/replays

Find a browser session on the Session Replays page, play it back next to its events and backend traces, and move between a trace and the replay that produced it.

The **Replays** page (**Session Replays** in its header) lists the browser sessions recorded by the [Browser SDK](/docs/session-replay/browser-sdk). Open a session to watch what the user saw, with the clicks, console messages, network requests, errors and backend traces from that session beside the player.

## Prerequisites

The [Browser SDK](/docs/session-replay/browser-sdk) initialized with replay enabled (the default). Sessions sampled out by `replay.sampleRate` are not recorded.

## Find a session

The toolbar shows how many sessions match and how many are **live** (still recording). Use the time range picker to change the window, and **Search by URL…** to find sessions that visited a page. The **Analytics** button opens [Web analytics](/docs/product-events/web-analytics) for the same window.

The sidebar narrows the list:

| Filter                   | What it matches                                                                              |
| ------------------------ | -------------------------------------------------------------------------------------------- |
| Session length           | Duration. **Bounced** is under 10 seconds. The other presets come from your own p50 and p95. |
| Active time              | Time the user was active. **Idle** is under 5 seconds, **Engaged** over 30 seconds.          |
| Page visited             | Sessions that visited a page.                                                                |
| Service                  | The `serviceName` the SDK was initialized with.                                              |
| Browser, Device, Country | The visitor's environment.                                                                   |
| Group                    | The company or team passed to `identify()`. Shown once any session has one.                  |
| Identity                 | **Name or email…** and **User ID…**, from `identify()`.                                      |

Sessions with errors are marked in the list.

## Watch a session

Click a session to open it. The player rebuilds the page as the user saw it.

- **Space** plays and pauses. The left and right arrow keys seek 5 seconds.
- Playback speed is 0.5×, 1×, 2×, 4× or 8×.
- **Skip idle** jumps over periods with no activity.

The rail on the right has three tabs:

- **Events.** Every recorded event in order. Filter it to **Product events**, **Console messages**, **Network requests** or **Errors**. Click an event to seek the player to that moment. A network request that carried a trace has an **Open trace** link.
- **Traces.** The backend traces observed during the session. Click one to seek the player to it, or open it in the trace view.
- **Session.** The session's metadata: identity (name, email, user id, group, traits), visitor id, entry and exit page, referrer and UTM source and campaign, browser, operating system, device, country, service, and session id.

From the **Session** tab, the links next to **User ID**, **Group** and **Visitor ID** list every session from that user, group or visitor.

## Replay and trace links

The SDK tags every span and replay event with the same `session.id`, so Maple links the two both ways:

- **From a replay to a trace.** Use the **Traces** tab, or **Open trace** on a network event.
- **From a trace to a replay.** When a browser session observed a trace, the trace page shows **View Session Replay**. It opens the replay at that session.

For a backend span to appear in the same trace as the browser request, the request must carry the `traceparent` header. Same-origin requests always do. For a cross-origin API, see [Connect browser and backend traces](/docs/session-replay/browser-sdk#connect-browser-and-backend-traces).

## Privacy

Masking happens in the browser, before anything is sent. A replay never contains what the SDK masked.

- Every `<input>` value is masked by default (`privacy.maskAllInputs`).
- `privacy.maskAllText: true` masks all rendered text.
- Elements with the `rr-block` class or `data-rr-block` attribute are recorded as placeholders.
- URLs are redacted: the values of credential-shaped parameters such as `token` or `password` are replaced, and `privacy.sanitizeUrl` can rewrite the rest.

See [Privacy and masking](/docs/session-replay/browser-sdk#privacy-and-masking) and [Consent](/docs/session-replay/browser-sdk#consent) for the options.

## Verify

Load a page with the SDK installed, click around, then close the tab. Open **Replays**. The session is listed within about a minute, and it plays back when you open it.

## Troubleshooting

- **"No sessions recorded yet".** Check that the SDK initializes on the page, that `replay.enabled` is not `false`, and that consent was granted if `privacy.requireConsent` is on.
- **Some sessions are missing.** `replay.sampleRate` below `1` records only that fraction of sessions.
- **The Traces tab is empty.** The session made no traced requests, or the backend traces are in a different trace because the request did not carry `traceparent`.
- **A replay stops partway.** A session records at most 1 GiB of replay data. See [Session size limit](/docs/session-replay/browser-sdk#session-size-limit).

## Next steps

- [Browser SDK](/docs/session-replay/browser-sdk)
- [Web analytics](/docs/product-events/web-analytics)
- [Traces](/docs/explore/traces)

---

# Product events

> Product Events · https://maple.dev/docs/product-events/overview

Track the steps that matter, like signup, checkout and plan started, from the browser, a span you already emit, or any backend, and build funnels, drop-off and path analysis on them.

A product event is a named thing a person did: `signup_completed`, `checkout_completed`, `plan_started`. Maple stores every one in a single `product_events` table, keyed to a person by `user_id`, `group_id` and `visitor_id`, and keeps it for 365 days. Funnels, drop-off and paths in the dashboard query builder run on that table, and **Web Analytics** filters and breaks down browser events.

Product events do not require session replay or the browser SDK. Any service that emits OpenTelemetry spans, or can make an HTTP request, can send them.

## Three ways to send one

| Source            | How                                                              | Use when                                                                 |
| ----------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------ |
| Browser           | `track(name, props)` in `@maple-dev/browser` or the Effect SDK   | The step is a user action in the page, and you want it in the replay.    |
| Span annotation   | `maple.product_event.name` on a span you already emit            | The step already runs inside a traced request or job.                    |
| `POST /v1/events` | NDJSON to the ingest gateway, or `MapleEvents` in the Effect SDK | There is no span: webhooks, batch scripts, mobile apps, other languages. |

All three key on the same `user_id`, `group_id` and `visitor_id`, so one funnel can mix them.

### Browser

```ts
import { MapleBrowser } from "@maple-dev/browser"

MapleBrowser.track("checkout_completed", { plan: "pro", seats: 12 })
```

The event appears in the session transcript next to the clicks and requests around it. Calls made before the SDK finishes initializing are queued. See the [Browser SDK](/docs/session-replay/browser-sdk#custom-events).

### From a span

```ts
span.setAttributes({
	"maple.product_event.name": "plan_started",
	"maple.product_event.user_id": user.id,
})
```

Every other attribute on that span becomes an event property, and the event links back to the trace that performed it. See [Product events from traces](/docs/product-events/from-traces).

### Direct

```bash
curl -X POST https://ingest.maple.dev/v1/events \
  -H "Authorization: Bearer YOUR_INGEST_KEY" \
  -H "Content-Type: application/x-ndjson" \
  --data-binary $'{"name":"plan_started","user_id":"user_123","attributes":{"plan":"startup"}}\n'
```

See the [Product events API](/docs/product-events/api) for the full field list, and the [Effect server SDK](/docs/sdks/effect-server#server-side-track) for a batching client.

## Identity

All three paths carry the same ids. `user_id` is whatever you pass to `identify()` in the browser and to your backend calls. `visitor_id` is the anonymous browser cookie, shared across your subdomains. Maple links a visitor to the user it later becomes. A funnel counted by person therefore joins an anonymous visit on the marketing site, the signed-in app session, and a server-side event from a webhook handler into one person.

`group_id` is the account, workspace or org the event belongs to. It is a filter and breakdown dimension. Funnels count persons, visitors, users or sessions, never groups. An event that carries only a `group_id` is stored and queryable, but it cannot be linked to a person.

## Naming

Names are 1 to 128 bytes. Names starting with `$` are reserved for Maple's SDKs. Use stable, lowercase, past-tense names like `plan_started`. The name is the funnel step key, so renaming an event splits its history.

## Verify

Send one event, then check within a minute:

- **Browser `track()`**: open **Web Analytics**. The event is listed on the **Events** card, and in the **Event** section of the sidebar.
- **Span annotation**: open the trace that produced it. The trace page lists it under **Product events**.
- **Any source**: open a dashboard, add a chart, and pick **Product events** as the data source. The event name is available to count and filter.

For `POST /v1/events`, a `200` response with `{"accepted": 1}` means the row was queued. `{"accepted": 0}` means the row was dropped, usually for a bad `name`, `source` or `timestamp`.

## Next steps

- [Product events from traces](/docs/product-events/from-traces)
- [Product events API](/docs/product-events/api)
- [Web analytics](/docs/product-events/web-analytics)

---

# Product events from traces

> Product Events · https://maple.dev/docs/product-events/from-traces

Turn a span you already emit into a product event by annotating it with maple.product_event.* attributes. No second SDK call, and each event links back to the request that performed it.

Browser `track()` calls and `POST /v1/events` are explicit calls made for analytics. Most conversion steps already happen inside a traced code path: the handler that creates the subscription, the job that finishes the import. Annotate that span and Maple writes it into the same `product_events` table as everything else, linked to the trace that did the work.

For when to use this path instead of the other two, see [Three ways to send one](/docs/product-events/overview#three-ways-to-send-one).

## Annotate the span

Set one attribute on the span that performs the step. Its presence is the switch. Nothing else has to be declared.

```ts
span.setAttributes({
	"maple.product_event.name": "plan_started", // required
	"maple.product_event.user_id": user.id, // optional identity
	"maple.product_event.group_id": org.id,
	"maple.product_event.visitor_id": anonId,
	"maple.product_event.url": req.url, // optional page context
})
```

With the Effect SDK the same keys go through `Effect.annotateCurrentSpan`:

```ts
const startPlan = Effect.gen(function* () {
	yield* Effect.annotateCurrentSpan({
		"maple.product_event.name": "plan_started",
		"maple.product_event.user_id": user.id,
		"maple.product_event.group_id": org.id,
	})
}).pipe(Effect.withSpan("start-plan"))
```

Any OpenTelemetry SDK works, in any language. The contract is a set of attribute keys on the span.

| Attribute                        | Notes                                                                                                                                                                                                                                  |
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `maple.product_event.name`       | **Required.** The event name. Names starting with `$` are reserved.                                                                                                                                                                    |
| `maple.product_event.user_id`    | The signed-in user, matching what you pass to `identify()` in the browser.                                                                                                                                                             |
| `maple.product_event.group_id`   | Account, workspace or org id.                                                                                                                                                                                                          |
| `maple.product_event.visitor_id` | The browser visitor cookie value, when the backend has it.                                                                                                                                                                             |
| `maple.product_event.url`        | Page the step happened on. `host` and `page_path` are derived from it.                                                                                                                                                                 |
| `session.id`                     | OpenTelemetry's own session key. When the span carries one, it becomes the event's session id, so a browser-originated trace joins its replay session. Without it the event has no session and takes no part in session-keyed funnels. |

The `service_name` of the event is the span's service, and its timestamp is the span's start time.

## Event properties

**Every other attribute on the span becomes an event property by default.** Whatever the code already sets (`plan`, `order.total`, the HTTP and database semconv keys) lands in the event's attributes and is available for funnel breakdowns and filters. The `maple.product_event.*` control keys are stripped, because they are promoted to their own columns.

Two optional attributes narrow or replace that default:

```ts
"maple.product_event.include": "plan,seats"   // copy ONLY these span keys
"maple.product_event.prop.plan": "pro"        // set a property explicitly; wins on collision
```

| `include`      | `prop.*` | Resulting properties                                              |
| -------------- | -------- | ----------------------------------------------------------------- |
| absent         | absent   | every span attribute                                              |
| absent         | set      | every span attribute, with the `prop.*` values overriding on ties |
| `"plan,seats"` | absent   | only `plan` and `seats`                                           |
| `""`           | set      | only the `prop.*` values                                          |

`include` switches on key presence, so an empty string means "no span attributes". A server span's attribute map is mostly semconv keys, and product events are kept for 365 days, longer than raw traces (see [Retention](/docs/reference/retention)). Set `include` once you know which properties you need.

## Where the link shows up

`product_events` rows from this path carry the trace and span id.

- The **trace detail page** lists the product events a trace produced, under **Product events**.
- On **Web Analytics**, setting the **Event** filter to an event that annotated spans produce shows a **Traces behind** panel with sample traces for that event. A drop-off in a funnel step is one click from the requests behind it.

## When events start

Maple turns annotated spans into product events as the spans are ingested. Only spans ingested after you deploy the annotation become events. Spans already stored are not converted, so a new annotation has no history before its deploy.

## Why an attribute

A product event has to be emitted by the code path that performed the step, at the moment it performed it. An attribute marks every trace that code path produces, and it is reviewed in your own diff like any other code change. The span is the record and the product event is its projection, so there is no second store to keep in sync.

## Verify

Deploy the annotation, trigger the step once, and open the resulting trace in **Traces**. Within a minute, the trace page lists the event under **Product events**. The event is then available to funnels and charts that use **Product events** as their data source.

If the trace appears but no product event does:

- Check that `maple.product_event.name` is set on a span in the trace, and that its value does not start with `$`.
- Check that the trace was ingested after the annotation was deployed.

---

# Product events API

> Product Events · https://maple.dev/docs/product-events/api

Post product events from a backend or mobile app to POST /v1/events on the Maple ingest gateway. The raw NDJSON contract every server-side track() call uses.

Browser page views and `track()` calls reach Maple through the session SDKs. Everything else is posted directly to the ingest gateway: a `signup_completed` from a webhook handler, a `plan_started` from your billing worker, a screen view from a native app. Rows land in the same `product_events` table as the browser events, so one funnel can span the marketing site, the app and the backend.

## Endpoint

```
POST https://ingest.maple.dev/v1/events
Authorization: Bearer YOUR_INGEST_KEY        # or X-Maple-Ingest-Key: YOUR_INGEST_KEY
Content-Type: application/x-ndjson
```

EU organizations use `https://ingest.eu.maple.dev`. The body is **NDJSON**: one JSON object per line, any number of lines. The organization is resolved from the ingest key. An `org_id` in the body is ignored.

```json
{"name":"signup_completed","user_id":"user_01H…","service_name":"maple-api"}
{"name":"plan_started","user_id":"user_01H…","group_id":"org_01H…","attributes":{"plan":"startup"}}
{"name":"$screen","source":"mobile","visitor_id":"install-8f3…","page_path":"Checkout"}
```

## Fields

| Field          | Type   | Notes                                                                                                                                                           |
| -------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`         | string | **Required.** 1–128 bytes. Names starting with `$` are reserved for Maple's SDKs and dropped, except `$screen` (mobile screen view, stored as `Kind = screen`). |
| `timestamp`    | string | RFC 3339 (`2026-08-17T10:15:30.123Z`) or `YYYY-MM-DD HH:MM:SS[.fff]` (UTC). Defaults to the time the gateway received the batch. Stored as UTC.                 |
| `source`       | string | `server` (default) or `mobile`. `browser` is reserved for the SDKs; other values drop the row.                                                                  |
| `visitor_id`   | string | Anonymous or device id: the browser SDK cookie value, or a persistent mobile install id. ≤ 256 bytes.                                                           |
| `user_id`      | string | Your user id after sign-in, matching what you pass to `identify()`. ≤ 256 bytes.                                                                                |
| `group_id`     | string | Account / workspace / org id. ≤ 256 bytes.                                                                                                                      |
| `session_id`   | string | Optional link to a browser or mobile session. ≤ 256 bytes.                                                                                                      |
| `service_name` | string | The emitting service (`maple-api`, `acme-ios`). ≤ 128 bytes.                                                                                                    |
| `url`          | string | Optional. `host` (lowercase) and `page_path` (pathname only) are derived from it.                                                                               |
| `page_path`    | string | Optional explicit path; overrides the one derived from `url`. Mobile `$screen` events put the screen name here.                                                 |
| `attributes`   | object | Optional properties. ≤ 32 keys, key ≤ 64 bytes, value ≤ 1024 bytes; non-string values are stringified.                                                          |

Over-long strings are truncated at the caps above. Unknown fields are discarded.

## Responses

| Status | Meaning                                                                                                                                |
| ------ | -------------------------------------------------------------------------------------------------------------------------------------- |
| `200`  | `{"accepted": <n>}`: rows durably queued. Malformed rows (bad `name`, `source`, `timestamp`) are dropped individually and not counted. |
| `400`  | A line is not valid JSON, or not a JSON object. The whole batch is rejected.                                                           |
| `401`  | Missing or invalid ingest key.                                                                                                         |
| `402`  | The organization is out of quota for product events (`product_events` is metered per event, separately from browser sessions).         |
| `503`  | Storage temporarily unavailable. Retry with backoff.                                                                                   |

Product events are metered as their own unit, one per accepted row. Browser `track()` calls count in the same unit.

## Example

```bash
curl -X POST https://ingest.maple.dev/v1/events \
  -H "Authorization: Bearer YOUR_INGEST_KEY" \
  -H "Content-Type: application/x-ndjson" \
  --data-binary $'{"name":"plan_started","user_id":"user_123","attributes":{"plan":"startup"}}\n'
```

## Verify

Send the example above. A `200` with `{"accepted": 1}` means the row was queued. Within a minute, the event is available in dashboard charts that use **Product events** as their data source. `{"accepted": 0}` means the row was dropped: check `name`, `source` and `timestamp` against the table above.

---

# Web analytics

> Product Events · https://maple.dev/docs/product-events/web-analytics

What the Web Analytics page shows: visitors, sessions, page views, referrers, devices, countries and custom events from the Maple browser SDK, and what data each part needs.

**Web Analytics** shows who visited your sites, which pages they read, and where they came from. It reads the sessions, page views and `track()` events that the Maple browser SDK already sends for session replay. There is no separate analytics script.

## Prerequisites

The [Browser SDK](/docs/session-replay/browser-sdk) (`@maple-dev/browser`), or the browser entry point of `@maple-dev/effect-sdk`, initialized on every site you want to measure. Server-side product events from spans or `POST /v1/events` do not appear on this page. Query them with a dashboard chart instead.

## What the page shows

The page opens on the last 7 days. The time range picker, the filters and the **Replays** button (which opens the same window on the Replays page) are in the header.

<figure class="shot">
  <img src="/screenshots/docs/web-analytics-01-overview.webp" alt="The Web Analytics page over the last 7 days: a live visitors badge, a note that 17% of sessions were bots and are excluded, eight headline metric tiles with sparklines, and a unique visitors and page views chart." loading="lazy" />
  <figcaption>Web Analytics over the last 7 days. Bot sessions are excluded from every number.</figcaption>
</figure>

### Live visitors

The badge left of the time range counts visitors active in the last few minutes. It ignores the selected range and refreshes every 15 seconds.

### Headline metrics

| Metric          | What it counts                                                                                      |
| --------------- | --------------------------------------------------------------------------------------------------- |
| Unique visitors | Distinct browsers, by visitor id. If some sessions have no visitor id, the tile says what share do. |
| Sessions        | Browser sessions in the window.                                                                     |
| Page views      | Page views across every session.                                                                    |
| Pages / session | Page views divided by sessions.                                                                     |
| Bounce rate     | Share of sessions that bounced, over sessions that report a visitor id.                             |
| Avg. session    | Average duration of sessions that ended.                                                            |
| New visitors    | Sessions that were a visitor's first ever.                                                          |
| Returning       | Sessions from visitors seen before this window.                                                     |

Pick a metric to chart it over the window.

### Breakdowns

Five cards break the traffic down, each with its own tabs. Click any row to filter the whole page by it.

- **Referrers**, **UTM source**, **Medium** and **Campaign**: where visitors came from.
- **Pages**, **Entries** and **Exits**: what they read, and where sessions started and ended.
- **Devices**, **Browsers** and **OS**.
- **Countries**, **Languages** and **Sites**.
- **Events**: custom events from `track()`, with the number of sessions that fired each.

**Sites** lists each host that sends data, so one organization can measure a marketing site and an app side by side. With more than one site, **Pages** shows each page's site icon.

### Filters

The sidebar filters by **Traffic**, **Site**, **Page**, **Event**, **Referrer**, **UTM source**, **UTM medium**, **UTM campaign**, **Visitor** (**New** or **Returning**), **Country** and **Language**. Active filters show as chips under the page title. **Clear all** removes them and keeps the time range.

**Traffic** chooses **Humans**, **Bots** or both. Only **Humans** is checked when the page opens. When more than 5% of recorded sessions in the window came from crawlers or automated browsers, a notice says how many and whether they are counted. It only covers bots that ran the browser SDK. Fetchers that do not run JavaScript never create a session.

When the **Event** filter names an event that annotated spans also produce, a **Traces behind** panel lists sample traces for it. See [Product events from traces](/docs/product-events/from-traces).

## What each part needs

| To see                                                 | You need                                                                                                                                                                                                                                    |
| ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Sessions, page views, pages, devices                   | The browser SDK initialized on the page.                                                                                                                                                                                                    |
| Unique, new and returning visitors, bounce rate        | The persistent visitor id. It is on by default. It is off when `privacy.persistVisitorId` is `false`, when `privacy.persistence` is `"memory"`, when consent is required and not granted, or when the browser sends Global Privacy Control. |
| One visitor across `example.com` and `app.example.com` | `privacy.crossSubdomainCookie` left on (the default) and the SDK on both sites.                                                                                                                                                             |
| Events                                                 | `MapleBrowser.track(name, props)` calls.                                                                                                                                                                                                    |
| UTM source, medium and campaign                        | `utm_source`, `utm_medium` and `utm_campaign` on the landing URL.                                                                                                                                                                           |
| Countries                                              | Country is resolved at the ingest gateway. When it is not available the tab says so, and it is never backfilled.                                                                                                                            |

See [Consent](/docs/session-replay/browser-sdk#consent) for how the SDK handles consent and privacy signals.

## Verify

Open your site in a browser with the SDK installed, visit two pages, then open **Web Analytics**. Within a minute, the live badge counts you, **Page views** includes your two pages, and both pages are listed under **Pages**.

## Troubleshooting

- **Page views appear but Unique visitors is 0.** No session carried a visitor id. Check `privacy.persistVisitorId`, consent, and Global Privacy Control.
- **Numbers lower than another analytics tool.** Bots are excluded by default, and sessions only start once the SDK runs. Check the **Traffic** filter and the bot notice.
- **Countries is empty.** Country is resolved at ingest, and only for traffic received after it was available.

## Next steps

- [Browser SDK](/docs/session-replay/browser-sdk)
- [Replays](/docs/session-replay/replays)
- [Product events](/docs/product-events/overview)

---

# Agent Sessions

> Agent Sessions · https://maple.dev/docs/agent-sessions/overview

Agent Sessions groups the OpenTelemetry traces of one AI agent conversation into a single view of its turns, model calls, tool calls, tokens, cost and failures.

Each user message in a conversation is usually its own trace. **Agent Sessions** groups those traces into one conversation and shows it turn by turn. To send your agent's traces, pick your framework in [Trace your AI agent](/docs/agent-tracing).

## Sessions, turns and calls

| Level          | What it is                                                           | Where it comes from                                                             |
| -------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| **Session**    | One conversation, from the first message to the last.                | Every trace that carries the same session id, such as `gen_ai.conversation.id`. |
| **Turn**       | One user message and everything the agent did to answer it.          | Usually one trace, rooted at an `invoke_agent` span.                            |
| **Model call** | One request to an LLM: the prompt, the reply, tokens, finish reason. | A `chat` (or `generate_content`, `text_completion`) span.                       |
| **Tool call**  | One function the model asked to run, with its arguments and result.  | An `execute_tool` span.                                                         |

Sub-agents show up inside a turn as their own lane, labeled with their `gen_ai.agent.name`. A background job with no user is also a session, usually one trace long.

## Read a session

The example below is a support agent where the customer asks to change a delivery address and ends up canceling the order.

<figure class="shot">
  <img src="/screenshots/docs/agent-sessions-02-overview.webp" alt="A session's overview page: a Needs attention check naming a rejected update_shipping_address call in turn 2, the passed and unchecked checks, a tools table with calls, failures and a timeline, and a right column with where the time went, cost by model and token buckets." loading="lazy" />
  <figcaption>The overview. The failed tool call leads the page: <code>update_shipping_address</code> rejected a call in turn 2 with <code>Order region is US</code>, and the check says what to tighten.</figcaption>
</figure>

The **overview** splits the session's wall clock into model time, tool time and idle time, and totals cost and tokens per model.

Below that is the verdict (completed, completed with warnings, or failed, with the span that ended it) and the checks behind it, failing ones first. A check that needs data your instrumentation doesn't record says it was skipped and what to capture.

<figure class="shot">
  <img src="/screenshots/docs/agent-sessions-03-transcript.webp" alt="The transcript view of a session: turns headed by their tokens, cost, start time and duration, system prompts, user and assistant messages in sequence with each reply labeled by its model, and a tool call row with its latency and payload sizes." loading="lazy" />
  <figcaption>The transcript. Each turn carries its tokens and cost, each reply its model; tool calls sit where the model made them.</figcaption>
</figure>

The **transcript** is the conversation as the model saw it: system instructions, user and assistant messages, and tool calls with arguments and results. It needs message content on your spans, which most instrumentations leave off by default; each framework guide shows the switch.

<figure class="shot">
  <img src="/screenshots/docs/agent-sessions-05-trace.webp" alt="The trace view of a session: three turns, each with an invoke_agent span, chat spans labeled with their model and token counts, and execute_tool spans, on a time axis with the idle gaps between turns removed. One tool span is marked with its error type." loading="lazy" />
  <figcaption>The trace. Spans grouped by turn, with 2 minutes 10 seconds of idle time cut from the axis and the failed tool span flagged.</figcaption>
</figure>

The **trace** view puts every span of every turn on one time axis, with the idle time between turns removed.

<figure class="shot">
  <img src="/screenshots/docs/agent-sessions-01-list.webp" alt="The Agent Sessions list in Maple, one row per session with services, model, duration, LLM call and tool call counts, tokens, cost, errors and start time, and a filter sidebar on the left." loading="lazy" />
  <figcaption>The list. One row per conversation, filterable by framework, service, environment, model, agent and tool.</figcaption>
</figure>

The **list** has one row per session with its model, duration, call counts, tokens, cost and errors. Sort by cost to find expensive conversations, or filter to sessions that called a given tool.

## Find the tools that fail

The **Tools** tab covers every tool call across all sessions.

<figure class="shot">
  <img src="/screenshots/docs/agent-sessions-04-tools.webp" alt="The Tools tab: a metric strip with tool calls, sessions, error rate and duration, a chart of calls over time, and a table ranking each tool by calls, p50, p90, p95, error rate, errors, sessions and last call." loading="lazy" />
  <figcaption><strong>Which tool is failing?</strong> Every tool ranked by calls, latency percentiles and error rate. <strong>Failing only</strong> keeps just the ones that have failed.</figcaption>
</figure>

<figure class="shot">
  <img src="/screenshots/docs/agent-sessions-08-tool-detail.webp" alt="A tool's detail page: four charts for tool calls, error rate, duration percentiles and calls per session, then an errors table with one row per error showing trend, share, count, sessions and last seen." loading="lazy" />
  <figcaption><strong>Why?</strong> A tool's page charts its calls, error rate, duration and calls per session, then groups its failures with a trend and the sessions they hit.</figcaption>
</figure>

<figure class="shot">
  <img src="/screenshots/docs/agent-sessions-09-tool-error.webp" alt="The error group dialog for a tool: the error type, how many calls and sessions it affected, a failed-calls-per-day chart, a Where it happens panel listing the model and service, a sessions list, and a sample failed call with its JSON arguments and JSON result side by side." loading="lazy" />
  <figcaption><strong>What exactly happened?</strong> An error group opens on the failed calls themselves: arguments on the left, the result on the right, and a link to the trace.</figcaption>
</figure>

A tool call counts as failed when its span has an `ERROR` status or an `error.type` attribute. Failures with the same message, ignoring ids and numbers, form one group.

## Query sessions from your coding agent

The [MCP server](/docs/reference/mcp) exposes the same data. `list_agent_sessions` finds sessions by cost, model, tool or failure, `get_agent_session` returns a session's verdict, checks and turns, `get_agent_session_transcript` returns what it did step by step, and `get_agent_tools_overview` and `get_agent_tool_error` return the tool rankings and failure groups.

## When a session doesn't look right

- **Every message is its own session.** No span carried a session id Maple reads (`gen_ai.conversation.id` for most frameworks, `session.id` for some). The framework's guide shows where to set it.
- **The transcript is empty.** Content capture is off, or the framework writes content only to span events or logs. Content on span attributes must be a JSON string such as a `[{role, parts}]` array.
- **Cost shows as unpriced.** Maple shows cost only when spans carry `gen_ai.usage.cost`, `gen_ai.usage.total_cost` or `llm.cost.total`; it doesn't price tokens itself.
- **Token totals look doubled.** Two instrumentations recorded the same model call, usually the framework's and a provider SDK instrumentor. Turn one off.
- **Nothing appears at all.** Check **Explore → Traces** for the service first. No traces there means the exporter isn't reaching Maple, often a short-lived script that exits before flushing. Traces there but no session means the framework's tracing isn't on.

A framework shown as **Unidentified** still gets sessions, transcripts and tools. If yours has no guide, [the OpenTelemetry GenAI guide](/docs/agent-tracing/opentelemetry) works for any agent, and a sample trace sent to [support@maple.dev](mailto:support@maple.dev) or [Discord](https://discord.gg/BnXjKuwJqP) helps us add one.

---

# Build dashboards

> Dashboards · https://maple.dev/docs/dashboards/build-dashboards

Create a Maple dashboard from scratch or a template, add charts, stats, tables and funnels over traces, logs, metrics and product events, add variables, and share it.

A dashboard is a grid of widgets that query your telemetry over one time range. Each widget is a chart, a single number, a table, a list, a funnel or a note, built with the query builder or with SQL. Dashboards live on the **Dashboards** page.

<figure class="shot">
  <img src="/screenshots/docs/dashboards-02-query-engine.webp" alt="A dashboard named Query Engine: four stat tiles for total queries, error rate, end-to-end P95 and warehouse SQL P95, then line charts of query latency percentiles and bar charts of query throughput by source and kind." loading="lazy" />
  <figcaption>A dashboard mixing stat tiles, latency percentiles and stacked throughput charts.</figcaption>
</figure>

## Create a dashboard

On **Dashboards**, pick one of three starting points:

- **Create Dashboard** opens an empty dashboard named "Untitled Dashboard". Click the name to rename it.
- **Browse templates** opens **Start from a template**. See [Templates](#templates).
- **Import** loads a dashboard from a JSON file exported from Maple.

<figure class="shot">
  <img src="/screenshots/docs/dashboards-01-list.webp" alt="The Dashboards list with a search box, All and Favorites tabs, tag and sort filters, and rows showing each dashboard's name, description, widget count, data sources and last update, next to Browse templates, Import and Create Dashboard buttons." loading="lazy" />
  <figcaption>The Dashboards list, with Browse templates, Import and Create Dashboard in the header.</figcaption>
</figure>

A new dashboard opens in edit mode. Click **Add Widget** to add the first widget.

## Templates

Templates are ready-made dashboards for common setups:

| Group          | Templates                                                                                                                                                   |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Application    | Service Health, HTTP Endpoints, gRPC Service, Error Tracking, Top Errors, Platform Overview, Metric Overview, Node.js Runtime, JVM Runtime, Blank Dashboard |
| Database       | Postgres Overview, MySQL Overview, MongoDB Overview, Redis Overview                                                                                         |
| Messaging      | Kafka Overview, NATS Overview, RabbitMQ Overview                                                                                                            |
| Infrastructure | Host Metrics, Docker Containers, Kubernetes Cluster, Kubernetes Pods, Cloudflare Edge, PlanetScale Databases, Google Cloud                                  |

Pick a template to see a live preview on your own data and what it needs. Some templates take **Parameters**, such as a service name. Click **Create dashboard** to create your own copy, which you can edit or delete freely.

If the data a template needs is not arriving yet, the panel links to the setup for it. **Create it anyway** creates the dashboard now. Its widgets stay empty until the data arrives, then fill in on their own.

## Add widgets

In edit mode, click **Add Widget**. The picker offers these widget types:

| Widget             | Shows                                                          |
| ------------------ | -------------------------------------------------------------- |
| Line, Area, Bar    | A value over time, optionally split into series by a group-by. |
| Horizontal Bar     | Values compared across groups.                                 |
| Pie                | Share of a total across groups.                                |
| Histogram, Heatmap | A distribution, such as request durations.                     |
| Stat               | One number, such as the current error rate.                    |
| Gauge              | One number against a range.                                    |
| Table              | Grouped values as rows.                                        |
| List               | Recent rows, such as the latest traces or log lines.           |
| Funnel             | Conversion through a sequence of product event steps.          |
| Paths              | The paths people take between product events.                  |
| Note               | Markdown text, for headings, explanations and links.           |

Most types come with presets you can start from. Pick one, then edit its query.

### Queries

A widget's editor has two tabs:

- **Query Builder** reads **Traces**, **Logs**, **Metrics** or **Product events**. Pick an aggregation, add filters and a group-by, and combine queries with a formula.
- **Raw SQL** runs your own SQL. The query must reference `$__orgFilter`. See the [SQL reference](/docs/reference/sql).

The preview updates as you edit. Save the widget to put it on the dashboard.

### Arrange and manage widgets

In edit mode, drag widgets to move them and drag their corners to resize. From the **⋮** menu in the dashboard header:

- **Auto Layout** rearranges the widgets to fill the grid without gaps.
- **Add group** adds a titled section you can move widgets into.
- **Variables** and **Tags** open their editors.

Each widget's **⋮** menu has **Edit**, **Clone**, **Create alert**, **Embed chart** and **Delete**. **Create alert** turns the widget's query into an [alert rule](/docs/alerting/alert-rules). **Embed chart** is covered in [Embed dashboard charts](/docs/dashboards/embed-charts).

Click **Done** to leave edit mode.

## Variables

A variable adds a selector to the dashboard header. Widgets that reference it re-query when the selection changes. Add variables from **⋮ → Variables** in edit mode.

| Type    | Values                                                                               |
| ------- | ------------------------------------------------------------------------------------ |
| Query   | Values from your telemetry: services, environments, or a span or resource attribute. |
| Custom  | A fixed, comma-separated list you define.                                            |
| Textbox | Free text, for example a search term.                                                |

Each variable has a **Name**, an optional **Label** and **Default value**, and can include an **All** option. Reference it in a widget's query as `$name`, for example `$service`.

The selected values are part of the URL (`?var-service=checkout`), so a link to the dashboard opens with the same selection.

## Time range and refresh

The time range picker in the header applies to every widget. The refresh control reloads widgets on an interval, and the dashboard remembers its default interval.

## Share a dashboard

Open **⋮ → Share…** and pick who can open the dashboard's link:

| Option                      | Who can view                                                                      |
| --------------------------- | --------------------------------------------------------------------------------- |
| Not shared                  | Members of your organization, inside Maple.                                       |
| Anyone in this organization | Signed-in members of your organization, from the link.                            |
| Anyone with the link        | Anyone who has the link, without signing in. They see the dashboard and its data. |

**Replace** issues a new link and stops the old one working. Shared views render query builder charts, raw SQL, notes, funnels, paths, and the errors, service and logs widgets. The dialog names any widget that will not render for viewers.

To put a single chart into another page, see [Embed dashboard charts](/docs/dashboards/embed-charts).

## Find, export and restore

- **Dashboards** lists every dashboard, with search, sorting, and tags. Star a dashboard to add it to **Favorites**. Favorites are stored on your device.
- **⋮ → Export as JSON** downloads the dashboard. **Import** on the Dashboards page loads it back, into this or another organization.
- **⋮ → Version history** lists every saved version of the dashboard, so you can look at an earlier one and restore it.

## Next steps

- [Embed dashboard charts](/docs/dashboards/embed-charts)
- [Alert rules](/docs/alerting/alert-rules)
- [SQL reference](/docs/reference/sql)

---

# Embed dashboard charts in your own product

> Dashboards · https://maple.dev/docs/dashboards/embed-charts

Put a live Maple chart in your admin panel, internal tool or customer-facing page with a plain iframe. Make the dashboard public, copy the chart's embed snippet, and set theme, time range, refresh and variables in the URL.

Any chart on a public Maple dashboard can go into another page as a plain `<iframe>`. The chart queries Maple every time it loads, so the page shows the same numbers as the dashboard, with no export job and no second copy of the data.

The example on this page is Fieldnote, a made-up B2B app. Its admin panel has a Growth page, and its three funnel charts are embedded from Maple.

<figure class="shot">
  <img src="/screenshots/docs/embed-charts-06-customer-product.webp" alt="A light-themed admin panel for an app called Fieldnote. Its Growth page shows three KPI cards and, below them, three funnel charts embedded from Maple: pricing page to paid plan, signup to activation, and invite to paid." loading="lazy" />
  <figcaption>Fieldnote's admin panel. The KPI cards are Fieldnote's own markup; the three funnels are Maple charts in iframes, using <code>theme=light</code>.</figcaption>
</figure>

The charts come from a Maple dashboard called **Customer funnel**:

<figure class="shot">
  <img src="/screenshots/docs/embed-charts-01-dashboard.webp" alt="The Customer funnel dashboard in Maple with three funnel charts: pricing page to paid plan, signup to activation, and invite to paid." loading="lazy" />
</figure>

Other chart types embed the same way. See [Which charts can be embedded](#which-charts-can-be-embedded).

## Embed a chart in three steps

### Step 1: make the dashboard public

A chart can only be embedded while its dashboard is shared with **Anyone with the link**. Open the **⋮** menu in the dashboard header, choose **Share…**, and pick that option.

<figure class="shot">
  <img src="/screenshots/docs/embed-charts-05-share-dialog.webp" alt="The Share dashboard dialog with three options: Not shared, Anyone in this organization, and Anyone with the link, which is selected. Below it is the dashboard's link with Copy and Replace buttons." loading="lazy" />
  <figcaption>The Share dialog. Embeds need <strong>Anyone with the link</strong>.</figcaption>
</figure>

This makes the whole dashboard viewable, without signing in, by anyone who has the dashboard's own link. If some charts on a dashboard should stay private, move the charts you want to embed to a dashboard of their own.

### Step 2: open Embed chart

Hover the chart, open its **⋮** menu and choose **Embed chart**.

<figure class="shot">
  <img src="/screenshots/docs/embed-charts-02-menu.webp" alt="A chart's menu on the dashboard, opened from the three-dot button in its top-right corner, showing an Embed chart item." loading="lazy" />
</figure>

If the dashboard isn't public yet, the dialog says so and offers to make it public for you. That button does exactly what Step 1 does.

<figure class="shot">
  <img src="/screenshots/docs/embed-charts-03-not-public.webp" alt="The Embed chart dialog on a dashboard that is not shared. It explains that embeds only work on public dashboards, lists the steps to share it, and has a Make dashboard public button." loading="lazy" />
  <figcaption>On a dashboard that isn't public, the dialog explains how to share it instead of showing a link.</figcaption>
</figure>

On a public dashboard, the dialog creates the chart's own link the first time you open it and shows it with a ready-to-paste snippet.

<figure class="shot">
  <img src="/screenshots/docs/embed-charts-04-embed-dialog.webp" alt="The Embed chart dialog on a public dashboard: the chart's link with Copy and Replace buttons, an iframe snippet with HTML and React tabs, and a list of URL options for theme, time range, refresh and dashboard variables." loading="lazy" />
  <figcaption>The chart's link, the iframe snippet and the URL options it accepts.</figcaption>
</figure>

### Step 3: paste the snippet into your page

Copy the snippet from the **HTML** tab for plain HTML, Vue, Svelte, Angular or a server-rendered template:

```html
<iframe
	src="https://app.maple.dev/share/<token>?embed=true"
	width="100%"
	height="400"
	style="border: 0"
	loading="lazy"
></iframe>
```

Use the **React** tab in JSX. React only accepts `style` as an object, so the HTML version fails to compile there:

```jsx
<iframe
	src="https://app.maple.dev/share/<token>?embed=true"
	width="100%"
	height={400}
	style={{ border: 0 }}
	loading="lazy"
/>
```

The chart fills the frame, so `height` sets the chart's height. `?embed=true` removes Maple's page header, and the embed has no background of its own: the chart's card sits directly on your page, as in the Fieldnote screenshot. The card has 8px of space around it inside the frame, so pull the iframe out by 8px if its edges need to line up with your own cards.

## Set theme, time range, refresh and variables in the URL

Append any of these to the link in `src`:

| Parameter    | Values                                                                                                                                                                      |
| ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `theme`      | `light` or `dark`. Without it, the embed uses the theme the viewer last chose in Maple, and dark for a viewer who has never opened Maple.                                   |
| `range`      | A relative window: minutes, hours, days, weeks or months, such as `30m`, `24h`, `7d`, `2w` or `1mo`, or `today`. Up to 31 days. Defaults to the dashboard's own time range. |
| `from`, `to` | A fixed window in UTC, `YYYY-MM-DD HH:MM:SS`. Set both. Takes precedence over `range`.                                                                                      |
| `refresh`    | Seconds between reloads: `5`, `10`, `30`, `60`, `300` or `900`. `0` turns it off. Defaults to the dashboard's own setting.                                                  |
| `var-<name>` | A value for one of the dashboard's variables, such as `var-service=checkout`.                                                                                               |

The dialog lists the dashboard's own variables with an example value for each. A space in `from` and `to` works as typed inside an HTML attribute.

For example, a light chart over the last 7 days that reloads every five minutes:

```html
<iframe
	src="https://app.maple.dev/share/<token>?embed=true&theme=light&range=7d&refresh=300"
	width="100%"
	height="330"
	style="border: 0"
	loading="lazy"
></iframe>
```

A relative window, from `range` or from the dashboard, is recalculated on every load and refresh, so the chart always ends at the current time. A `range` the chart can't read is ignored and the dashboard's range is used instead.

## What a viewer of the embed can see

- **This chart and nothing else.** The chart's link only returns data for that one chart. It can't open the rest of the dashboard, and the chart's query definition isn't sent to the browser.
- **Live data.** Every load and every `refresh` runs the chart's query against your Maple data at that moment.
- **The chart as it is now.** Edit the chart on the dashboard and the embed shows the change on its next load. Delete the chart and the embed shows "This link isn't available".
- **No sign-in.** Anyone who has the link can load it, the same as the dashboard's own public link. Requests are rate-limited per link and per viewer IP.

## Which charts can be embedded

Only charts on a dashboard. Pages such as Traces, Logs, Errors or the Service Map can't be embedded; put the chart you need on a dashboard first.

On a dashboard, every chart you can add today can be embedded except **Recent Traces**. That covers query builder charts, raw SQL charts, markdown, funnels, paths and the service, error and log presets.

Older dashboards can hold charts built on data sources Maple no longer offers, such as trace facets or metric summaries. Those can't be embedded either. For any chart that can't be embedded, **Embed chart** is greyed out, and hovering it says why.

## FAQ

### How do I stop embedding a chart?

Pick the option that matches how permanent you want it to be:

- **Turn off every embed on the dashboard, reversibly.** Set the dashboard back to **Not shared** in its Share dialog. Every chart link on it stops working. Set it to **Anyone with the link** again and the same links, and the embeds using them, work again.
- **Kill one chart's link for good.** Open **Embed chart** on that chart and click **Replace**. The old link never works again, and the dialog shows a new one. Anyone still embedding the chart needs the new link.
- **Remove one chart's link without replacing it via the API.** Use a key that has `dashboards:write`:

```bash
curl -X DELETE \
  -H "Authorization: Bearer $MAPLE_API_KEY" \
  https://api.maple.dev/v2/dashboards/<dashboard_id>/widgets/<widget_id>/share
```

Opening **Embed chart** on that chart later creates a new link.

### Does replacing the dashboard's share link break my embeds?

No. **Replace** in the Share dialog only replaces the dashboard's own link. Chart embeds keep working as long as the dashboard stays public.

### Why does my embed say "This link isn't available"?

One of these happened: the dashboard is no longer shared, the chart's link was replaced or removed, or the chart was deleted from the dashboard.

### Can I embed a whole dashboard?

No. Only a single chart's link can be shown inside another site. The dashboard's own link shows "This link can't be embedded" in an iframe; open it directly or link to it instead.

### What if my dashboard is shared with "Anyone in this organization"?

Embeds don't work, even for people who are signed in to Maple. A chart link is never more open than its dashboard, and embedding needs the dashboard to be public.

### Do I need to change my Content Security Policy?

Only if your page sets `frame-src` (or `child-src`) in its `Content-Security-Policy`. Add `https://app.maple.dev`, or `https://app.eu.maple.dev` if your organization is in the EU region.

The chart is a JavaScript app, so it won't render inside an iframe whose `sandbox` attribute leaves out `allow-scripts`. Some page builders add that restriction to embedded HTML.

### Does the embed work when my app uses a light theme?

Add `&theme=light` to the link. Without `theme`, a viewer who has never opened Maple sees the dark theme, and a viewer who has sees the theme they last chose. Set `theme` explicitly so every viewer sees the same chart. The embed has no background, so the chart's card sits on your page in either theme.

---

# Alert rules

> Alerting · https://maple.dev/docs/alerting/alert-rules

Create an alert rule in Maple: pick a signal, scope it to services and environments, set the threshold and evaluation timing, attach destinations, and preview it against past data. In the app or over the API.

An **alert rule** watches one signal, checks it against a threshold every minute, and opens an [incident](/docs/alerting/incidents) when the threshold is crossed for long enough. Each incident notifies the [destinations](/docs/alerting/notification-destinations) attached to the rule.

You manage rules on the **Alerts** page. Creating, editing and deleting rules requires the organization admin role.

<figure class="shot">
  <img src="/screenshots/docs/alert-rules-01-overview.webp" alt="The Alerts overview: counts of firing, needs-attention, healthy and disabled rules, then a table of rules with severity, status, a 24-hour check strip, last value and last evaluation time." loading="lazy" />
  <figcaption>The Alerts overview. The strip in each row shows the last 24 hours of checks.</figcaption>
</figure>

## Prerequisites

- Telemetry arriving in Maple. Built-in signals read traces.
- At least one [notification destination](/docs/alerting/notification-destinations). A rule cannot be saved without one.

## Create a rule

1. Open **Alerts** and click **New rule**. To start from a service, open the service and click **Create Alert**, which fills in the scope.
2. **Start with a template** opens first. Pick a template, **Start blank**, or **From a dashboard chart**. Every field a template sets stays editable.
3. Fill in **Signal & threshold**, **Scope**, **Notifications** and **Details**, described below.
4. Click **Test rule** to preview the rule against past data.
5. Click **Create rule**.

<figure class="shot">
  <img src="/screenshots/docs/alert-rules-02-templates.webp" alt="The Start with a template dialog with six options: High error rate, Slow P95 latency, Slow P99 latency, Low Apdex score, Throughput drop and Start blank, plus From a dashboard chart." loading="lazy" />
  <figcaption>The template picker that opens when you create a rule.</figcaption>
</figure>

The templates:

| Template         | Signal     | Fires when          | Window | Also sets                                        |
| ---------------- | ---------- | ------------------- | ------ | ------------------------------------------------ |
| High error rate  | Error rate | above 5%            | 5 min  | **Group by** `service.name` when no scope is set |
| Slow P95 latency | P95        | above 1000 ms       | 5 min  |                                                  |
| Slow P99 latency | P99        | above 2000 ms       | 5 min  |                                                  |
| Low Apdex score  | Apdex      | below 0.8 (T 500ms) | 5 min  |                                                  |
| Throughput drop  | Throughput | below 100           | 5 min  | **Min samples** 0                                |

## Signals

The signal kind is **Built-in**, **Query** or **Raw SQL**.

**Built-in** signals read the entry-point spans of each service: server and consumer spans, and trace roots. That is the same set of requests the service pages chart.

| Signal     | API `signal_type` | Value compared against the threshold                                                                         |
| ---------- | ----------------- | ------------------------------------------------------------------------------------------------------------ |
| Error rate | `error_rate`      | Share of requests with status `Error`. Entered as a percent in the app, a 0 to 1 ratio in the API.           |
| P95        | `p95_latency`     | 95th percentile duration, in milliseconds.                                                                   |
| P99        | `p99_latency`     | 99th percentile duration, in milliseconds.                                                                   |
| Apdex      | `apdex`           | Apdex score from 0 to 1, against the **Apdex target (ms)**. See [Apdex alerts](/docs/alerting/apdex-alerts). |
| Throughput | `throughput`      | Estimated number of requests in the window.                                                                  |

Counts and rates are weighted for sampling. See [Sampling and throughput](/docs/concepts/sampling-throughput).

A built-in signal only sees entry-point spans. A service that records a failure on a child span and still returns success from its entry point stays healthy on these signals. Use a **Query** or **Raw SQL** rule for that case.

**Query** (`builder_query`) uses the same query builder as dashboard charts. It can read traces, logs, metrics or product events, with its own filters and group-by. The quickest way to build one is from a chart: open a dashboard, open the chart's menu and choose **Create alert**.

**Raw SQL** (`raw_query`) runs your own SQL. The query must include `$__orgFilter` and a `$__timeFilter(...)` on the time column, and return a time bucket and a value. **Reduce buckets by** turns the buckets in the window into one value: **Last bucket**, **Sum**, **Average**, **Minimum** or **Maximum**. To evaluate several groups, return a group column. Return a `samples` column with the number of events behind each row: **Min samples** sums it, and without it every row counts as one sample, so the minimum counts buckets rather than events. See the [SQL reference](/docs/reference/sql).

For **Query** and **Raw SQL** rules the query carries its own filters, so the **Scope** section is hidden.

## Threshold

**Condition** is one of `>`, `>=`, `<`, `<=`, `=`, `!=`, `between` or `not between`. The range conditions use **Lower** and **Upper**. **Threshold** is the value the signal is compared against.

**Severity** is **Warning** or **Critical**. It is shown on the incident and in every notification.

## Scope

Scope applies to built-in signals.

- **Services.** Empty means every service. When you pick more than one service, Maple evaluates each service on its own and opens a separate incident per service.
- **Environments.** Empty means every environment. Otherwise the rule only counts spans from the listed deployment environments (`deployment.environment.name`).
- **Group by.** With no services selected, a group-by such as `service.name` or an attribute like `attr.http.route` evaluates each group on its own. You get one incident per group that breaches, instead of one blended value that may never cross the threshold.
- **Exclude services.** Skips named services. It needs **Group by** set to `service.name` and no services selected.

## Evaluation timing

Maple evaluates every enabled rule once a minute. Each check aggregates the last **Window (min)** minutes, so a 5-minute window is a rolling 5 minutes re-scored every 60 seconds.

**Evaluation timing** is collapsed to a summary line (for example `5min · 2× · renotify 30min`) until you open it:

| Field              | App default | API field                       | API default | What it does                                                        |
| ------------------ | ----------- | ------------------------------- | ----------- | ------------------------------------------------------------------- |
| Window (min)       | 5           | `window_minutes`                | required    | Length of the window each check aggregates. 1 to 1440 minutes.      |
| Breaches to fire   | 2           | `consecutive_breaches_required` | 2           | Consecutive breaching checks before an incident opens.              |
| Healthy to resolve | 2           | `consecutive_healthy_required`  | 2           | Consecutive healthy checks before an incident resolves.             |
| Min samples        | 50          | `minimum_sample_count`          | 0           | A check with fewer samples than this is skipped.                    |
| Renotify (min)     | 30          | `renotify_interval_minutes`     | 30          | How often an open incident notifies again while it keeps breaching. |

A skipped check counts neither as a breach nor as healthy. It leaves the breach and healthy counters where they were.

A window with no data at all is skipped, with two exceptions: a rule with **Alert when there is no data** on (`alert_on_no_data`) counts it as a breach, and otherwise a **Throughput** rule with `<` or `<=` treats it as zero. Turn the switch on for **Raw SQL** rules, where a query that stops matching otherwise goes quiet instead of firing. It is not available on grouped rules: a group that stops reporting keeps its open incident until its telemetry returns. On a **Raw SQL** rule that returns a group column, it fires only when the query returns no rows at all.

The **Min samples** check runs before the threshold comparison, and that zero still counts as zero samples. For **Throughput** the sample count is the signal, so a drop rule with the blank form's default of 50 skips every window below 50 requests, including a full outage, and cannot fire for those windows. Set **Min samples** to 0 for throughput drop rules, as the **Throughput drop** template does. Then traffic stopping entirely fires the rule.

Short windows on low-traffic services are noisy, because a few slow or failed requests move the value a long way. Raise **Min samples** or widen the window for those services.

## Notifications

Pick one or more destinations in **Notifications**. **Send test** delivers a test notification for this rule to the selected destinations.

**Message template** customizes the title and Markdown body of Slack, Discord, Telegram and PagerDuty notifications with `{{ variable }}` substitution, for example `{{ rule.name }}`, `{{ value }}` or `{{ links.app }}`. Leave it blank for the built-in format. Email, webhook and Hazel destinations always use the built-in format.

## Details

**Rule name** is required. **Tags** (up to 20, each up to 32 characters) group and filter rules and incidents on the Alerts page. **Notes** are free text shown with the rule.

## Preview a rule

The chart at the top of the form replays the rule over a past time range. Pick the range and click **Test rule**. The chart shows the value for each window, the threshold, and shaded spans where the rule would have held an incident open. The badge reads **Would trigger** or **Within threshold** for the latest window.

The preview sends nothing. Use it to check that a threshold would not have fired all week, or would have caught last Tuesday's incident.

## Create a rule over the API

Rules are available at `/v2/alerts/rules` on `https://api.maple.dev` (`https://api.eu.maple.dev` for EU organizations). Use an API key (`maple_ak_…`) from **Settings → Keys**. Creating, updating, deleting and testing a rule requires the `alerts:write` scope and the org admin role.

```bash
curl -X POST https://api.maple.dev/v2/alerts/rules \
  -H "Authorization: Bearer maple_ak_…" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Checkout error rate",
    "signal_type": "error_rate",
    "comparator": "gt",
    "threshold": 0.05,
    "window_minutes": 5,
    "service_names": ["checkout"],
    "environments": ["production"],
    "severity": "critical",
    "minimum_sample_count": 50,
    "destination_ids": ["dest_oybbpTBhtSFGShMjjLiCrh"]
  }'
```

Error rate thresholds are 0 to 1 ratios in the API (`0.05` is 5%). `destination_ids` must name at least one existing destination.

To preview a rule over a past range, `POST /v2/alerts/rules/preview` with the rule under `rule` and the range in `start_time` and `end_time`. It needs the `alerts:read` scope and sends nothing.

```bash
curl -X POST https://api.maple.dev/v2/alerts/rules/preview \
  -H "Authorization: Bearer maple_ak_…" \
  -H "Content-Type: application/json" \
  -d '{
    "rule": {
      "name": "Checkout error rate",
      "signal_type": "error_rate",
      "comparator": "gt",
      "threshold": 0.05,
      "window_minutes": 5,
      "service_names": ["checkout"],
      "severity": "critical",
      "destination_ids": ["dest_oybbpTBhtSFGShMjjLiCrh"]
    },
    "start_time": "2026-07-14T00:00:00.000Z",
    "end_time": "2026-07-15T00:00:00.000Z"
  }'
```

The response lists the value and status of each window per group in `series`, and the spans where an incident would have been open in `would_fire`.

`POST /v2/alerts/rules/test` evaluates a rule once against current data. Set `send_notification` to `true` to also deliver a test notification. See the [API reference](/docs/reference/api) for every endpoint and field.

## Troubleshooting

- **The rule never fires.** Open the rule and look at its checks. If every check is skipped, each one says why: **no data** means the scope or query matches nothing, **below min samples** means the window has fewer samples than **Min samples**. Check the service names and environments against what is arriving.
- **The rule fires and resolves over and over.** Raise **Breaches to fire** and **Healthy to resolve**, or widen the window.
- **Save is disabled.** The action bar lists what is missing, such as a rule name or a destination.

## Next steps

- [Incidents](/docs/alerting/incidents): what happens after a rule fires.
- [Apdex alerts](/docs/alerting/apdex-alerts): choosing the target and threshold for an Apdex rule.
- [Uptime monitoring](/docs/alerting/uptime-monitoring): availability alerts from traces, and HTTP checks from the OpenTelemetry Collector.
- [Notification destinations](/docs/alerting/notification-destinations): where notifications go.

---

# Incidents

> Alerting · https://maple.dev/docs/alerting/incidents

What happens after a Maple alert rule fires: how an incident opens, renotifies, waits on missing data and resolves, what each notification contains, and where to see incidents.

An **incident** is one period during which an [alert rule](/docs/alerting/alert-rules) is breaching. Maple opens it, notifies the rule's [destinations](/docs/alerting/notification-destinations), repeats the notification while the breach continues, and resolves it when the signal recovers.

A rule that evaluates several groups (several services, or a **Group by**) keeps a separate incident for each group. One slow route opens one incident. It does not open or resolve the others.

## Lifecycle

Maple checks every enabled rule once a minute. Each check is **breached**, **healthy** or **skipped** (too few samples, or no data).

| Step     | When it happens                                                                                         | Notification |
| -------- | ------------------------------------------------------------------------------------------------------- | ------------ |
| Open     | The rule breaches on **Breaches to fire** consecutive checks.                                           | `trigger`    |
| Renotify | The incident is still breaching and **Renotify (min)** minutes have passed since the last notification. | `renotify`   |
| Resolve  | The rule is healthy on **Healthy to resolve** consecutive checks.                                       | `resolve`    |

Skipped checks do not move the incident either way. The breach and healthy counters keep their values until the next check that has enough data.

An incident has two statuses, **open** and **resolved**. A resolved incident stays resolved. If the rule breaches again later, Maple opens a new incident.

### Flapping

If a rule resolves and then opens a new incident for the same group within its **Renotify (min)** interval, Maple opens the incident without sending a `trigger`. The matching `resolve` is not sent either. This stops a signal that hovers around the threshold from sending a notification every few minutes.

## Waiting on data

A breach can stop appearing because the problem is fixed, or because the service stopped sending telemetry. Before it resolves an incident on an empty window, Maple checks whether the service's traffic is still arriving. It compares the current window with the traffic just before the incident opened, and with the same time the day before.

If Maple cannot confirm the traffic, the incident stays open and shows **Waiting on data** in place of its current value. Hovering the badge shows why:

| Situation                                                                     | Resolves                                                     |
| ----------------------------------------------------------------------------- | ------------------------------------------------------------ |
| The service stopped reporting.                                                | When telemetry returns, or after 6 hours.                    |
| Traffic fell below half of its usual level.                                   | When data returns, or after 3 windows (at least 30 minutes). |
| Raw span counts fell while sample-weighted counts did not (sampling changed). | When the signal settles, or after 6 hours.                   |
| Maple could not verify telemetry for the service.                             | On the next successful check.                                |

If the breach reappears while the incident is waiting, it goes back to firing. If the time limit passes without a breach, Maple resolves the incident and says so in the resolve reason.

## What a notification contains

The built-in format carries:

- The rule name and the event (**Triggered**, **Re-notification**, **Resolved** or **Test**).
- The severity (**Warning** or **Critical**).
- The signal, the observed value and the threshold it was compared against.
- The group, or `all` for an ungrouped rule.
- The evaluation window.
- A link to the alert in Maple and a link to ask Maple AI about it.

A rule's **Message template** replaces the title and body for Slack, Discord, Telegram and PagerDuty destinations. Email, webhook and Hazel destinations use the built-in format. PagerDuty receives a stable `dedup_key` so the trigger and resolve land on the same PagerDuty incident. The webhook payload is in the [alert webhooks reference](/docs/reference/webhooks).

If a delivery fails with an error that can be retried, Maple retries it up to 5 attempts, with a delay that doubles from 1 minute up to 15 minutes.

## Where to see incidents

- **Alerts → Overview.** When anything is firing, **Active incidents** lists every open incident with its **Severity**, **Rule**, **Group**, **Current value**, **Duration** and **Last notified** time. The summary cards count rules that are **Firing**, **Needs attention**, **Healthy** and **Disabled**.
- **The rule's page.** Click a rule or an incident row. The **Overview** tab charts the signal with the threshold and lists recent checks. The **History** tab lists past incidents, filterable by **All**, **Fired** and **Resolved**, with **Total triggered**, **Avg resolution** and **Top contributors**.
- **MCP.** `list_alert_incidents` and `get_incident_timeline` on the [MCP server](/docs/reference/mcp) return the same data to an AI assistant.

## Troubleshooting

- **An incident is stuck on Waiting on data.** The service stopped sending the telemetry the rule reads. Check the service under **Traces**. The incident resolves on its own once data returns or the time limit passes.
- **An incident opened but nobody was notified.** Check that the rule has destinations and that none is marked **Disabled** on the **Destinations** tab. A re-opened incident inside the renotify interval is silent by design (see [Flapping](#flapping)).
- **Too many notifications.** Raise **Renotify (min)**, or raise **Breaches to fire** and **Healthy to resolve** on the rule.

## Next steps

- [Alert rules](/docs/alerting/alert-rules): the settings that control when incidents open and resolve.
- [Notification destinations](/docs/alerting/notification-destinations): where notifications go.
- [Alert webhooks reference](/docs/reference/webhooks): the webhook payload.

---

# Apdex alerts

> Alerting · https://maple.dev/docs/alerting/apdex-alerts

How to set up an Apdex alert in Maple: pick the target latency T, choose the score worth paging for, scope and tune the rule, and create the same rule over the API.

A p95 latency alert tells you that 5% of requests were slower than some number. It does not tell you whether that was 5 requests or 50,000, and it says nothing about the requests that failed outright.

**Apdex answers a different question:** of everything that hit this service in the last five minutes, what share of it was <span class="hl-ok">fast enough to keep a user happy</span>?

<div class="my-7 grid gap-3 sm:grid-cols-2 not-prose">
  <div class="rounded-lg border border-border p-4" style="background: color-mix(in oklab, var(--bg-elevated) 45%, transparent)">
    <div class="text-[10px] uppercase tracking-wider" style="color: var(--muted-foreground)">A p95 alert says</div>
    <div class="mt-2 text-sm text-fg">“The slow 5% crossed 1 second.”</div>
    <div class="mt-2 text-xs text-fg-muted">Shape of the tail. Silent when requests fail fast.</div>
  </div>
  <div class="rounded-lg border p-4" style="border-color: color-mix(in oklab, var(--primary) 45%, var(--border)); background: color-mix(in oklab, var(--primary) 8%, transparent)">
    <div class="text-[10px] uppercase tracking-wider" style="color: var(--primary)">An Apdex alert says</div>
    <div class="mt-2 text-sm text-fg">“One request in five was slow or broken.”</div>
    <div class="mt-2 text-xs text-fg-muted">Size of the tail, counting failures as unhappy users.</div>
  </div>
</div>

## The score in one line

Apdex compresses a latency distribution into a single number between <span class="hl-bad">0</span> and <span class="hl-ok">1</span>: the share of requests that were fast enough to keep a user happy. You set one target time, <span class="hl-t">T</span>. Requests under <span class="hl-t">T</span> are <span class="hl-ok">satisfied</span> and score a point, requests under <span class="hl-t">4T</span> are <span class="hl-warn">tolerating</span> and score half, and everything slower is <span class="hl-bad">frustrated</span> and scores nothing. A request that failed is frustrated too, at any speed.

<div class="my-6 rounded-lg border border-border p-5 text-center not-prose" style="background: color-mix(in oklab, var(--bg-elevated) 55%, transparent)">
  <div class="font-mono text-[15px] text-fg">Apdex = (satisfied + 0.5 × tolerating) / total</div>
  <div class="mt-2 text-xs text-fg-muted">Always between 0 and 1. Both boundaries come from the single value you set for T.</div>
</div>

Counting failures as frustrated is what makes this worth alerting on. A dependency that starts failing 15% of requests in 40ms makes your p95 _faster_, so a latency alert stays quiet while Apdex falls from 0.97 to about 0.83.

[**What is Apdex?**](/guides/what-is-apdex) walks through the formula, a worked example, and where Apdex and latency percentiles disagree. The rest of this page is the Maple-specific part: picking T for a rule, and building it.

## Choosing T for the rule

<span class="hl-t">T</span> is the whole rule. It sets the <span class="hl-ok">satisfied</span> boundary directly and the <span class="hl-bad">frustrated</span> boundary implicitly, since the <span class="hl-warn">tolerating</span> band always ends at <span class="hl-t">4T</span>. Move T from 500ms to 1s and you have also moved the frustrated line from 2s to 4s.

Pick T as the latency at which your users stop perceiving the response as immediate, per class of traffic:

<div class="my-6 overflow-hidden rounded-lg border border-border not-prose">
  <div class="grid gap-px sm:grid-cols-2" style="background: var(--border)">
    <div class="p-4" style="background: var(--background)">
      <div class="font-mono text-sm" style="color: var(--primary)">200ms – 300ms</div>
      <div class="mt-1 text-sm text-fg">Internal or service-to-service APIs</div>
      <div class="mt-1 text-xs text-fg-muted">They sit inside someone else's request, so their budget is small.</div>
    </div>
    <div class="p-4" style="background: var(--background)">
      <div class="font-mono text-sm" style="color: var(--primary)">500ms</div>
      <div class="mt-1 text-sm text-fg">User-facing API endpoints</div>
      <div class="mt-1 text-xs text-fg-muted">Maple's default, and a fair starting point for a JSON API behind a UI.</div>
    </div>
    <div class="p-4" style="background: var(--background)">
      <div class="font-mono text-sm" style="color: var(--primary)">1s – 2s</div>
      <div class="mt-1 text-sm text-fg">Full page loads and heavy reports</div>
      <div class="mt-1 text-xs text-fg-muted">Users accept more from something that visibly does more work.</div>
    </div>
    <div class="p-4" style="background: var(--background)">
      <div class="font-mono text-sm text-fg-muted">n/a</div>
      <div class="mt-1 text-sm text-fg">Background and batch endpoints</div>
      <div class="mt-1 text-xs text-fg-muted">Nobody is waiting. Alert on throughput or failure rate instead.</div>
    </div>
  </div>
</div>

> **Two ways to pick a T that tells you nothing.** Setting T to your current p95 guarantees a score near 0.95 forever. Setting it so tight that a healthy week already reads 0.6 buries every threshold you might pick inside your normal noise.

To choose, open the service's Apdex chart in Maple, set T, and check that a healthy week reads somewhere in the 0.9 to 1.0 band. That leaves the 0.8 line meaningful.

## Choosing the threshold you defend

<div class="my-6 space-y-2 not-prose">
  <div class="flex items-center gap-3 rounded-lg border border-border px-4 py-3" style="background: color-mix(in oklab, var(--bg-elevated) 55%, transparent)">
    <span class="w-[92px] shrink-0 font-mono text-sm" style="color: var(--success)">0.94 – 1.00</span>
    <span class="text-sm text-fg-muted">Healthy, with a normal slow tail.</span>
  </div>
  <div class="flex items-center gap-3 rounded-lg border border-border px-4 py-3" style="background: color-mix(in oklab, var(--bg-elevated) 55%, transparent)">
    <span class="w-[92px] shrink-0 font-mono text-sm" style="color: var(--warning)">0.85 – 0.94</span>
    <span class="text-sm text-fg-muted">Degraded, and users can feel it.</span>
  </div>
  <div class="flex items-center gap-3 rounded-lg border border-border px-4 py-3" style="background: color-mix(in oklab, var(--bg-elevated) 55%, transparent)">
    <span class="w-[92px] shrink-0 font-mono text-sm" style="color: var(--destructive)">below 0.80</span>
    <span class="text-sm text-fg-muted">The common alerting line. Roughly a fifth of traffic is frustrated or worse.</span>
  </div>
  <div class="flex items-center gap-3 rounded-lg border border-border px-4 py-3" style="background: color-mix(in oklab, var(--bg-elevated) 55%, transparent)">
    <span class="w-[92px] shrink-0 font-mono text-sm" style="color: var(--destructive)">below 0.50</span>
    <span class="text-sm text-fg-muted">Most requests are failing or timing out.</span>
  </div>
</div>

0.8 is a common convention. Set the threshold from where your service sits when it is healthy, and how far it has to fall before you would want to be woken up.

## Setting up an Apdex alert in Maple

Maple computes Apdex over the entry-point spans of a service: server spans, consumer spans and trace roots. The service's Apdex chart draws the same set of requests, so a rule you build here matches the chart.

### 1. Start from the Low Apdex score template

Open **Alerts → New rule**. Starting from a service page instead (**Services → your service → Create Alert**) pre-fills the scope.

The rule builder opens on **Start with a template**. Pick <span class="hl-t">Low Apdex score</span> and every field below is filled: signal Apdex, target 500ms, fires below 0.8, five-minute window. Nothing is locked afterwards.

<figure class="shot">
  <img src="/screenshots/docs/apdex-01-template.webp" alt="The Start with a template dialog in Maple's rule builder, with the Low Apdex score template reading “Apdex < 0.8 / 5min (target 500ms)”." loading="lazy" />
  <figcaption>The Low Apdex score template. Every value it sets stays editable.</figcaption>
</figure>

### 2. Set the target, then the score you defend

<span class="hl-t">Apdex target (ms)</span> is your T: requests under it count as <span class="hl-ok">fully satisfied</span>, and the <span class="hl-bad">frustrated</span> line follows automatically at 4T. Then set **Condition** to `<` and **Threshold** to the score worth paging for, typically `0.8`.

The panel also states the measurement boundary: built-in signals read entry-point spans only. A service that swallows failures on child spans and returns success from its entry point stays healthy here at any threshold. Use a **Query** or **Raw SQL** rule for that case.

<figure class="shot">
  <img src="/screenshots/docs/apdex-02-signal.webp" alt="The Signal & threshold panel with the Apdex chip selected, Apdex target 500 ms, condition less-than, threshold 0.8 and severity Warning." loading="lazy" />
  <figcaption>Apdex target 500&nbsp;ms, condition <code>&lt;</code>, threshold 0.8. (The decimal comma is the browser's locale, not a Maple setting.)</figcaption>
</figure>

### 3. Scope it

One T fits one class of traffic. If a rule covers many services or routes, set **Group by** to `service.name` or `attr.http.route` so each group is scored on its own and opens its own incident. Scope, grouping and the other fields shared by every signal are described in [Alert rules](/docs/alerting/alert-rules#scope).

<figure class="shot">
  <img src="/screenshots/docs/apdex-03-scope.webp" alt="The Scope panel of the rule builder, with Services, Environments, Group by and Exclude services fields." loading="lazy" />
  <figcaption>Scope and grouping. Empty means every service and every environment.</figcaption>
</figure>

### 4. Window, timing, destinations

Five minutes is the default window. Short windows on low-traffic services are noisy for Apdex in particular, because a handful of slow requests moves the ratio a long way. Keep **Min samples** at 50 or higher, and raise it on low-traffic services, so a service with 4 requests at 3am cannot post a score of 0.5 and page someone.

<figure class="shot">
  <img src="/screenshots/docs/apdex-04-timing.webp" alt="The severity toggle set to Warning, and the expanded evaluation timing row: window 5 minutes, 2 breaches to fire, 2 healthy checks to resolve, minimum 50 samples, renotify every 30 minutes." loading="lazy" />
  <figcaption>Severity and evaluation timing. The defaults shown are the ones the template sets.</figcaption>
</figure>

An Apdex rule skips windows with no requests, so a service that goes quiet overnight does not read as a score of 0. The other timing fields (**Breaches to fire**, **Healthy to resolve**, **Renotify (min)**) work as on every rule. See [Evaluation timing](/docs/alerting/alert-rules#evaluation-timing).

Pick a **Severity**, attach your [notification destinations](/docs/alerting/notification-destinations), click **Test rule** to replay the rule over the past week, and save.

### Creating the same rule over the API

Every field in the form is a field on the API. `apdex_threshold_ms` is T in milliseconds. It defaults to 500 when omitted.

```bash
curl -X POST https://api.maple.dev/v2/alerts/rules \
  -H "Authorization: Bearer maple_ak_…" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Checkout Apdex below 0.8",
    "signal_type": "apdex",
    "apdex_threshold_ms": 500,
    "comparator": "lt",
    "threshold": 0.8,
    "window_minutes": 5,
    "service_names": ["checkout"],
    "environments": ["production"],
    "severity": "critical",
    "minimum_sample_count": 50,
    "consecutive_breaches_required": 2,
    "destination_ids": ["dest_oybbpTBhtSFGShMjjLiCrh"]
  }'
```

To see what the rule would have done over a past range before it can wake anyone, send the same object as `rule` to `POST /v2/alerts/rules/preview`, with `start_time` and `end_time`:

```json
{
	"rule": {
		"name": "Checkout Apdex below 0.8",
		"signal_type": "apdex",
		"apdex_threshold_ms": 500,
		"comparator": "lt",
		"threshold": 0.8,
		"window_minutes": 5,
		"service_names": ["checkout"],
		"severity": "critical",
		"destination_ids": ["dest_oybbpTBhtSFGShMjjLiCrh"]
	},
	"start_time": "2026-07-08T00:00:00.000Z",
	"end_time": "2026-07-15T00:00:00.000Z"
}
```

See [Alert rules](/docs/alerting/alert-rules#create-a-rule-over-the-api) and the [API reference](/docs/reference/api) for the full schema.

### Or hand it to an agent

You can also create the rule from an AI assistant. Connect Maple's [MCP server](/docs/reference/mcp) to any MCP client, then ask for the rule in plain language:

> Create an Apdex alert on the checkout service. Target 500ms, page me when the score drops below 0.8, and send it to the on-call Slack channel.

The agent builds the same rule you would build by hand. It cannot save an Apdex rule without a target, and it only sees your own organization's data.

After the rule exists, you can ask which of your Apdex rules would never fire, or why one paged last night. The agent reads the rule's checks and incidents to answer.

## Known limitations

- **Apdex is computed on sampled spans.** Under uniform sampling the score stays representative. A sampler that keeps slow or failed traces preferentially, without reporting its sampling weight, drags the score down. See [Sampling and throughput](/docs/concepts/sampling-throughput).
- **One T per rule.** A service whose `/healthz` and `/reports/export` share a rule is being measured against a target that fits neither. Group by `attr.http.route`, or write separate rules.
- **The score hides magnitude.** It counts frustrated requests without weighting how frustrated they were. Pair it with a p99 rule.

The [What is Apdex?](/guides/what-is-apdex) guide covers the formula, a worked example, what counts as a good score, and how Apdex compares to p95 and p99.

---

# Notification destinations

> Alerting · https://maple.dev/docs/alerting/notification-destinations

Route Maple alerts to Slack, email, PagerDuty, Discord, Telegram, Hazel or any HTTP endpoint. How to add a destination, send a test, and get the right credentials for each provider.

A **notification destination** is where Maple delivers an alert when one of your rules fires. You add a destination once and attach it to any number of [alert rules](/docs/alerting/alert-rules). When a rule opens an incident, Maple sends a `trigger`. When the incident resolves, it sends a `resolve`. See [Incidents](/docs/alerting/incidents) for when each is sent.

Destinations live on the **Destinations** tab of the **Alerts** page. Click **Add destination**, pick a provider, and fill in the **Connection** fields. Only organization admins can add, edit or delete destinations. Provider credentials are encrypted at rest and never returned to the browser after they are saved.

## Sending a test

Every destination has a **Send test** button. It delivers a sample alert through the same code path that delivers real alerts, so it confirms your credentials before you attach the destination to a rule.

If a test fails, Maple shows the provider's own rejection reason in the toast and on the destination card. For example, a bad PagerDuty key reports `PagerDuty delivery failed with 400: Invalid routing key`.

## Failed deliveries

A delivery that fails with an error retrying cannot fix, such as a rejected credential, counts against the destination. After 3 such failures in a row, Maple disables the destination and marks it **Disabled** on its card. A successful delivery resets the count. Fix the credential, send a test, and re-enable the destination.

## Slack

Link your Slack workspace to Maple, then choose the channel alerts go to. Maple posts through the linked workspace's bot. Incoming-webhook URLs are not supported.

1. Open **Settings → Integrations → Slack** and connect your workspace. See the [Slack integration](/docs/integrations/slack).
2. On the **Destinations** tab, click **Add destination** and pick the linked workspace.
3. Pick a **Channel** and save.

## Email

Send alerts by email to members of your Maple organization.

1. Click **Add destination** and pick **Email**.
2. Under **Recipients**, select up to 10 members of the organization.
3. Save, then click **Send test**.

Recipients must be members of the organization. Maple looks up each member's email address when you save, so you cannot enter an arbitrary address. To alert a distribution list or an external address, use a [Webhook](#webhook) destination.

Email always uses Maple's built-in message format. A rule's **Message template** does not apply to it.

## PagerDuty

Maple triggers PagerDuty incidents through the **Events API v2**. It needs an Events API v2 **integration key** (also called a routing key): a 32-character string scoped to one PagerDuty service.

> A PagerDuty **REST API token** (from _User Settings_ or _API Access Keys_) does not work here. It fails the test with `PagerDuty delivery failed with 400: Invalid routing key`. The REST API manages PagerDuty itself. The Events API is what Maple posts alerts to.

To get the right key:

1. In PagerDuty, go to **Services → Service Directory** and open (or create) the service that should receive these alerts.
2. Open the **Integrations** tab.
3. Click **Add integration** and choose **Events API v2**.
4. Copy that integration's **Integration Key** (32 characters).
5. Paste it into Maple's **Integration key** field and click **Send test**.

See PagerDuty's [services and integrations guide](https://support.pagerduty.com/main/docs/services-and-integrations) for screenshots.

| Field               | Notes                                                            |
| ------------------- | ---------------------------------------------------------------- |
| **Integration key** | The 32-character Events API v2 routing key from the steps above. |

Maple sends each `trigger` with a stable `dedup_key` and a matching `resolve` when the incident closes, so PagerDuty groups the lifecycle into one incident.

## Discord

Post alerts to a Discord channel through an incoming webhook.

1. In Discord, open **Channel settings → Integrations → Webhooks → New Webhook**.
2. Copy the webhook URL (`https://discord.com/api/webhooks/...`) into the **Discord webhook URL** field.

## Telegram

Send alerts to a Telegram chat, group or channel through a bot you create. Telegram has no per-channel webhook, so a destination needs two values: the bot's token and the id of the chat it posts to.

**1. Create the bot.** In Telegram, message [@BotFather](https://t.me/BotFather), send `/newbot`, and follow the prompts. BotFather replies with a token of the form `123456789:AAH…`. That is the **Bot token** field. Copy it without the `bot` prefix.

**2. Add the bot to the chat.** Invite it to the group or channel you want alerts in. For a channel, add it as an administrator with permission to post messages. A bot cannot message a chat it is not a member of.

**3. Pick the chat.** In Maple, paste the bot token and click **Detect chats**. Maple asks Telegram which chats the bot can see and lists them by name. Pick one and the chat ID fills in.

Detection reads the bot's recent updates:

- **Adding the bot is enough.** You do not need to send a message first. Telegram notifies the bot when it is added to a chat, and Maple reads that notification.
- **Telegram keeps about 24 hours of updates.** An empty list usually means the bot was added longer ago than that. Send it a message, or remove and re-add it, and detect again.
- **A bot with a webhook registered cannot be inspected this way.** Telegram allows one reader at a time. If you pointed this bot at your own webhook, enter the chat ID by hand.

To find the ID by hand, post a message in the chat and open `https://api.telegram.org/bot<your-token>/getUpdates`, then read `result[].message.chat.id`. Group and channel IDs are negative (`-1001234567890`). A one-to-one chat is positive. Public channels can use `@channelusername` instead.

| Field         | Notes                                                         |
| ------------- | ------------------------------------------------------------- |
| **Bot token** | From @BotFather. Write-only: never returned after saving.     |
| **Chat ID**   | `-1001234567890`, or `@channelusername` for a public channel. |

When you save, Maple verifies the token and checks that the bot can reach the chat. A valid token pointed at a group the bot was never added to fails at save time, not at the first real alert.

Alerts arrive as a formatted message with **Open in Maple** and **Ask Maple AI** buttons, and the alert chart as the message preview when one is available.

## Webhook

POST a signed JSON payload to any HTTP endpoint you control. Use it for custom routing, on-call tools without a native integration, or your own automation.

- Maple sends a JSON body describing the rule, the observed value, and links back into Maple.
- Set an optional **signing secret** to receive an `x-maple-signature` HMAC-SHA256 header, so your endpoint can verify the payload came from Maple.
- Your endpoint should respond with a `2xx` status. Any other status is a delivery failure and is shown on the destination.

The payload, headers, signature verification and retry behavior are in the [alert webhooks reference](/docs/reference/webhooks).

## Hazel

Connect [Hazel](https://hazel.sh/docs/integrations/maple) through OAuth and pick a workspace channel to route alerts into. See Hazel's [Maple integration guide](https://hazel.sh/docs/integrations/maple).

## Next steps

- [Alert rules](/docs/alerting/alert-rules): attach destinations to a rule.
- [Incidents](/docs/alerting/incidents): when Maple sends trigger, renotify and resolve.
- [Alert webhooks reference](/docs/reference/webhooks): the webhook payload.

---

# Uptime monitoring

> Alerting · https://maple.dev/docs/alerting/uptime-monitoring

Monitor availability in Maple: alert on traffic stopping and failing requests from your traces, and add HTTP checks from the OpenTelemetry Collector for endpoints and TLS certificates.

Maple monitors availability from two sources. Alert rules on your traces fire when a service stops receiving requests or starts failing them. HTTP checks from the OpenTelemetry Collector cover what real traffic cannot: endpoints with little traffic, and failures that happen before a request reaches your code, such as a DNS change or an expired certificate.

Maple does not run hosted probes. The checks run in a Collector you operate.

> **Recommended reading first.** [Uptime monitoring with traces and HTTP checks](/guides/uptime-monitoring) is the background for this page: which failures a probe catches, which ones only traces see, and when a probe is worth adding. This page is the setup.

## Prerequisites

- Traces arriving from the services you want to watch.
- At least one [notification destination](/docs/alerting/notification-destinations).
- For HTTP checks: the [OpenTelemetry Collector Contrib](https://github.com/open-telemetry/opentelemetry-collector-releases/releases) distribution (`otelcol-contrib`) on a host outside the system it checks, and a private ingest key (`maple_sk_…`) from **Settings → Ingestion**.

## Alert on real traffic

1. Open **Alerts**, click **New rule** and pick the **Throughput drop** template.
2. Under **Scope**, select the service. The template sets **Min samples** to 0, so a window with no requests counts as zero and fires.
3. Set **Threshold** below the service's quietest five minutes on a normal day.
4. Pick a range above the chart to replay the rule. The shaded periods are where it would have held an incident open.
5. Attach a destination and click **Create rule**.

<figure class="shot">
  <img src="/screenshots/docs/uptime-monitoring-01-throughput-rule.webp" alt="The alert rule form with the Throughput signal selected, condition below 100, the checkout service in scope, and Evaluation timing opened to show a 5 minute window and Min samples 0." loading="lazy" />
  <figcaption>A Throughput drop rule scoped to one service. Evaluation timing is opened to show Min samples at 0.</figcaption>
</figure>

<figure class="shot">
  <img src="/screenshots/docs/uptime-monitoring-02-throughput-preview.webp" alt="The rule preview chart over the last hour: throughput near 200 requests per window, a drop to zero for about 15 minutes, and a shaded band marked Would have fired 1 time, longest 25 minutes." loading="lazy" />
  <figcaption>The rule replayed over the last hour. Requests stopped for about 15 minutes.</figcaption>
</figure>

Add the **High error rate** template for failing requests and **Low Apdex score** for slow ones. [Alert rules](/docs/alerting/alert-rules) describes every field.

A throughput rule needs traffic to drop from. For a service that sits idle for hours, use an HTTP check.

## Add HTTP checks

The Collector's `http_check` receiver requests a list of URLs on an interval and reports each result as metrics. Save this as `config.yaml`:

```yaml
receivers:
    http_check:
        collection_interval: 60s
        metrics:
            httpcheck.tls.cert_remaining:
                enabled: true
        targets:
            - method: GET
              endpoints:
                  - https://example.com
                  - https://api.example.com/health

processors:
    resource/uptime:
        attributes:
            - key: service.name
              value: uptime-checks
              action: upsert
    batch: {}

exporters:
    otlphttp/maple:
        endpoint: https://ingest.maple.dev
        compression: gzip
        headers:
            x-maple-ingest-key: ${env:MAPLE_INGEST_KEY}

service:
    pipelines:
        metrics/uptime:
            receivers: [http_check]
            processors: [resource/uptime, batch]
            exporters: [otlphttp/maple]
```

Start the Collector with your key in the environment:

```bash
MAPLE_INGEST_KEY=YOUR_INGEST_KEY otelcol-contrib --config config.yaml
```

Notes on the config:

- **Older Collector releases name the receiver `httpcheck`.** Use that name in both places if your release rejects `http_check`.
- **`resource/uptime` sets `service.name`.** The checks appear in Maple under the service `uptime-checks`.
- **EU organizations** use `https://ingest.eu.maple.dev` as the endpoint.
- **Run the Collector outside the system it checks.** On the same machine as the service, it goes down with it.

## What the receiver reports

| Metric                         | Value                                                                                                                                            |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `httpcheck.status`             | One data point per status class (`1xx` to `5xx`): 1 for the class the response matched, 0 for the rest. A failed connection reports 0 for every class. |
| `httpcheck.error`              | 1 when the request failed before a response arrived, with the reason in `error.message`.                                                         |
| `httpcheck.duration`           | Total request time in milliseconds.                                                                                                              |
| `httpcheck.tls.cert_remaining` | Seconds until the certificate expires. Off unless enabled, as in the config above.                                                               |

Every data point carries the checked address in `http.url`.

## Alert when an endpoint stops answering

1. Open **Metrics** and open `httpcheck.status`.
2. Set **Aggregate** to `max`, **Where** to `attr.http.status_class = "2xx"`, and **Group by** to `attr.http.url`. Keep the `attr.` prefix on the filter, or it is dropped when the chart becomes a rule.
3. Click **Create alert**.
4. Set **Condition** to `<` and **Threshold** to `1`.
5. Open **Evaluation timing**. Set **Window (min)** to 2 and **Min samples** to 1. A two-minute window holds two data points per URL, so the default of 50 would skip every check.
6. Leave **Breaches to fire** at 2.
7. Attach a destination and click **Create rule**.

The rule reads: no check of this URL returned a 2xx in the last two minutes. It opens one incident per URL, about three minutes into an outage, and one failed check on its own does not fire it.

<figure class="shot">
  <img src="/screenshots/docs/uptime-monitoring-03-httpcheck-metric.webp" alt="The metric page for httpcheck.status with Aggregate max, the filter attr.http.status_class equals 2xx and Group by attr.http.url. The chart shows two URLs at 1, and one of them at 0 for five minutes." loading="lazy" />
  <figcaption>Step 2: the check charted per URL. One endpoint returned 503 for five checks while the other stayed up.</figcaption>
</figure>

<figure class="shot">
  <img src="/screenshots/docs/uptime-monitoring-04-httpcheck-rule.webp" alt="The Signal and threshold section of the alert rule form: a Metrics query on httpcheck.status filtered to the 2xx status class and grouped by attr.http.url, condition below 1, window 2 minutes, breaches to fire 2, Min samples 1." loading="lazy" />
  <figcaption>Steps 4 to 6: the condition, the window and Min samples.</figcaption>
</figure>

<figure class="shot">
  <img src="/screenshots/docs/uptime-monitoring-05-httpcheck-preview.webp" alt="The rule preview chart over the last 30 minutes, grouped by URL. One URL drops from 1 to 0 for about five minutes inside a shaded band marked Would have fired 1 time, longest 8 minutes." loading="lazy" />
  <figcaption>The rule replayed over the last 30 minutes. It would have fired once, for the endpoint that returned 503.</figcaption>
</figure>

## Alert when the checks stop arriving

A grouped rule skips a window with no data, so a stopped Collector would go unnoticed. Create a second rule on `httpcheck.status` with no group-by:

- **Aggregate** `count`, **Condition** `<`, **Threshold** `1`
- **Min samples** 1
- **Alert when there is no data** switched on

## Alert before a certificate expires

Create a rule on `httpcheck.tls.cert_remaining`:

- **Aggregate** `min`, **Group by** `attr.http.url`
- **Condition** `<`, **Threshold** `1209600` (14 days in seconds)
- **Min samples** 1

## Verify

Open **Metrics** and search for `httpcheck`. The metrics appear after the first check, emitted by the service `uptime-checks`. Open `httpcheck.status` and group by `attr.http.url` to see one series per target.

## Limits

- One Collector probes from one place, through one DNS resolver. It cannot show that users in another region are cut off, and a network problem next to the Collector looks the same as an outage.
- There is no status page.

For probes from several regions, a public status page, or an availability report for customers, run a dedicated uptime product next to Maple.

## Troubleshooting

- **No `httpcheck` metrics appear.** Check the Collector's output for exporter errors. A `401` means the ingest key is wrong or was rotated.
- **The alert form warns "Unsupported metrics filter ignored".** The filter key is missing the `attr.` prefix. Use `attr.http.status_class`.
- **Every check of the rule is skipped.** The window holds fewer data points than **Min samples**. Set it to 1.
- **The rule fires on a single failed check.** **Aggregate** is `min`. With `max`, every check in the window has to fail.

## Next steps

- [Alert rules](/docs/alerting/alert-rules): every signal, field and template.
- [Incidents](/docs/alerting/incidents): what happens after a rule fires.
- [Metrics](/docs/explore/metrics): chart `httpcheck.duration` and the other check metrics.
- [Uptime monitoring with traces and HTTP checks](/guides/uptime-monitoring): the reasoning behind this setup.

---

# Prometheus scraping

> Integrations · https://maple.dev/docs/integrations/prometheus

Point Maple at any Prometheus exposition endpoint. Maple scrapes it on a schedule, converts the samples to OpenTelemetry metrics, and records the health of every scrape.

Maple can scrape any endpoint that serves the Prometheus or OpenMetrics text format. You add the endpoint as a scrape target. Maple polls it at the interval you choose, converts the samples to OpenTelemetry metrics, and ingests them like your own OTLP traffic. Scraped metrics appear in the [metrics explorer](/docs/explore/metrics), dashboards, and alert rules, and each target keeps a history of its scrapes.

## Prerequisites

- An endpoint reachable from the public internet that serves `/metrics` in the Prometheus exposition format.

## Add a scrape target

Open **Integrations → Prometheus** in Maple and click **Add Target**. The **Add Scrape Target** dialog has these fields:

| Field                         | Notes                                                                                                                                |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| **Name**                      | Display name. Used as the service name when **Service Name** is empty.                                                               |
| **Service Name**              | Optional. Sets `service.name` on the resource and the `job` attribute on every data point.                                           |
| **URL**                       | Full endpoint URL, for example `https://myapp.com:9090/metrics`. Loopback, private-range, and cloud-metadata addresses are rejected. |
| **Scrape Interval (seconds)** | 5 to 300. Default 15.                                                                                                                |
| **Authentication**            | **None**, **Bearer Token** (sent as `Authorization: Bearer …`), or **Basic Auth** (**Username** and **Password**).                   |

Credentials are encrypted at rest. The scraper receives the decrypted auth header from Maple's API for each run and sends it directly to your endpoint. Requests carry the user agent `maple-prometheus-scraper`.

Each request times out after the scrape interval minus one second, capped at 60 seconds.

You can also manage targets through the [REST API](/docs/reference/api) under `/v2/scrape_targets`, with endpoints to create, update, delete, probe, and list checks. The API accepts one extra field, `labels_json`: a JSON object of labels added to every sample (for example `{"cluster": "prod"}`). The keys `job` and `instance`, and any key starting with `maple_` or `__`, are reserved and rejected.

## How the data looks

- Metric names are kept as they appear in the exposition.
- Counters become cumulative monotonic sums. Gauges and untyped metrics become gauges. Histograms become OTLP histograms with the original bucket bounds.
- Summaries become three series: `<name>_sum` as a cumulative sum, `<name>_count` as a cumulative monotonic sum, and the quantiles as a gauge named `<name>` with a `quantile` attribute. Summary samples that are not finite numbers (such as a `NaN` quantile with no observations yet) are dropped.
- Every data point carries `job` (the service name) and `instance` (the host of the target URL), plus the target's labels and the sample's own labels.

## Verify

1. On the target row, click **Test**. Maple runs an immediate scrape and reports success or the exact failure: HTTP status, timeout, TLS or connection error.
2. Wait one scrape interval. The status badge changes from **No checks** to **Up**, and the row shows **Last scrape** with a relative time.
3. Open the target to see its check history. Each run lists **Time**, **State**, **Duration**, and **Samples**.
4. Search for one of your metric names in the [metrics explorer](/docs/explore/metrics).

## Troubleshooting

- **Status is Down.** Open the target. The error message and a **How to fix** hint explain the failure. A failed scrape never advances the last successful scrape time, so the data gap stays visible next to the error.
- **The URL is rejected.** The endpoint resolves to a loopback, private, or metadata address. Expose it through an authenticated public endpoint, or use the collector option below.
- **401 or 403.** Check the authentication type and credentials. Basic Auth sends the username and password exactly as entered.
- **The endpoint is only reachable inside your network.** Run an OpenTelemetry Collector inside the network with a `prometheus` receiver and an OTLP exporter pointed at Maple's [ingest endpoint](/docs/reference/ingest).

## Next steps

- [WarpStream](/docs/integrations/warpstream): scrape WarpStream Agents or the hosted Prometheus endpoint.
- [PlanetScale](/docs/integrations/planetscale): branch metrics discovered automatically.
- [Alert rules](/docs/alerting/alert-rules): alert on a scraped metric.

---

# WarpStream

> Integrations · https://maple.dev/docs/integrations/warpstream

Monitor WarpStream clusters in Maple. Scrape Agent /metrics endpoints directly, or pull consumer lag, request latency, and object-store health from WarpStream's hosted Prometheus endpoint.

WarpStream exposes Prometheus metrics in two places. Every Agent serves a local `/metrics` endpoint, and the WarpStream control plane offers a hosted, authenticated Prometheus endpoint per virtual cluster. Both work with [Maple's Prometheus scraping](/docs/integrations/prometheus). Pick the one that matches your network topology.

## Option A: Hosted Prometheus endpoint (recommended)

The control plane serves cluster health (consumer group lag, partition sizes, agent heartbeats) at a single internet-reachable URL. Maple can scrape it without network changes:

```
https://api.warpstream.com/api/v1/monitoring/prometheus/virtual_clusters/$VIRTUAL_CLUSTER_ID
```

1. In WarpStream, create a **read-only Agent Key** for the cluster (least privilege; an account-level API key also works if you want one key for several clusters).
2. In Maple, open **Integrations → WarpStream** (or **Prometheus**), click **Add Target**, and configure:
    - **URL**: the hosted endpoint above, with your virtual cluster ID (`vci_…`)
    - **Authentication**: **Basic Auth**, with **Username** `prometheus` and **Password** set to the API key
    - **Scrape Interval (seconds)**: 30 to 60 is enough for control-plane metrics

    To tag every series with a cluster name, set `labels_json` (for example `{"cluster": "prod-kafka"}`) through the [scrape targets API](/docs/integrations/prometheus#add-a-scrape-target).

See WarpStream's [Hosted Prometheus Endpoint docs](https://docs.warpstream.com/warpstream/agent-setup/monitor-the-warpstream-agents/hosted-prometheus-endpoint) for the metric set, which includes Tableflow and Schema Registry metrics on those cluster types.

## Option B: Scrape the Agents directly

Each WarpStream Agent serves the full agent-level metric set (request latency histograms, produce/fetch byte counters, object-store operation latency) on its internal port. The endpoint is enabled by default and has no authentication:

```
http://$AGENT_IP:8080/metrics
```

Because Agents run inside your VPC under WarpStream's BYOC model, Maple's scraper can only reach them if you expose the endpoint (for example through an internal load balancer with auth, which you can pair with Maple's **Bearer Token** or **Basic Auth** options). If the Agents are fully private, run an OpenTelemetry Collector next to them with a `prometheus` receiver scraping `:8080/metrics` and an OTLP exporter pointed at Maple's [ingest endpoint](/docs/reference/ingest) instead.

Add one Maple target per agent (or per load-balanced agent pool), with the agent host in **URL**. Every series already carries an `instance` attribute with the target host. To add cluster or agent names, set `labels_json` such as `{"cluster": "...", "agent": "..."}` through the API.

## Metrics worth alerting on

All WarpStream metrics carry the `warpstream_` prefix. From WarpStream's [Important Metrics and Logs](https://docs.warpstream.com/warpstream/agent-setup/monitor-the-warpstream-agents/important-metrics-and-logs):

| Metric                                                                 | Why it matters                                                     |
| ---------------------------------------------------------------------- | ------------------------------------------------------------------ |
| `warpstream_consumer_group_lag`                                        | Consumer lag in offsets. The main "are we keeping up" gauge.       |
| `warpstream_agent_kafka_request_latency`                               | Produce/fetch latency histogram by request type.                   |
| `warpstream_agent_kafka_request_outcome`                               | Success vs. error counters per Kafka request type.                 |
| `warpstream_blob_store_operation_latency`                              | Object-store PUT/GET health. Check this first when latency spikes. |
| `warpstream_agent_control_plane_operation_latency`                     | Agent ↔ control-plane RPC health.                                  |
| `warpstream_topics_count` / `warpstream_partitions_count` (+ `_limit`) | Headroom against cluster limits.                                   |

Once samples arrive, build dashboards and alert rules on these like any other Maple metric. For example, alert when `warpstream_consumer_group_lag` stays above a threshold for 5 minutes.

## Troubleshooting

- **401 from the hosted endpoint.** The Basic Auth username must be exactly `prometheus`. The password is the API or Agent key.
- **Timeouts scraping Agents.** Confirm the Agent's internal port (8080 by default) and that the endpoint is reachable from the internet. `curl $AGENT_IP:8080/v1/status` should return `OK`.
- Use the target's **Test** button in Maple to see the exact upstream error.

---

# PlanetScale

> Integrations · https://maple.dev/docs/integrations/planetscale

Connect a PlanetScale organization to Maple. Maple discovers every database branch's metrics endpoint, scrapes connections, WAL size, and pod CPU, and adds your databases to the service map.

PlanetScale publishes Prometheus metrics per database branch behind a [service-discovery API](https://planetscale.com/docs/vitess/integrations/prometheus). One organization endpoint returns the current list of branch metrics targets, and that list changes as branches are created and deleted. You connect the organization once. Maple runs the discovery call, scrapes every branch it returns, and refreshes the branch list every 10 minutes.

Setup has two parts. An OAuth authorization covers the management plane: database inventory, query insights, and webhooks. PlanetScale serves branch metrics only to service tokens, so you add a service token to turn on metrics.

## Prerequisites

- Permission in the PlanetScale organization to authorize an OAuth application and create a service token.

## 1. Authorize Maple

Open **Integrations → PlanetScale** in Maple and click **Connect PlanetScale**. A popup takes you to PlanetScale to authorize Maple's OAuth application. You can revoke access from PlanetScale at any time.

- If the authorization covers one organization, Maple binds it automatically.
- If it covers several, pick one. You can also set **Only these branches (optional)** and **Exclude branches (optional)**. Both take glob patterns, where `*` matches any run of characters and `?` matches exactly one (for example `pr-*, preview-*`).

To switch organizations later, click **Change organization**.

The OAuth application needs `read_databases` for the database inventory, the service map, and query insights. If it is missing, the **Permissions** step shows **Reauthorize with read_databases**.

## 2. Add a service token for branch metrics

After authorization the card shows a four-step checklist: **Authorization**, **Permissions**, **Branch metrics access**, and **Metrics arriving**. The **Branch metrics access** step asks for a service token. If the OAuth authorization already covers the metrics endpoints, this step completes without one.

1. Click **Create a token in PlanetScale**. Create a service token with the `read_metrics_endpoints` organization permission. No other permission is needed.
2. Paste the **Service token ID** and **Service token secret** into Maple.
3. Click **Enable metrics**.

The secret is encrypted at rest. Maple derives the discovery URL (`https://api.planetscale.com/v1/organizations/{org}/metrics`), authenticates discovery and every scrape with the token, and scrapes each branch as its own target. New branches are picked up on the next discovery refresh. Deleted branches stop being scraped.

To replace the token later, click **Rotate token**.

## 3. Optional: webhooks

Under **Webhooks**, click **Show setup** to reveal an endpoint URL and signing secret. Register them in each database's webhook settings on PlanetScale. Only Maple organization admins can view the secret.

## Verify

1. The **Metrics arriving** step changes from **Waiting for the first scrape.** to done, and the checklist collapses to a single health row reading **Streaming branch metrics from** your organization. The card polls while it waits, so you do not need to reload.
2. Open **Infrastructure → PlanetScale** to see your databases and branches.
3. In the [metrics explorer](/docs/explore/metrics), search for `planetscale_` to see branch series.

## What you get

Each branch series carries PlanetScale's discovery labels. The most useful is `planetscale_database_branch_id`, which keys every series to a branch. Highlights from the metric set ([Postgres](https://planetscale.com/docs/postgres/monitoring/prometheus-postgres), [Vitess](https://planetscale.com/docs/vitess/integrations/prometheus)):

| Metric                                         | What it tells you                                          |
| ---------------------------------------------- | ---------------------------------------------------------- |
| `planetscale_postgres_connection_state`        | Connections by state (active, idle, idle in transaction).  |
| `planetscale_edge_postgres_active_connections` | Active connections at the edge.                            |
| `planetscale_postgres_wal_size_bytes`          | WAL size. An early warning for replication and disk usage. |
| `planetscale_pgbouncer_current_connections`    | PgBouncer pool utilization.                                |
| `planetscale_pods_cpu_util_percentages`        | CPU per pod backing the branch.                            |
| `planetscale_vtgate_total_pods`                | (Vitess) vtgate pods per availability zone.                |

Group dashboards and [alert rules](/docs/alerting/alert-rules) by `planetscale_database_branch_id`. For example, alert when WAL size passes a threshold or active connections approach your pool limit.

## Troubleshooting

- **Metrics arriving shows "The last scrape failed."** The health row shows the error. A failure for one branch is prefixed with `[branch:<id>]`.
- **401 or 403 on discovery or scrapes.** The service token was deleted or lacks `read_metrics_endpoints`. Click **Rotate token** and paste a new one.
- **Authorization shows as revoked.** Click **Reconnect** to authorize again.
- **Metrics stopped arriving.** The **Metrics arriving** step reports a stall after three scrape intervals without data. Check the error on the health row.
- **A discovery call fails.** Maple keeps scraping the last known branch list and shows the discovery error on the target, so branch metrics do not drop out during a PlanetScale API outage.
- **Branch filters did not apply.** Changes take effect on the next scrape because the cached branch list is cleared on save.

## Next steps

- [Service map](/docs/explore/service-map): PlanetScale databases linked to the services that query them.
- [Prometheus scraping](/docs/integrations/prometheus): how scraped samples become OpenTelemetry metrics.
- [Alert rules](/docs/alerting/alert-rules): alert on branch metrics.

---

# GitHub

> Integrations · https://maple.dev/docs/integrations/github

Install the Maple GitHub App to sync repositories and commit history, resolve commit SHAs in your telemetry, trace your GitHub Actions runs, and give the MCP server read access to your source code.

The Maple GitHub App syncs your repositories and their commit history into Maple. You install the app, choose which repositories to share, and pick one branch to track per repository. Maple backfills the last 90 days of commits on that branch and keeps it current through GitHub webhooks.

Once connected, the integration does three things:

- **Commit context in telemetry.** When your services report the `vcs.ref.head.revision` resource attribute, Maple resolves the SHA against synced commits. The trace view and service pages show a commit hover card with the author, message, and a link to GitHub.
- **CI traces.** Every finished GitHub Actions run becomes a trace under the `github-actions` service: the run is the root span, each job a child span, and each step a span inside its job. No workflow changes are needed. Each job's log comes along as logs on those spans.
- **Source code tools for the MCP server.** The [MCP server](/docs/reference/mcp) gains three read-only tools that reach your repositories through the app installation: `list_source_repositories`, `search_source_code` (GitHub code search on the default branch), and `read_source_file` (a line range at a branch, tag, or commit SHA). An assistant can go from an exception in a trace to the code that raised it.

## Prerequisites

- Permission to install GitHub Apps on the personal account or organization that owns the repositories.

## 1. Connect your GitHub account

Open **Integrations → GitHub** in Maple and click **Connect GitHub**. A popup walks you through installing the **Maple GitHub App**. During install you choose the scope:

- **All repositories**: Maple syncs every repository the account can access, including ones added later.
- **Only select repositories**: Maple syncs only the repositories you pick. You can change the selection from GitHub at any time.

When the install completes, the popup closes and the card shows **Connected** with the account and repository scope. Maple discovers repositories and queues a backfill for each. Each row moves from **Queued** to **Syncing** to **Synced**.

## 2. Pick a tracked branch per repository

Each repository tracks one branch. It starts as the repository's default branch, and Maple syncs commits only from that branch. Use the **Tracked branch** selector on a repository row to change it.

Changing the tracked branch is destructive. Maple deletes the repository's synced commits and re-syncs the last 90 days from the new branch. The app asks you to confirm before it applies the change.

## How syncing works

- **90-day backfill** on connect and whenever you change the tracked branch.
- **Webhooks** deliver pushes to a tracked branch as they happen. Force-pushes are reconciled. Repositories added to or removed from the installation, and a suspended app, are picked up automatically.
- **A scheduled reconcile** runs every 12 hours for every installation, so a missed webhook does not leave gaps.

## CI traces

Maple receives GitHub's `workflow_run` and `workflow_job` webhooks for the repositories the app can see and writes one trace per run attempt. A re-run gets its own trace.

- **Spans:** the run is a `SERVER` span named `RUN <workflow>` (for example `RUN CI`), from when it started to when it finished. A job that waited a second or more for a runner spans from when it was queued to when it finished, with a `Waiting for a runner` child span for that wait; otherwise it starts when the runner picked it up, and a job that never started has no length. The wait is always on the job span as `maple.cicd.queue_duration_ms`. Each step is a span inside its job, named without the commit a pinned action points at (`Run actions/checkout`, not `Run actions/checkout@3d3c42e…`). Steps that never ran are left out.
- **Status:** failed and timed-out runs, jobs, and steps are `Error` with `error.type` set, passing ones are `Ok`, and cancelled or skipped ones carry no status.
- **Failure reasons:** for a failed job, Maple reads the annotations GitHub recorded on it (test failures, compiler errors). The most specific one becomes the job span's status message and `maple.cicd.failure.reason`, and up to 20 of them, failures first, are `cicd.annotation` events on the job span with `code.file.path` and `code.line.number`.
- **Attributes:** spans follow the OpenTelemetry CI/CD conventions (`cicd.pipeline.name`, `cicd.pipeline.run.id`, `cicd.pipeline.action.name`, `cicd.pipeline.result`, `cicd.pipeline.task.name`, `cicd.pipeline.task.run.id`, `cicd.pipeline.task.run.result`, `cicd.pipeline.task.run.url.full`, `cicd.worker.name`) plus `vcs.ref.head.name`, `vcs.ref.head.revision`, and `vcs.change.id` (the pull request number). Jobs and steps are both tasks; a step's URL links to its line in the job log. The resource carries `vcs.repository.name`, `vcs.owner.name`, and `vcs.repository.url.full`, so you can filter CI traces by repository.

CI traces and logs are on by default. CI spans and logs are ingested like any other telemetry and count toward your usage. To stop both, an organization admin turns off **CI traces and logs** on **Integrations → GitHub**; runs that finish while it is off are not recorded.

## CI logs

Maple also downloads each finished job's log from GitHub and writes every line as a log record under the `github-actions` service. Nothing runs inside your workflows: the app's existing **Actions: Read** permission covers the download.

- **Linked to steps:** each record carries the job's trace id and the span id of the step that was running when the line was written, so a step's output shows up with that step in the trace view. Lines from before the first step belong to the job span.
- **Timestamps** come from the runner's own per-line stamps. A line without one, such as the rest of a multi-line message, takes the stamp of the line before it.
- **Severity:** `##[error]` lines (including `::error::` workflow commands) are `ERROR`, `##[warning]` lines are `WARN`, `##[debug]` lines are `DEBUG`, and everything else is `INFO`. The runner's markers and color codes are removed from the text, and blank lines are dropped.
- **Attributes:** `cicd.pipeline.name`, `cicd.pipeline.run.id`, `cicd.pipeline.task.name` and `cicd.pipeline.task.run.id` (the same as on the step span, or the job's for job-level lines), `maple.cicd.step.number`, `maple.cicd.job.name`, `vcs.ref.head.name`, and `vcs.ref.head.revision`.
- **Size:** Maple keeps the first 5 MB of a job's log and cuts lines longer than 8,000 characters. A cut log ends with a `WARN` record that links to the full log on GitHub.
- **Secrets:** GitHub masks registered secrets as `***` before Maple reads the log. Values a job prints that were never registered as secrets are kept as written.

Logs arrive a few seconds after their job's spans, since GitHub publishes a log shortly after the job finishes.

## Verify

1. On **Integrations → GitHub**, every repository row reads **Synced** with a recent sync time. The card polls while backfills run, so you do not need to reload.
2. Deploy a service that sets `vcs.ref.head.revision` to the commit SHA it was built from. See [OpenTelemetry conventions](/docs/concepts/otel-conventions).
3. Open a trace from that service. The short SHA in the trace header shows the commit author and message on hover.

## Managing the connection

- **Refresh** reloads repository status.
- **Manage** reopens the GitHub App install screen so you can add or remove repositories.
- **Needs attention** lists repositories whose access was removed on GitHub. Their commit history is kept. Re-enable them in the [GitHub App settings](https://github.com/settings/installations) to resume syncing, or click **Delete** to remove the repository and its synced commits from Maple.
- **Disconnect** removes the app connection and permanently deletes all synced repositories and their commit history from Maple. Reconnecting later re-syncs from scratch.

## Troubleshooting

- **A repository shows Sync failed.** Click **Refresh**. Failed backfills are retried automatically. The row shows the underlying error, most often a GitHub rate limit (retried once the limit resets) or revoked access.
- **No repositories appear after connecting.** Discovery and the first backfill run in the background. The card polls and fills in as they complete.
- **Commits don't show up for a repository.** Confirm the tracked branch is the one your commits land on. Only the tracked branch is synced.
- **Traces show a SHA without a hover card.** The SHA must be a full 40-character commit on a synced tracked branch within the synced history.
- **No CI traces appear.** Traces are written when a run or job finishes, so a run in progress has none yet. Check that **CI traces and logs** is on under **Integrations → GitHub**. If finished runs are still missing, open the [GitHub App settings](https://github.com/settings/installations) and accept any pending permission request from the Maple app.
- **CI traces appear but no CI logs.** A job whose steps never ran has no log to send, and a log GitHub takes too long to serve is skipped.
- **`search_source_code` finds nothing.** GitHub code search indexes only the default branch and matches whole tokens. Search for exact identifiers or exception text.

## Next steps

- [OpenTelemetry conventions](/docs/concepts/otel-conventions): the `vcs.*` resource attributes Maple reads.
- [MCP server](/docs/reference/mcp): connect an assistant and use the source code tools.
- [Traces](/docs/explore/traces): where commit context appears.

---

# Slack

> Integrations · https://maple.dev/docs/integrations/slack

Add the Maple bot to a Slack workspace. Mention it in a channel to ask about your traces, logs, metrics, and errors, and use the same workspace as an alert destination.

The Slack integration adds the Maple bot to a Slack workspace and links that workspace to your Maple organization. Mention the bot in a channel and it answers in a thread, using the same assistant as Maple's chat, with access to your organization's traces, logs, metrics, and errors. Once the workspace is linked, its channels are also available as [alert destinations](/docs/alerting/notification-destinations).

## Prerequisites

- You are an admin of the Maple organization. Only organization admins can connect, change, or disconnect Slack workspaces.
- You can install apps in the Slack workspace.

## Install the bot

1. Open **Integrations → Slack** in Maple.
2. Click **Add to Slack**. Maple redirects you to Slack.
3. Choose the workspace and approve the install. Slack returns you to Maple, which shows **Connected** with the workspace name.

To link more workspaces to the same organization, click **Add another workspace**.

A Slack workspace can be linked to one Maple organization at a time. Enterprise Grid org-wide installs are not supported. Install the app into a single workspace instead.

### Permissions

The bot requests these Slack scopes. It requests no user token.

| Scope                                                              | Used for                                           |
| ------------------------------------------------------------------ | -------------------------------------------------- |
| `app_mentions:read`                                                | Seeing when someone mentions the bot.              |
| `channels:history`, `groups:history`, `im:history`, `mpim:history` | Reading the thread it is replying in, for context. |
| `chat:write`                                                       | Posting and updating its replies.                  |
| `chat:write.public`, `channels:read`, `groups:read`                | Listing channels and posting alerts to them.       |

## Ask the bot a question

1. Invite the bot to a channel: `/invite @Maple`.
2. Mention it with a question, for example `@Maple why is checkout slow?`.

The bot replies in a thread on your message and updates that reply as the answer comes in. It can post charts.

- **Follow-ups**: reply in the bot's thread within 24 hours to continue without mentioning it again.
- **Context**: the bot reads up to 20 earlier messages in the thread.
- **One question at a time**: while an answer is in progress, new messages in the same conversation get a note asking you to wait.
- Messages that do not mention the bot and are not in one of its threads are ignored.

The bot has no slash commands.

### What the bot can do

The bot reads your Maple data the same way the [MCP server](/docs/reference/mcp) does: traces, logs, metrics, services, errors, dashboards, and alerts. When a request would change something, such as creating a dashboard, editing an alert rule, or updating an error issue, the bot does not make the change on its own. It posts an **Approve** prompt naming the change, and the change runs only after someone approves it.

### Identity

The workspace is linked to the organization, not to individual people. No Slack account is tied to a Maple user. As a result:

- Anyone in a channel with the bot can ask about the linked organization's data.
- Anyone who can see a proposed change can approve it, and the approved change runs with organization-level permissions.

Invite the bot only to channels whose members should see your data and be able to make these changes.

## Slack as an alert destination

After the workspace is linked, Slack alerts go through the same bot. There is no separate webhook setup.

1. Open the alert destination settings. See [Notification destinations](/docs/alerting/notification-destinations).
2. Pick the tile named after your Slack workspace.
3. Choose a channel. Private channels are marked **(private)**.

The bot can post to public channels it has not been invited to. To post to a private channel, invite the bot first.

The two features are independent. You can link a workspace only for alerts and never mention the bot, or use the bot without routing any alerts to Slack.

## Verify

1. **Integrations → Slack** shows your workspace with **Connected**.
2. In a channel with the bot, mention it with a question such as `@Maple which services had errors in the last hour?`. A threaded reply appears.

## Disconnect

On **Integrations → Slack**, click **Disconnect** next to the workspace and confirm. The workspace is unlinked from your organization immediately. To remove the bot from the workspace itself, uninstall the app in Slack. Uninstalling the app in Slack also unlinks the workspace in Maple.

## Troubleshooting

- **"The install link expired."** Start the install again from **Add to Slack**.
- **"That workspace is already linked to a different Maple organization."** Disconnect it from the other organization first.
- **"Only organization admins can connect Slack."** Ask an organization admin to install it.
- **The bot does not reply.** Make sure you mentioned it, that it is in the channel (`/invite @Maple`), and that the workspace shows **Connected** in Maple.
- **"Maple's agent can't be reached from here right now."** The assistant is temporarily unavailable. Try again later.
- **The channel list for alerts is empty or asks you to reinstall.** The workspace was linked before the alert scopes were added. Click **Open Slack integration** and reinstall with **Add to Slack**.

## Next steps

- [Notification destinations](/docs/alerting/notification-destinations): route alerts to Slack channels.
- [Alert rules](/docs/alerting/alert-rules): decide what gets sent.
- [MCP server](/docs/reference/mcp): the same data in your editor or coding agent.

---

# Discord

> Integrations · https://maple.dev/docs/integrations/discord

Add the Maple bot to a Discord server. Mention it in a channel to ask about your traces, logs, metrics, and errors, and post alerts to the server's channels.

The Discord integration adds the Maple bot to a Discord server and links that server to your Maple organization. Mention the bot in a channel and it answers in a thread, using the same assistant as Maple's chat. Once the server is linked, its channels are also available as [alert destinations](/docs/alerting/notification-destinations).

## Prerequisites

- You are an admin of the Maple organization.
- You have the **Manage Server** permission on the Discord server. Discord requires it to add an app.

## Add the bot to a server

1. Open **Integrations → Discord** in Maple.
2. Click **Add to Discord**. Maple redirects you to Discord.
3. Pick the server and approve the install. Discord returns you to Maple, which shows **Connected** with the server name.

To link more servers to the same organization, click **Add another workspace**. A server can be linked to one Maple organization at a time.

### Permissions

The bot asks for no member, role, channel, or moderation permissions.

| Permission                                      | Used for                                    |
| ----------------------------------------------- | ------------------------------------------- |
| View Channels, Read Message History             | Reading the conversation it is replying in. |
| Send Messages, Embed Links                      | Posting answers, charts, and alerts.        |
| Create Public Threads, Send Messages in Threads | Answering in a thread on your message.      |
| Add Reactions                                   | Marking a request as picked up.             |

## Link your Discord account

The bot answers only people who have linked their Discord account to a Maple user. Each person links once.

1. Open **Integrations → Discord** in Maple.
2. Under **Your Discord account**, click **Link your account** and approve in Discord.

The link asks Discord for your user ID and nothing else. It adds no permissions on any server. You can only link your own account, and only members of the Maple organization get answers.

## Ask the bot a question

Mention the bot in any channel it can see, for example `@Maple why is checkout slow?`. If your account is not linked, the bot replies with a link to do it and reads nothing.

The bot starts a thread on your message and edits its reply there as the answer comes in. It reads the surrounding messages for context, so a follow-up like "and the payments call?" works.

- **Follow-ups**: reply inside the bot's thread. No mention is needed. Replies from people who have not linked are ignored.
- **New conversation**: mention the bot in another channel.
- **One question at a time**: a message sent while an answer is still being written gets a note asking you to wait. It is not queued.

### Answers run as your Maple user

The bot reads your Maple data the same way the [MCP server](/docs/reference/mcp) does, as your Maple user with your current roles. If you leave the organization, the bot stops answering you even while the link exists.

When a request would change something, such as creating a dashboard or editing an alert rule, the bot posts a card with **Run** and **Skip** buttons instead of acting. Any linked person can press them. The change runs as that person's Maple user, so a change that needs an admin fails for a member, the same as in the app. The card then updates with the result.

Answers are posted in the channel, so everyone who can read the channel sees them. Add the bot only where the members should see your data.

This differs from [Slack](/docs/integrations/slack), which has no account linking: anyone in a Slack channel with the bot can ask, and anyone who can see a card can approve it.

## Discord as an alert destination

After the server is linked, alerts can go through the bot.

1. Open the alert destination settings. See [Notification destinations](/docs/alerting/notification-destinations).
2. Pick the tile named after your Discord server.
3. Choose a text or announcement channel the bot can see.

To post alerts to a server without installing the bot, use the [Discord webhook destination](/docs/alerting/notification-destinations#discord) instead.

## Verify

1. **Integrations → Discord** shows your server with **Connected**.
2. In a channel the bot can see, send `@Maple which services had errors in the last hour?`. A thread opens with the answer.

## Disconnect

On **Integrations → Discord**, click **Disconnect** next to the server and confirm. The server is unlinked immediately. To remove the bot from the server, kick it in Discord. Removing the bot in Discord also unlinks the server in Maple.

To unlink your own account, click **Unlink** under **Your Discord account**.

## Troubleshooting

- **"This workspace isn't connected to a Maple organization yet."** The bot is in the server but the server is not linked. An admin runs **Add to Discord** from Maple.
- **"Only organization admins can connect Discord."** Ask an organization admin to install it.
- **"Link your Discord account to Maple to ask it questions."** Link your account in Maple, then mention the bot again.
- **"Link your Discord account to Maple to approve this."** Link your account in Maple, then press **Run** again.
- **"The Maple account this chat account is linked to is no longer in this organization."** Ask an admin to add you back, or link a different Maple account.
- **The bot does not reply.** Make sure you mentioned it and that it can see the channel. Check the channel's permission overrides for the Maple role.

## Next steps

- [Notification destinations](/docs/alerting/notification-destinations): route alerts to Discord channels.
- [Alert rules](/docs/alerting/alert-rules): decide what gets sent.
- [MCP server](/docs/reference/mcp): the same data in your editor or coding agent.

---

# Cloudflare

> Integrations · https://maple.dev/docs/integrations/cloudflare

Connect a Cloudflare account with OAuth. Maple polls zone traffic, Workers, security, DNS, Queues, and Durable Objects analytics every 5 minutes and shows them under Infrastructure and on the service map.

The Cloudflare integration connects one or more Cloudflare accounts to Maple through OAuth. Maple polls Cloudflare's GraphQL Analytics API every 5 minutes and stores the results as OpenTelemetry metrics. You get zone traffic, cache, and latency; Workers invocations, errors, and CPU time; firewall and DNS activity; and Queues and Durable Objects stats. The data appears under **Infrastructure → Cloudflare**, on the service map, and in the [metrics explorer](/docs/explore/metrics). You do not need to set up Logpush or run an agent.

## Prerequisites

- You are an admin of the Maple organization.
- You can authorize OAuth applications on the Cloudflare account.

## Connect

1. Open **Integrations → Cloudflare** in Maple.
2. Click **Connect Cloudflare**. A Cloudflare consent screen opens in a popup.
3. Choose the accounts to share and approve. One authorization can cover several accounts.

The popup closes and the card lists your **Connected accounts**. Maple starts collecting right away.

### Permissions requested

| Scope                                                                                                                                            | Used for                                                                                      |
| ------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------- |
| `account-analytics.read`                                                                                                                         | Workers, Queues, and Durable Objects analytics.                                               |
| `analytics.read`                                                                                                                                 | Zone HTTP, firewall, and DNS analytics.                                                       |
| `zone.read`                                                                                                                                      | Listing your zones.                                                                           |
| `query-cache.read`                                                                                                                               | Listing Hyperdrive configs for the service map. Optional.                                     |
| `account-settings.read`, `workers-scripts.read`, `workers-scripts.write`, `workers-observability.write`, `workers-observability-telemetry.write` | Also requested on the consent screen. Analytics collection needs only the three scopes above. |

Maple also requests `offline_access` so the connection can refresh its token without you reconnecting.

## What Maple collects

Maple reads these Cloudflare datasets in 5-minute buckets: HTTP requests (with breakdowns by path, country, and client), Workers invocations, firewall events, DNS queries, Queue backlog and consumers, and Durable Object invocations. On connect it backfills the last 24 hours, or less if your Cloudflare plan keeps less history.

The data is stored as metrics named `cloudflare.*`, for example:

| Metric                                                                                                                                                | Service name                      |
| ----------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- |
| `cloudflare.http.requests`, `cloudflare.http.bytes`, `cloudflare.http.edge.ttfb`, `cloudflare.http.origin.duration`                                   | `cloudflare/<zone name>`          |
| `cloudflare.firewall.events`, `cloudflare.dns.queries`                                                                                                | `cloudflare/<zone name>`          |
| `cloudflare.worker.requests`, `cloudflare.worker.errors`, `cloudflare.worker.subrequests`, `cloudflare.worker.cpu_time`, `cloudflare.worker.duration` | `cloudflare-worker/<script name>` |
| `cloudflare.queue.backlog.messages`, `cloudflare.queue.consumer.concurrency`                                                                          | `cloudflare-queue/<queue id>`     |

Counters are delta sums per 5-minute bucket. Latency percentiles are gauges with a `quantile` attribute (`0.5`, `0.95`, `0.99`). Chart and alert on them like any other metric.

## Where the data shows

### Infrastructure → Cloudflare

- **Zones**: **Zone**, **Requests**, **Error rate**, **Cache hit**, **Bandwidth**, and **TTFB p99**, with more sort options (visits, TTFB p50, origin p99).
- **Workers**: **Script**, **Invocations**, **Error rate**, **CPU p99**, and **Duration p99**.
- **Platform**: Queues (backlog and consumers) and Durable Objects (requests and error rate).

Click a zone for its detail page:

- A stat rail with **Edge requests**, **5xx error rate**, **Bandwidth**, and **Visits**.
- Charts: **Requests by status class**, **Requests by cache status**, and **Latency percentiles**.
- **Security events by action** and a table of top security rules (**Rule**, **Source**, **Action**, **Host**, **Events**).
- **DNS queries by response code** and a table of query names with **Queries** and **NXDOMAIN** counts.

### Service map

When a Worker also sends traces, Maple matches the script to the traced service by service name or `faas.name`. The service's panel on the [service map](/docs/explore/service-map) then gets a **Cloudflare edge** section with edge-reported (unsampled) **Requests**, **Error Rate**, and CPU p99. Scripts without a matching traced service, and zones, do not create nodes of their own.

### Integrations → Cloudflare

The integration card shows each connected account, a **Zones** board with a status per zone (**Live**, **Issues**, **No data**, **Paused**, or **Disabled**) and 24-hour request counts, and a **Workers** card.

## Verify

Cloudflare publishes analytics in 5-minute batches and needs about 10 minutes before a batch is complete. The first numbers usually land within 15 minutes of connecting.

While you wait, **Infrastructure → Cloudflare** shows progress: **Finding your zones and Workers**, **Collecting your first Cloudflare data**, how many zones are reporting, and the backfill percentage. When data flows, it reads **Receiving Cloudflare data** and the zones table fills in.

## Troubleshooting

- **"No Cloudflare data has arrived" after 30 minutes.** Check that the zones received traffic in the last day and that the authorization included the accounts you expect. Reconnect to pick different accounts.
- **"Update access to collect analytics".** The connection predates a required scope. Click **Update access** and approve the new permissions.
- **"Cloudflare access was revoked".** Click **Reconnect**.
- **"Analytics isn't enabled for this zone in Cloudflare."** Enable analytics for the zone in Cloudflare.
- **"Some analytics aren't available on this Cloudflare plan."** Some datasets require a higher Cloudflare plan. Maple collects what your plan allows.
- **"This zone was removed from your Cloudflare account."** The zone no longer exists in the connected account.
- **A Worker has no Cloudflare edge section on the service map.** The Worker's traced `service.name` (or `faas.name`) must match its script name.

## Disconnect

On **Integrations → Cloudflare**, click **Disconnect** (or **Disconnect all** with several accounts). To change which accounts are connected, click **Reconnect** or **Edit accounts** and pick them again on Cloudflare's consent screen.

## Next steps

- [Service map](/docs/explore/service-map): Workers and Hyperdrive on the map.
- [Metrics](/docs/explore/metrics): chart any `cloudflare.*` metric.
- [Alert rules](/docs/alerting/alert-rules): alert on zone error rate or Worker errors.

---

# Google Cloud

> Integrations · https://maple.dev/docs/integrations/gcp

Connect a Google Cloud organization, folder or project with a script in Cloud Shell, by hand in the console, or with Terraform or Alchemy. Maple receives Cloud Logging entries through Pub/Sub and reads Cloud Monitoring metrics every 5 minutes.

The Google Cloud integration connects an organization, a folder or a single project to Maple. You choose what each connection collects:

| What to collect           | Collects                                                                                                                                                                                                                                                                       |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Log forwarding**        | A Log Router sink sends Cloud Logging entries through Pub/Sub to Maple. They appear in [Logs](/docs/explore/logs) under the service that wrote them.                                                                                                                           |
| **Metrics and resources** | Maple reads Cloud Monitoring metrics for Cloud Run, Cloud Functions, GKE, Compute Engine, Cloud SQL, Pub/Sub and HTTP(S) load balancers every 5 minutes, and lists the resources behind them every hour. The metrics are named `gcp.*` and work in dashboards and alert rules. |

There is no OAuth step and no service account key, and Maple gets no write access to Google Cloud. You set a connection up in one of three ways, chosen in the dialog's **Method** step. Each creates the same resources under the same names:

| Method                     | What you do                                                                                                                                                 |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Cloud Shell script**     | Paste a generated `gcloud` script into Cloud Shell. About a minute. This page covers it.                                                                    |
| **Manual setup**           | Create each resource in the Google Cloud console from a step list in the dialog. About 20 minutes. See [Manual setup](/docs/integrations/gcp-manual-setup). |
| **Infrastructure as code** | Copy a generated Terraform or Alchemy file into your own repo and apply it. See [Terraform and Alchemy](/docs/integrations/gcp-infrastructure-as-code).     |

If your workloads also send telemetry over OpenTelemetry, read [Google Cloud with OpenTelemetry](/docs/integrations/gcp-opentelemetry) first: it covers which logs to collect where.

## Before you start

You need to be an admin of the Maple organization, and in Google Cloud:

| You connect     | Roles you need                                                                                                                                      |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| A project       | Owner on the project.                                                                                                                               |
| A folder        | Owner on the host project. On the folder: Logs Configuration Writer for the log sink, and Folder IAM Admin for the read-only roles.                 |
| An organization | Owner on the host project. On the organization: Logs Configuration Writer for the log sink, and Organization Administrator for the read-only roles. |

The host project holds Maple's Pub/Sub topic, subscription and service accounts, and Google bills the Pub/Sub and Cloud Monitoring usage to it. A project is its own host project. For a folder or organization, name a shared operations project inside it that won't be deleted. **Metrics and resources** needs billing enabled on the host project.

| ID              | Where to find it                                                                                                                            |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| Project ID      | The console's project picker, or `gcloud projects list`. Not the project name or number.                                                    |
| Organization ID | **IAM & Admin → [Manage resources](https://console.cloud.google.com/cloud-resource-manager)**, or `gcloud organizations list`. Digits only. |
| Folder ID       | The same console page, or `gcloud resource-manager folders list --organization=ORGANIZATION_ID`. Digits only.                               |

## Connect

1. In Maple, open **Integrations → Google Cloud** and click **Connect Google Cloud**.
2. On **Scope**, choose **Organization**, **Folder** or **Project** and enter its ID, plus the host project for a folder or organization. An organization or folder covers every project in it, including new ones.
3. On **Data**, choose what to collect and, for logs, the [log filter](#log-filter). Click **Continue**: Maple creates the connection.
4. On **Method**, leave **Cloud Shell script** selected and click **Continue**. For the other methods, follow [Manual setup](/docs/integrations/gcp-manual-setup) or [Terraform and Alchemy](/docs/integrations/gcp-infrastructure-as-code) from here.
5. Click **Open Cloud Shell**, then **Copy script**, paste it into Cloud Shell and press Enter. It takes about a minute.
6. The dialog updates on its own and shows **Google Cloud matches this configuration** once the script reports back.

If a step fails, the script stops and prints what to do. Fix it and paste the script again: it continues where it stopped, and running it again is always safe.

Logs start arriving within about 10 minutes of the sink being created; entries written before then are not forwarded. The first metrics read lands within about 10 minutes, and Maple then reads every 5 minutes, about 5 minutes behind.

The script contains a secret that lets anyone send logs to your Maple organization. Don't share or commit it, and run it in Cloud Shell, where it stays out of shell history. If it leaks, disconnect and connect again to get a new one. Manual and infrastructure-as-code setups carry no secret: Pub/Sub signs each push instead.

### Roles for running the script

Owner on the host project is the simplest. Without it, these predefined roles together cover what the script does. Before it changes anything, the script checks the signed-in account's permissions and names any that are missing.

| On                                  | Role                                                                                                                                                                                             | For                                                   |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------- |
| The host project                    | Service Usage Admin (`roles/serviceusage.serviceUsageAdmin`)                                                                                                                                     | Switching on the APIs.                                |
| The host project                    | Project IAM Admin (`roles/resourcemanager.projectIamAdmin`)                                                                                                                                      | The role grants on the host project.                  |
| The host project                    | Pub/Sub Admin (`roles/pubsub.admin`)                                                                                                                                                             | Log forwarding: the topic and the subscription.       |
| The host project                    | Service Account Admin (`roles/iam.serviceAccountAdmin`)                                                                                                                                          | Metrics and resources: the read-only service account. |
| The project, folder or organization | Logs Configuration Writer (`roles/logging.configWriter`)                                                                                                                                         | Log forwarding: the sink.                             |
| The project, folder or organization | Project IAM Admin (`roles/resourcemanager.projectIamAdmin`), Folder IAM Admin (`roles/resourcemanager.folderIamAdmin`) or Organization Administrator (`roles/resourcemanager.organizationAdmin`) | Metrics and resources: the read-only roles.           |

### Change what a connection collects

Click **Configure** on the connection's row, change **Log forwarding**, **Metrics and resources** or the log filter, and click **Save and continue**. Then apply the change the way you set the connection up: run the script again, follow the dialog's console steps, or apply the regenerated file.

Turning something off takes effect in Maple at once. Google Cloud keeps the sink or the read-only account until the change is applied, and the connection says so until then. A connection collects at least one of the two; to stop collecting, [disconnect](#disconnect).

### Log filter

Platform, request and audit logs are forwarded. The default filter leaves out what costs a lot and says little:

| Left out                    | Reason                                                                                                                                                                                     |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Data Access audit logs      | They record every API read.                                                                                                                                                                |
| Load balancer health checks | Probes hit each backend every few seconds.                                                                                                                                                 |
| Kubernetes lease renewals   | Hundreds of audit entries a minute for an idle node.                                                                                                                                       |
| VM serial console output    | Thousands of lines of raw terminal output at boot.                                                                                                                                         |
| Application output          | What your code writes to stdout and stderr: send it over OpenTelemetry. See [Application output](/docs/integrations/gcp-opentelemetry#application-output) to forward it for some runtimes. |

The filter, one clause per line:

```text
NOT log_id("cloudaudit.googleapis.com/data_access")
AND NOT httpRequest.userAgent:"GoogleHC"
AND NOT protoPayload.methodName="io.k8s.coordination.v1.leases.update"
AND NOT logName:"serialconsole.googleapis.com"
AND NOT (log_id("run.googleapis.com/stdout") OR log_id("run.googleapis.com/stderr"))
AND NOT log_id("cloudfunctions.googleapis.com/cloud-functions")
AND NOT (resource.type="gae_app" AND (log_id("stdout") OR log_id("stderr")))
AND NOT resource.type="k8s_container"
```

For any other filter, paste the script into an editor, edit `LOG_FILTER` near its top and set `LOG_FILTER_MODE` to `set`; in the console or a file, edit the sink's filter. It uses the [Logging query language](https://cloud.google.com/logging/docs/view/logging-query-language). On a connection that already has a sink, the dialog starts on **Keep current filter**, which leaves your filter alone.

## Check that it works

**Integrations → Google Cloud** shows each connection with a status for **Logs** and one for **Metrics**, and a band with the fix when something needs attention. Once data arrives, the page also shows log and error counts for the last 24 hours, the workloads that reported metrics, and the resource inventory. Then:

1. Open [Logs](/docs/explore/logs) and look for a service named after one of your Cloud Run services, functions or instances.
2. In the [metrics explorer](/docs/explore/metrics), search for the prefix `gcp`.
3. Under **Dashboards → Browse templates**, create a dashboard from the **Google Cloud** template.
4. Open **Infrastructure → Google Cloud** for a table of your workloads per service.

## What Maple collects

### Logs

Each entry gets its `service.name` from the resource that wrote it, so workload logs join the traced service of the same name:

| Resource                | `service.name`                                                    |
| ----------------------- | ----------------------------------------------------------------- |
| Cloud Run service, job  | The service or job name.                                          |
| Cloud Function          | The function name.                                                |
| GKE container           | The container name.                                               |
| Compute Engine instance | The instance name, or `gcp/gce_instance` when the entry has none. |
| App Engine              | The module ID.                                                    |
| Everything else         | `gcp/<resource type>`, for example `gcp/cloudsql_database`.       |

Entries carry `cloud.provider`, `cloud.account.id` (the project ID), `cloud.region`, the resource's labels as `gcp.resource.labels.*`, and the Kubernetes, Cloud Run or Compute Engine attributes of their resource. Fields with an OpenTelemetry convention take its name (`httpRequest` becomes `http.request.method`, `url.full`, `http.response.status_code` and so on); the rest keep their Cloud Logging names under `gcp.`. An entry's `trace` and `spanId` become the log's trace and span IDs. Entries in a log named `maple-setup` are setup reports: Maple reads them and doesn't store them, so don't log under that name.

### Metrics

One-minute resolution, named like OpenTelemetry metrics under `gcp.`:

| Service                              | Metrics                                                                                                                                                                                                     |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Cloud Run, Cloud Functions (2nd gen) | `gcp.cloud_run.requests`, `.request.duration`, `.container.instance.count`, `.container.cpu.utilization`, `.container.memory.utilization`, `.container.request.max_concurrency`, `.container.billable_time` |
| Cloud Functions (1st gen)            | `gcp.cloud_functions.executions`, `.execution.duration`, `.instance.count`, `.memory.usage`, `.network.io`                                                                                                  |
| GKE                                  | `gcp.gke.container.cpu.time`, `.cpu.limit.utilization`, `.memory.usage`, `.memory.limit.utilization`, `.restarts`, `gcp.gke.node.cpu.allocatable.utilization`, `.memory.allocatable.utilization`            |
| Compute Engine                       | `gcp.gce.cpu.utilization`, `.memory.usage` (E2 machine types only), `.network.io`, `.disk.io`                                                                                                               |
| Cloud SQL                            | `gcp.cloud_sql.cpu.utilization`, `.memory.utilization`, `.disk.utilization`, `.connection.count`, `.disk.operations`, `.replication.lag`                                                                    |
| Pub/Sub                              | `gcp.pubsub.subscription.undelivered_messages`, `.oldest_unacked_message.age`, `.sent_messages`, `.acked_messages`, `.push_requests`, `.dead_lettered_messages`, `gcp.pubsub.topic.publish_requests`        |
| Load balancers                       | `gcp.load_balancing.requests`, `.request.duration`, `.backend.requests`, `.backend.request.duration`, `.network.io`                                                                                         |

- Counters are delta sums per minute: chart them with `sum`, not `rate`.
- Distributions, such as latency, are gauges with a `quantile` attribute (`0.5`, `0.95`, `0.99`).
- Utilization is a fraction from 0 to 1; durations are in seconds.
- Service names follow the same rule as logs. Every series carries `cloud.account.id` and `cloud.region`; Google aggregates away most other labels before Maple reads them.

### Resources

Every hour Maple lists projects, Cloud Run services and jobs, Cloud Functions, GKE clusters, Compute Engine instances, Cloud SQL instances, Pub/Sub topics and subscriptions, and load balancers from Cloud Asset Inventory, with their location, state and labels.

## Infrastructure → Google Cloud

While a connection has **Metrics and resources** on, **Infrastructure → Google Cloud** shows one tab per service with a row per workload, a band that counts saturated, elevated and erroring workloads, and a **Resources** tab with the inventory. Each workload opens a page with its charts and links to its logs, its traces and the Google Cloud console.

## What Maple can access

The script, the console steps and the generated files grant the same roles. Maple reads as a read-only service account through short-lived tokens; no key is created.

- **Monitoring Viewer** and **Cloud Asset Viewer** on the project, folder or organization: reading metrics and listing resources. Both are read-only; Cloud Asset Viewer can also read resource metadata and IAM policies.
- **Service Usage Consumer** on the host project: calling both APIs through it.
- **Service Account Token Creator** for Maple's service account, on the read-only account only.
- **Pub/Sub Publisher** on Maple's topic for the sink's Google-managed writer identity.

## Google Cloud costs

Google bills Pub/Sub for the log entries that pass through the topic and subscription (a narrower filter lowers it), and Cloud Monitoring reads to the host project: each connection runs 46 `timeSeries.list` queries every 5 minutes, growing with the number of time series. Cloud Asset Inventory searches are free. See [Pub/Sub pricing](https://cloud.google.com/pubsub/pricing) and [Google Cloud Observability pricing](https://cloud.google.com/stackdriver/pricing).

## Limits

- Pub/Sub delivers each log entry at least once; a redelivered entry is stored again (its `insertId` is in `log.record.uid`). Entries Maple could not accept within a day are lost.
- Connecting an organization and a folder or project inside it collects that part twice.
- One read takes up to about 4,000 series per metric. For a large folder or organization, connect its folders or projects separately.
- The first read, and the first after a pause, covers at most the last hour.
- Maple reads up to 10 connections of one Maple organization every 5 minutes; more take turns.
- The resource list holds up to 10,000 resources per connection.
- Load balancer metrics cover global external Application Load Balancers only.

## Troubleshooting

The connection's band names the problem and the fix; when it asks for another setup run, its button **Set up again** opens the dialog at **Method**. For the script, read its last lines in Cloud Shell. A few cases need more:

| You see                                                  | Do                                                                                                                                                      |
| -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Waiting for the first entry** for more than 20 minutes | Run the `gcloud logging write` command the band shows. If the entry doesn't arrive within a minute, set the connection up again.                        |
| A workload's logs appear twice                           | Its application output is forwarded and it also sends logs over OpenTelemetry. Untick its runtime under [Log filter](#log-filter) and apply the change. |
| A workload's stdout and stderr are missing               | Application output isn't forwarded by default. Send it over OpenTelemetry, or tick its runtime under **Log filter**.                                    |
| A wrong project, folder or organization ID               | A connection's ID can't be changed: disconnect it and connect the right one.                                                                            |

On a self-hosted Maple, **Metrics and resources** reads **Not available on this Maple deployment** until `MAPLE_GCP_SERVICE_ACCOUNT_EMAIL` names a Google service account the deployment owns. Reading metrics also needs `MAPLE_GCP_SERVICE_ACCOUNT_KEY`, that account's key file, base64-encoded.

### Domain restricted sharing

An organization policy can restrict which identities may hold IAM roles in your organization (`constraints/iam.allowedPolicyMemberDomains`, or a custom constraint on member domains). Maple's service account lives outside your organization, so such a policy rejects the grant that lets Maple read as your read-only service account. The script stops with "An organization policy (domain restricted sharing) blocks granting Maple's service account"; the console refuses the grant.

Don't turn the policy off. An Organization Policy Administrator can allow Maple's one service account on the host project, and nothing else:

1. Use the managed constraint **Restrict allowed policy members in IAM allow policies** (`iam.managed.allowedPolicyMembers`). The older `iam.allowedPolicyMemberDomains` can't make an exception for a single principal.
2. On the host project, set a policy whose allowed principals are your organization's principal set plus Maple's service account. Google's steps: [Restrict identities with domain-restricted sharing](https://cloud.google.com/resource-manager/docs/organization-policy/restricting-domains).
3. Make the grant again. Leave the exception in place: Google checks it at every grant.

## Disconnect

Open the **⋮** menu on the connection's row and choose **Disconnect**. Remove Maple's resources from Google Cloud first:

- **Set up by script**: run the cleanup script the dialog shows. Maple sees it finish.
- **Set up in the console**: delete them in the order the dialog lists. See [Removing](/docs/integrations/gcp-manual-setup#removing).
- **Set up with Terraform or Alchemy**: destroy it there. See [Change or remove](/docs/integrations/gcp-infrastructure-as-code#change-or-remove).

Then click **Disconnect**. Maple stops reading the connection's metrics and accepting its logs; data already in Maple is kept. Disconnecting first leaves Google Cloud publishing logs to Pub/Sub, billed by Google, and Maple keeps the cleanup script on the page until you dismiss it.

## Next steps

- [Google Cloud with OpenTelemetry](/docs/integrations/gcp-opentelemetry): what to instrument on GKE, Cloud Run and Compute Engine.
- [Dashboards](/docs/dashboards/build-dashboards#templates): start from the **Google Cloud** template.
- [Alert rules](/docs/alerting/alert-rules): alert on any `gcp.*` metric.
- [API reference](/docs/reference/api): manage connections with the `/v2/integrations/gcp` endpoints.

---

# Google Cloud with OpenTelemetry

> Integrations · https://maple.dev/docs/integrations/gcp-opentelemetry

Which telemetry to send from your workloads over OpenTelemetry and which to collect through the Google Cloud connection, for GKE, Cloud Run and Compute Engine.

Your workloads send traces, logs and metrics straight to Maple over OpenTelemetry. The [Google Cloud connection](/docs/integrations/gcp) forwards Cloud Logging entries and reads Cloud Monitoring metrics for everything your code never sees. Set up both, and keep container logs on the OpenTelemetry side.

| Signal    | Over OpenTelemetry                       | Google Cloud connection                                                                                                             |
| --------- | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| Traces    | Every span your services create          | None                                                                                                                                |
| Logs      | Application logs, linked to their traces | Cloud Run and load balancer request logs, audit logs, managed-service logs such as Cloud SQL and Pub/Sub, GKE node and cluster logs |
| Metrics   | Application and runtime metrics          | Platform metrics from Cloud Monitoring, named `gcp.*`                                                                               |
| Resources | None                                     | The resource list: projects, services, clusters, instances and databases                                                            |

## Recommended setup

### GKE

1. Instrument each service with an OpenTelemetry SDK. Export to Maple, or to the [Maple Kubernetes collector](/docs/infrastructure/kubernetes) in the cluster.
2. [Connect Google Cloud](/docs/integrations/gcp#connect) and leave **Log filter** as it is: no application output is forwarded.

The connection then adds node and cluster logs, audit logs and the `gcp.gke.*` metrics.

### Cloud Run

1. Instrument the service with an OpenTelemetry SDK that exports to Maple, with the settings under [Instrument a Cloud Run service](#instrument-a-cloud-run-service).
2. [Connect Google Cloud](/docs/integrations/gcp#connect).
3. If the service exports logs over OpenTelemetry, untick **Cloud Run** under [**Log filter**](#change-the-filter): by default the connection forwards what a Cloud Run container writes to stdout and stderr.

The connection adds Cloud Run's request logs and the `gcp.cloud_run.*` metrics. Cloud Run sends a `traceparent` header with every request. An SDK that continues it gives the server span the trace ID of Google's request log for that request, so the request log, your spans and your log lines join on one trace.

### Compute Engine

1. Instrument the application with an OpenTelemetry SDK that exports to Maple.
2. Run the OpenTelemetry Collector on the VM for [host metrics](/docs/infrastructure/hosts).
3. [Connect Google Cloud](/docs/integrations/gcp#connect).

The connection adds audit logs and the `gcp.gce.*` metrics.

## Application output

Application output is what the code of a runtime writes to stdout and stderr. The connection forwards none by default: send it over OpenTelemetry. Platform, request and audit logs are forwarded either way. For workloads that can't, click **Can't forward via OpenTelemetry** under **Log filter** and tick their runtimes.

| Runtime             | Left out when unticked                                                                                | Still forwarded                          |
| ------------------- | ----------------------------------------------------------------------------------------------------- | ---------------------------------------- |
| **Cloud Run**       | `run.googleapis.com/stdout` and `stderr` of services, jobs, worker pools and 2nd gen functions        | Request logs, system logs, audit logs    |
| **Cloud Functions** | `cloudfunctions.googleapis.com/cloud-functions` of 1st gen functions, with its start and finish lines | Audit logs                               |
| **App Engine**      | `stdout` and `stderr` of `gae_app`                                                                    | Request logs, audit logs                 |
| **GKE containers**  | Everything of `resource.type="k8s_container"`                                                         | Node, pod and cluster events, audit logs |

Don't tick a runtime whose workloads already send their logs to Maple:

| Reason        | Detail                                                                                                                                                                                 |
| ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Duplicates    | A workload that exports logs over OpenTelemetry and also prints them, or a collector that reads pod logs, already delivers each line. Forwarding it from Google Cloud stores it twice. |
| No trace link | The OpenTelemetry copy carries the trace and span ID of the request that wrote it. The copy from Google Cloud has neither.                                                             |
| Volume        | Application output grows with traffic. Every forwarded line counts toward your Maple plan and toward the Pub/Sub usage Google bills to your account.                                   |

Maple does not check which workloads send their logs another way. A runtime with both kinds of workloads is one tick: tick it when the uninstrumented ones matter more, since Cloud Logging is the only place their lines exist.

Other places your code runs have no tick and are always forwarded: Compute Engine logs collected by the Ops Agent, and the job output of Batch, Dataflow, Dataproc, Cloud Composer and Vertex AI. Leave them out with a filter of your own.

## Change the filter

On the connection's row, click **Configure**. Under **Log filter**, choose **Replace filter**. To forward application output, click **Can't forward via OpenTelemetry** and tick its runtimes. Click **Continue to the script**, then copy the script and run it in Cloud Shell again.

| Log filter              | The sink forwards                                                                     |
| ----------------------- | ------------------------------------------------------------------------------------- |
| **Keep current filter** | What it forwards now. Preselected once a run has set log forwarding up.               |
| **Replace filter**      | Platform, request and audit logs, plus the application output of any ticked runtimes. |

For any other filter, paste the script into an editor, edit `LOG_FILTER` near its top, set `LOG_FILTER_MODE` to `set` and run it. The filter uses the [Logging query language](https://cloud.google.com/logging/docs/view/logging-query-language).

## Instrument a Cloud Run service

These settings decide whether every trace arrives and joins Google's request log. They apply in any language. The example is Node.js, and the [Node.js guide](/docs/guides/instrumentation-nodejs) covers the SDK basics.

| Setting      | On Cloud Run                                                                                                                                                                                                                                                                                                                               |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Service name | Set `service.name` to the Cloud Run service name, `K_SERVICE`. Google's request logs arrive in Maple under that name. With another one, your spans and logs land in a different service than the request logs.                                                                                                                             |
| Sampling     | Sample every trace: `AlwaysOnSampler`, or `OTEL_TRACES_SAMPLER=always_on`. Cloud Run sets the sampled flag of `traceparent` on about one request per ten seconds per instance. The SDK's default sampler follows that flag and drops the rest, while the log records of those requests still arrive.                                       |
| Flushing     | Flush the span and log processors when the server span ends. Call `sdk.shutdown()` on `SIGTERM` and keep `node` as PID 1. Cloud Run throttles the container's CPU once the response is sent: with the default batch timers, telemetry arrives 5 to 6 seconds after the request on a warm instance and about 30 seconds after a cold start. |
| Logs         | Emit application logs over OpenTelemetry, not stdout. Each record carries the trace and span ID of its request, and nothing is stored twice.                                                                                                                                                                                               |

For Node.js 22 with `@opentelemetry/sdk-node`:

```js
// tracing.mjs
import { register } from "node:module"
import { SpanKind } from "@opentelemetry/api"
import { NodeSDK } from "@opentelemetry/sdk-node"
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node"
import { envDetector, processDetector, resourceFromAttributes } from "@opentelemetry/resources"
import { gcpDetector } from "@opentelemetry/resource-detector-gcp"
import { AlwaysOnSampler, BatchSpanProcessor } from "@opentelemetry/sdk-trace-base"
import { BatchLogRecordProcessor } from "@opentelemetry/sdk-logs"
import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics"
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto"
import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-proto"
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-proto"

// Lets the instrumentations patch ES modules.
register("@opentelemetry/instrumentation/hook.mjs", import.meta.url)

// The exporters read OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS.
const spans = new BatchSpanProcessor(new OTLPTraceExporter())
const logRecords = new BatchLogRecordProcessor({ exporter: new OTLPLogExporter() })
const metricExporter = new OTLPMetricExporter()

// Export as each request finishes, before Cloud Run throttles the CPU.
const flushOnRequestEnd = {
	onStart() {},
	onEnd(span) {
		if (span.kind !== SpanKind.SERVER) return
		void spans.forceFlush()
		void logRecords.forceFlush()
	},
	forceFlush: () => Promise.resolve(),
	shutdown: () => Promise.resolve(),
}

const sdk = new NodeSDK({
	// The name Google's request logs carry.
	resource: resourceFromAttributes({ "service.name": process.env.K_SERVICE }),
	// gcpDetector adds cloud.platform, cloud.region and the faas.* attributes.
	resourceDetectors: [envDetector, processDetector, gcpDetector],
	sampler: new AlwaysOnSampler(),
	spanProcessors: [spans, flushOnRequestEnd],
	logRecordProcessors: [logRecords],
	metricReaders: [new PeriodicExportingMetricReader({ exporter: metricExporter })],
	instrumentations: [getNodeAutoInstrumentations()],
})
sdk.start()

// Cloud Run sends SIGTERM before it stops an instance.
process.on("SIGTERM", () => {
	sdk.shutdown().finally(() => process.exit(0))
})
```

Start the container with `node` as PID 1, so that it receives `SIGTERM`:

```dockerfile
CMD ["node", "--import", "./tracing.mjs", "server.mjs"]
```

Set these on the service:

- `OTEL_EXPORTER_OTLP_ENDPOINT`: `https://ingest.maple.dev`, or `https://ingest.eu.maple.dev` for an EU organization.
- `OTEL_EXPORTER_OTLP_HEADERS`: `Authorization=Bearer YOUR_INGEST_KEY`.

Emit a log record inside a request and it carries the active trace and span ID:

```js
import { logs, SeverityNumber } from "@opentelemetry/api-logs"

const logger = logs.getLogger("checkout")

logger.emit({
	severityText: "INFO",
	severityNumber: SeverityNumber.INFO,
	body: "checkout completed",
	attributes: { "order.id": orderId },
})
```

A service that has to log to stdout keeps the link by writing JSON lines with `logging.googleapis.com/trace` (`projects/PROJECT_ID/traces/TRACE_ID`) and `logging.googleapis.com/spanId`. Send each line one way only: over OpenTelemetry or through stdout.

Limits of this setup:

- Your service's top span is the child of a span inside Google's infrastructure that Maple does not receive, so the trace has no root span.
- Flushing per request makes one export per request. Check the overhead before you use it on a service with a high request rate.

## Instrumentation guides

- Languages: [Node.js](/docs/guides/instrumentation-nodejs), [Next.js](/docs/guides/instrumentation-nextjs), [Python](/docs/guides/instrumentation-python), [Go](/docs/guides/instrumentation-go), [Rust](/docs/guides/instrumentation-rust), [Java](/docs/guides/instrumentation-java), [Kotlin](/docs/guides/instrumentation-kotlin), [C# / .NET](/docs/guides/instrumentation-csharp) and [Laravel](/docs/guides/instrumentation-laravel).
- [Kubernetes infrastructure](/docs/infrastructure/kubernetes): the collector that receives OTLP from in-cluster services.
- [Hosts](/docs/infrastructure/hosts): host metrics from a VM.
- [Google Cloud](/docs/integrations/gcp): connect, verify and troubleshoot the connection.

---

# Google Cloud manual setup

> Integrations · https://maple.dev/docs/integrations/gcp-manual-setup

Connect Google Cloud to Maple by creating each resource yourself in the Google Cloud console, without running a script.

Manual setup creates the same resources as the [setup script](/docs/integrations/gcp), by hand in the Google Cloud console. It takes about 20 minutes.

In Maple, open **Integrations → Google Cloud**, click **Connect Google Cloud**, choose the scope and what to collect, then choose **Manual setup** in **Method** and click **Continue**. The dialog lists the steps below with this connection's names and values: copy them from there. Each step has an **Open in console** link to the right page.

## Roles you need

| You connect     | Roles you need                                                                                                                                      |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| A project       | Owner on the project.                                                                                                                               |
| A folder        | Owner on the host project. On the folder: Logs Configuration Writer for the log sink, and Folder IAM Admin for the read-only roles.                 |
| An organization | Owner on the host project. On the organization: Logs Configuration Writer for the log sink, and Organization Administrator for the read-only roles. |

## Step by step

The screenshots show a project connection. Steps 2 to 7 are for **Log forwarding**, the rest for **Metrics and resources**: the dialog leaves out what you didn't choose. A folder or organization has one more step, Service Usage Consumer on the host project.

### 1. Enable the APIs

In **APIs & Services → Library**, search for each API the dialog lists, open it and click **Enable**. An API that shows **Manage** and **API Enabled** is on already. `pubsub.googleapis.com` is the **Cloud Pub/Sub API**, not Pub/Sub Lite.

<figure class="shot">
  <img src="/screenshots/docs/gcp-manual-01-api-enabled.webp" alt="The Cloud Asset API page in the API Library, with a Manage button and a green API Enabled label." loading="lazy" />
  <figcaption>An API that is already on: <strong>Manage</strong> and <strong>API Enabled</strong>.</figcaption>
</figure>

### 2. Create the Pub/Sub topic

In **Pub/Sub → Topics**, click **Create topic**. Paste the **Topic ID**, untick **Add a default subscription**, and click **Create**. The new topic warns that messages will be lost: the subscription in step 4 takes care of that.

<figure class="shot">
  <img src="/screenshots/docs/gcp-manual-02-topic.webp" alt="The Create topic form with the Topic ID filled in and Add a default subscription unticked." loading="lazy" />
  <figcaption>Untick <strong>Add a default subscription</strong>: it would keep a copy of every log that nobody reads.</figcaption>
</figure>

### 3. Create the push service account

Pub/Sub signs each push to Maple as this account.

1. In **IAM & Admin → Service accounts**, click **Create service account**.
2. Paste the **Service account name**, then the **Service account ID**: the console fills the ID from the name, so replace it. Paste the description and click **Create and close**, without roles.
3. Open the new account, go to its **Principals with access** tab, not **Permissions**, and click **Grant access**.
4. In **New principals**, type `gcp-sa-pubsub` and pick **Cloud Pub/Sub Service Account for Project …**. The role picker starts on **Service Account Admin**: replace it with **Service Account Token Creator**, and click **Save**.

Step 4 is only needed where the Pub/Sub service agent was created before April 8, 2021, and harmless elsewhere. The grant is listed only with **Include Google-provided role grants** ticked.

<figure class="shot">
  <img src="/screenshots/docs/gcp-manual-03-push-account.webp" alt="The Create service account form with the name Maple log push identity, the ID maple-push-… and the description filled in, and the Create and close button." loading="lazy" />
  <figcaption>Replace the ID the console suggests with the one from the dialog, then <strong>Create and close</strong>.</figcaption>
</figure>

<figure class="shot">
  <img src="/screenshots/docs/gcp-manual-03-principals-tab.webp" alt="The service account's tabs Details, Permissions, Keys, Metrics, Logs and Principals with access, with a note pointing to Principals with access." loading="lazy" />
  <figcaption>Grants on the account itself live under <strong>Principals with access</strong>.</figcaption>
</figure>

<figure class="shot">
  <img src="/screenshots/docs/gcp-manual-03-token-creator.webp" alt="The Grant access panel with the Pub/Sub service agent as new principal and the role Service Account Token Creator." loading="lazy" />
  <figcaption>The Pub/Sub service agent gets <strong>Service Account Token Creator</strong>.</figcaption>
</figure>

### 4. Create the push subscription

In **Pub/Sub → Subscriptions**, click **Create subscription** and set:

- **Subscription ID**, and the topic from step 2 under **Select a Cloud Pub/Sub topic**.
- **Delivery type: Push**, with the **Endpoint URL** from the dialog.
- Tick **Enable authentication**, pick the push service account from step 3, and paste the endpoint again as **Audience**. The console marks Audience optional; Maple needs it.
- Tick **Enable payload unwrapping** and leave **Write metadata** unticked. Without unwrapping Maple can't read what Pub/Sub sends and drops it.
- Under **Failure handling → Retry policy**: **Retry after exponential backoff delay**, 10 to 600 seconds (the defaults).
- **Acknowledgement deadline**: 30 seconds.
- **Message retention duration**: 1 day, 0 hours, 0 minutes.
- **Expiry period**: **Never expire**. The default, 31 days, deletes an idle subscription.

Click **Create**.

<figure class="shot">
  <img src="/screenshots/docs/gcp-manual-04-subscription-push.webp" alt="The Create subscription form: subscription ID, topic, Delivery type Push, Endpoint URL, Enable authentication ticked with the push service account, Audience, and Enable payload unwrapping ticked with Write metadata unticked." loading="lazy" />
  <figcaption>Push delivery, signed by the push account, with payload unwrapping on.</figcaption>
</figure>

<figure class="shot">
  <img src="/screenshots/docs/gcp-manual-04-subscription-retry.webp" alt="Failure handling with Retry after exponential backoff delay selected, 10 and 600 seconds, and an acknowledgement deadline of 30 seconds." loading="lazy" />
  <figcaption>Retry policy and acknowledgement deadline.</figcaption>
</figure>

<figure class="shot">
  <img src="/screenshots/docs/gcp-manual-04-subscription-lifetime.webp" alt="Message retention duration 1 day, 0 hours, 0 minutes, and Expiry period set to Never expire, above the Create button." loading="lazy" />
  <figcaption>One day of retention and <strong>Never expire</strong>.</figcaption>
</figure>

### 5. Create the Log Router sink

In **Logging → Log router**, click **Create sink**:

1. **Sink details**: paste the **Sink name**, click **Next**.
2. **Sink destination**: **Select sink service** → **Cloud Pub/Sub topic**, pick the topic from step 2, click **Next**.
3. **Choose logs to include in sink**: paste the filter from the dialog into **Build inclusion filter**, unchanged, and click **Next**.
4. Skip **Choose logs to filter out of sink** and click **Create sink**.

A new sink takes a few minutes to start forwarding. Entries written in that time are dropped, not delayed.

<figure class="shot">
  <img src="/screenshots/docs/gcp-manual-05-sink-filter.webp" alt="The Create logs routing sink page with sink details and destination done, and the filter pasted into Build inclusion filter." loading="lazy" />
  <figcaption>Paste the filter as the dialog shows it.</figcaption>
</figure>

<figure class="shot">
  <img src="/screenshots/docs/gcp-manual-05-sink-review.webp" alt="The sink's four sections filled in: destination Cloud Pub/Sub topic, the inclusion filter, no exclusions, and the Create sink button." loading="lazy" />
  <figcaption>Ready to create: Cloud Pub/Sub topic, the filter, no exclusions.</figcaption>
</figure>

### 6. Let the sink publish to the topic

1. In **Log router**, open the sink's **⋮** menu → **View sink details** and copy its **Writer identity**.
2. In **Pub/Sub → Topics**, tick the topic. In the info panel on the right (**Show info panel** if it is hidden), open **Permissions** → **Add principal**.
3. Paste the writer identity **without its leading `serviceAccount:`**: with it, the console refuses the principal. Pick the role **Pub/Sub Publisher** and click **Save**.

Grant it on the topic, not on the IAM page, where it would cover every topic in the project.

<figure class="shot">
  <img src="/screenshots/docs/gcp-manual-06-writer-identity.webp" alt="The Sink details modal showing the destination and the writer identity serviceAccount:service-…@gcp-sa-logging.iam.gserviceaccount.com with a copy button." loading="lazy" />
  <figcaption>The writer identity, in <strong>View sink details</strong>.</figcaption>
</figure>

<figure class="shot">
  <img src="/screenshots/docs/gcp-manual-06-topic-permissions.webp" alt="The Topics list with the Maple topic ticked and the info panel open on Permissions, with an Add principal button." loading="lazy" />
  <figcaption>The topic's own <strong>Permissions</strong>, in the info panel.</figcaption>
</figure>

<figure class="shot">
  <img src="/screenshots/docs/gcp-manual-06-publisher-grant.webp" alt="The Grant access panel for the topic with the logging service agent as new principal and the role Pub/Sub Publisher." loading="lazy" />
  <figcaption>The identity without <code>serviceAccount:</code>, with <strong>Pub/Sub Publisher</strong>.</figcaption>
</figure>

### 7. Send a test message (optional)

On the topic's **Messages** tab, click **Publish message**, paste the dialog's message body as **Message**, and click **Publish**. Maple confirms log forwarding within seconds. The message proves the topic, subscription and push; the sink is proven by the first real log.

<figure class="shot">
  <img src="/screenshots/docs/gcp-manual-07-test-message.webp" alt="The Publish message panel with the test message body pasted into Message and the Publish button." loading="lazy" />
  <figcaption>The test message from the dialog.</figcaption>
</figure>

### 8. Create the read-only service account

Maple reads metrics and resources as this account, with short-lived tokens and no key. In **IAM & Admin → Service accounts**, click **Create service account**, paste the name, replace the ID the console fills in, and click **Create and close**.

<figure class="shot">
  <img src="/screenshots/docs/gcp-manual-08-reader-account.webp" alt="The Create service account form with the name Maple metrics and resource reader and the ID maple-… filled in." loading="lazy" />
  <figcaption>The read-only account, created without roles.</figcaption>
</figure>

### 9. Let Maple use the read-only account

Open the read-only account → **Principals with access** → **Grant access**. Paste Maple's service account from the dialog as **New principals**, replace the preselected **Service Account Admin** with **Service Account Token Creator**, and click **Save**.

If an organization policy restricts members to your domain, Google refuses this grant and nothing is left behind: see [Domain restricted sharing](/docs/integrations/gcp#domain-restricted-sharing).

<figure class="shot">
  <img src="/screenshots/docs/gcp-manual-09-maple-token-creator.webp" alt="The Grant access panel on the read-only account with Maple's service account as new principal and the role Service Account Token Creator." loading="lazy" />
  <figcaption>Maple's account gets <strong>Service Account Token Creator</strong> on the read-only account.</figcaption>
</figure>

### 10. Grant the read-only roles

In **IAM & Admin → IAM**, click **Grant access**, paste the read-only account as **New principals**, and add **Monitoring Viewer**, **Cloud Asset Viewer** and **Service Usage Consumer**, each after the first with **Add another role**. Click **Save**. Cloud Monitoring only answers for a host project with billing linked.

<figure class="shot">
  <img src="/screenshots/docs/gcp-manual-10-reader-roles.webp" alt="The Grant access panel on the project with the read-only account and three roles: Monitoring Viewer, Cloud Asset Viewer and Service Usage Consumer." loading="lazy" />
  <figcaption>The three read-only roles.</figcaption>
</figure>

## Folders and organizations

Everything goes in the host project except these, which you do on the folder or organization (the dialog's links open them there):

- **The Log Router sink.** Choose **Other resource** as the sink service and paste the full destination from the dialog, `pubsub.googleapis.com/projects/HOST_PROJECT/topics/…`: the picker lists only the current project's topics. Choose **Include logs ingested by this folder (organization) and all child resources**, not **Intercept logs**, which keeps those logs from the projects' own sinks and log buckets. The writer identity in step 6 is the folder's or organization's own.
- **Monitoring Viewer and Cloud Asset Viewer** on the folder or organization. **Service Usage Consumer** goes on the host project, as its own step in the dialog.

## Confirm the connection

A manual setup sends Maple no report. Maple confirms each part by its first data:

- **Log forwarding** by the first push Maple receives: within seconds with the test message, or within a few minutes of the sink.
- **Metrics and resources** by the first metrics read, a few minutes after step 10. Until then the connection may say Maple can't sign in as the read-only account yet.

## Removing

When you switch **Log forwarding** off, the dialog lists what to delete, in this order, each with an **Open in console** link:

1. The log sink: **⋮ → Delete sink**. Then wait two minutes: Google keeps routing to the topic for over a minute, and a missing topic gets your project a sink error.
2. The subscription.
3. The topic. The console asks you to type `delete`.
4. The push service account.

Then click **I've removed them** in the dialog: Maple can't see a removal and takes your word for it.

<figure class="shot">
  <img src="/screenshots/docs/gcp-manual-11-delete-sink.webp" alt="The Confirm sink deletion modal naming the Maple sink." loading="lazy" />
  <figcaption>The sink goes first.</figcaption>
</figure>

<figure class="shot">
  <img src="/screenshots/docs/gcp-manual-12-delete-topic.webp" alt="The Delete topic modal, with delete typed into the confirmation box." loading="lazy" />
  <figcaption>Deleting the topic asks you to type <code>delete</code>.</figcaption>
</figure>

A connection collects at least one of the two. To stop collecting altogether, open the connection's menu → **Disconnect**: under **Delete them in the console instead** it lists everything still set up. For the read-only account, remove its roles in **IAM & Admin → IAM** (tick it → **Remove access**), then delete the account.

<figure class="shot">
  <img src="/screenshots/docs/gcp-manual-13-remove-access.webp" alt="The confirmation to remove the read-only service account's access, listing that it loses all non-inherited roles on the resource." loading="lazy" />
  <figcaption>Removing the read-only account's roles before deleting it.</figcaption>
</figure>

Leave **Logs Writer** for a sink's writer identity on the host project: every sink there writes as that identity.

---

# Google Cloud with Terraform or Alchemy

> Integrations · https://maple.dev/docs/integrations/gcp-infrastructure-as-code

Connect Google Cloud to Maple from your own infrastructure code: a generated Terraform or Alchemy file with every resource inline.

Maple generates the Google Cloud connection as a file for your own repository: `maple.tf` for Terraform's `google` provider, or `maple.ts` for [Alchemy](https://alchemy.run)'s `alchemy/GCP` resources. It creates the same resources as the [setup script](/docs/integrations/gcp) under the same names, so you can review every one before it exists.

The file carries no secret, so you can commit it: Pub/Sub signs each push as a push service account that holds no role, and Maple checks the signature. IAM grants are additive (`*_iam_member`): your other grants are left alone.

## Get the file

1. In Maple, open **Integrations → Google Cloud**, click **Connect Google Cloud**, choose the scope and what to collect, and click **Continue**.
2. In **Method**, choose **Infrastructure as code**, then **Terraform** or **Alchemy**, and click **Continue**.
3. Click **Copy**, or **Copy prompt for coding agent** (see [below](#with-a-coding-agent)).

Without a script there is no sink filter to keep: the file sets the recommended filter, or the one chosen under **Log filter**.

## Terraform

Put `maple.tf` in a root module of its own: its resource names are fixed per connection, and `import` blocks only work in a root module. If your configuration already declares the `google` and `time` providers, delete the file's `terraform` block. It needs Terraform 1.5 or later.

Run `terraform plan`, review it, then `terraform apply` with an account that has Owner on the host project. For a folder or organization it also needs Logs Configuration Writer on the scope, and Folder IAM Admin or Organization Administrator for the read-only roles.

If an earlier setup created the connection, the file has `import` blocks: Terraform takes over the topic, subscription, sink and read-only account instead of failing on duplicates. A commented `import` for the push service account (a setup in the console has one) and for a read-only account an earlier setup left in place are there to uncomment.

## Alchemy

`maple.ts` is a self-contained stack, named after the connection. Its header has the commands:

1. Install what its `bun add` line names: `alchemy` at the version the file was written for, `effect` 4 and `@effect/platform-bun` 4.
2. With log forwarding, fill in `PROJECT_NUMBER` at the top: `gcloud projects describe HOST_PROJECT --format='value(projectNumber)'`.
3. Switch the APIs on first with the `gcloud services enable` command. Alchemy doesn't wait for an API to be switched on.
4. Deploy with the header's command. Alchemy creates the topic, the subscription and a project's sink in the project it signs in to, which must be the host project. After a setup by script the command has `--adopt`, which takes over what the script created.

Alchemy's IAM resource grants roles on projects only. For a folder or organization with **Metrics and resources**, `maple.ts` defines its own small resource, `ScopeIamMember`, that grants the two read roles on the folder's or organization's IAM policy and removes them again on destroy. The account that deploys needs to be allowed to set that policy, as Folder IAM Admin or Organization Administrator, like Terraform.

`bunx alchemy destroy maple.ts --stage maple` removes it again, with the state Alchemy keeps in `.alchemy/`. It deletes the topic right after the sink; to avoid a sink error, delete the sink in the console two minutes earlier.

## With a coding agent

**Copy prompt for coding agent** copies the file with instructions for an agent working in your repository: where to put the file, to show you the plan and apply only after you approve it, the `gcloud` commands that check the result, and how Maple confirms it. Paste it into the agent from the repository that holds your infrastructure code.

## Confirm the connection

Nothing in Terraform or Alchemy reports to Maple. Maple confirms each part by its first data:

- **Log forwarding** by the first push Maple receives. A new sink takes a few minutes to start forwarding, and entries written in that time are dropped, not delayed.
- **Read-only access** by the first metrics read, within 5 minutes.

## Change or remove

To change what a connection collects, change it in Maple, copy the regenerated file over the old one and apply it. A capability you switched off is no longer in the file, so the apply removes what the file created. What the file didn't create (a setup by script or in the console) stays: delete it in the console, as the dialog lists. Then click **I've removed them** in the dialog: Maple can't see a removal and takes your word for it.

Without the script there is no sink filter to keep: if you customised `log_filter`, keep yours when you copy a regenerated file.

The file doesn't manage the **Logs Writer** role of the sink's writer identity on the host project: every sink there writes as that identity, so a destroy must not revoke it.

To disconnect, run `terraform destroy` (or the Alchemy destroy above), then disconnect in Maple.

---

# Instrument your application

> Instrumentation · https://maple.dev/docs/instrumentation

Setup guides for every language, framework and runtime Maple supports.

import GuideFilter from "../../components/docs/GuideFilter.astro"
import GuideGrid from "../../components/docs/GuideGrid.astro"

Maple ingests OpenTelemetry over OTLP/HTTP, so any OTLP exporter works without a Maple-specific plugin. Each guide below installs the SDK, points it at the ingest endpoint with an ingest key from **Settings → Ingestion**, and checks that the first trace arrives. New to Maple? The [Quickstart](/docs/getting-started/quickstart) gets a first trace in about five minutes. The [maple-onboard](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-onboard) skill can do the same setup from a coding agent.

<GuideFilter />

<section data-guide-section="javascript">

## JavaScript and TypeScript

<GuideGrid section="javascript" />

</section>

<section data-guide-section="python">

## Python

<GuideGrid section="python" />

</section>

<section data-guide-section="jvm">

## Java and Kotlin

<GuideGrid section="jvm" />

</section>

<section data-guide-section="other">

## Go, Rust, .NET and PHP

<GuideGrid section="other" />

</section>

<section data-guide-section="frontend">

## Frontend and product analytics

<GuideGrid section="frontend" />

</section>

## AI agents and LLM calls

Agent frameworks and LLM gateways have their own guides, so each conversation shows up as one [Agent Session](/docs/agent-sessions/overview) with its transcript, tool calls, tokens and cost: Vercel AI SDK, OpenAI Agents SDK, LangChain and LangGraph, Mastra, Pydantic AI, CrewAI, Google ADK, OpenRouter, LiteLLM and more. See [Trace your AI agent](/docs/agent-tracing).

<section data-guide-section="infrastructure">

## Infrastructure and data sources

<GuideGrid section="infrastructure" />

</section>

## Anything that speaks OTLP

If your language or framework is not listed, point its OpenTelemetry SDK (or an OpenTelemetry Collector) at the ingest endpoint with the standard environment variables. Ingest keys are scoped to one organization and can only write telemetry. Only the public key (`maple_pk_…`) is safe to ship in client code; keep the private key (`maple_sk_…`) in an environment variable or a secret store. See [Authentication](/docs/reference/authentication).

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
export OTEL_SERVICE_NAME="my-service"
```

Maple reads the standard semantic conventions; the attributes it relies on for the service map, error grouping and sampling-aware throughput are listed in [OpenTelemetry conventions](/docs/concepts/otel-conventions). To develop against a local sink instead of the cloud, run [Maple Local](/docs/local-mode) and point the endpoint at `http://localhost:4318`.

## Data regions

Maple Cloud runs in two regions. An organization's region is chosen during onboarding and cannot be changed afterwards. Send telemetry to the ingest endpoint of your organization's region, with an ingest key created in that region's dashboard. A key from one region is rejected by the other.

| Region                     | Dashboard                  | Ingest endpoint               |
| -------------------------- | -------------------------- | ----------------------------- |
| United States              | `https://app.maple.dev`    | `https://ingest.maple.dev`    |
| European Union (Frankfurt) | `https://app.eu.maple.dev` | `https://ingest.eu.maple.dev` |

Every guide on this site uses the US endpoint. For an EU organization, replace `ingest.maple.dev` with `ingest.eu.maple.dev` and nothing else changes. See [Regions](/docs/reference/regions) for details and [EU hosting](/eu) for what the EU region covers.

Missing a guide for your stack? [Open an issue](https://github.com/MapleTechLabs/maple/issues) and tell us what you run.

---

# Effect SDK

> Instrumentation · https://maple.dev/docs/sdks/effect

OpenTelemetry traces, logs, and metrics for Effect applications across Node.js, Bun, Deno, browsers, and Cloudflare Workers.

`@maple-dev/effect-sdk` is Maple's SDK for Effect applications. It provides an Effect Layer that sets up OpenTelemetry traces, logs and metrics, fills in resource attributes from the runtime, and sends everything to Maple's ingest for your region. In most setups an ingest key from **Settings → Ingestion** is the only configuration it needs.

<div class="flex flex-wrap gap-2 mb-8 not-prose">
    <span class="text-[10px] uppercase tracking-wider px-2 py-1 border border-border text-fg-muted">Node.js</span>
    <span class="text-[10px] uppercase tracking-wider px-2 py-1 border border-border text-fg-muted">Bun</span>
    <span class="text-[10px] uppercase tracking-wider px-2 py-1 border border-border text-fg-muted">Deno</span>
    <span class="text-[10px] uppercase tracking-wider px-2 py-1 border border-border text-fg-muted">Browsers</span>
    <span class="text-[10px] uppercase tracking-wider px-2 py-1 border border-border text-fg-muted">Cloudflare Workers</span>
</div>

## Install

**Effect 4** (`effect` 4.0.0 or later)

```bash
npm install @maple-dev/effect-sdk effect
```

**Effect 3**

```bash
npm install @maple-dev/effect-sdk@effect-v3 effect @effect/platform @effect/opentelemetry
```

> The Effect 3 build is published under the `effect-v3` npm tag and is older than the Effect 4 release. The options on these pages describe the Effect 4 release.

## Pick your platform

The SDK ships three entry points, each with its own page:

- [**Server**](/docs/sdks/effect-server): Node.js, Bun, Deno. Background export fiber, configuration from environment variables, graceful shutdown.
- [**Browser**](/docs/sdks/effect-client): single-page apps. Configuration passed in code, browser metadata added to resource attributes, session replay.
- [**Cloudflare Workers**](/docs/sdks/effect-cloudflare): short-lived isolates. In-isolate buffering, `flush()` in `ctx.waitUntil`, configuration read from the Worker `env` on first flush.

## Custom spans

Use `Effect.withSpan` to trace operations. Add attributes with `Effect.annotateCurrentSpan`:

```typescript
import { Effect } from "effect"

const processOrder = (orderId: string) =>
	Effect.gen(function* () {
		yield* Effect.annotateCurrentSpan("order.id", orderId)
		yield* Effect.annotateCurrentSpan("payment.method", "card")
		const result = yield* chargePayment(orderId)
		return result
	}).pipe(Effect.withSpan("process-order"))
```

Service map edges come from instrumented client spans that propagate `traceparent` to an instrumented callee, not from attributes such as `peer.service`. See [Service map](/docs/explore/service-map).

## Log correlation

`Effect.log` includes the trace context when called inside a span, with no extra setup:

```typescript
const program = Effect.gen(function* () {
	yield* Effect.log("Processing started")
	yield* doWork()
	yield* Effect.log("Processing complete")
}).pipe(Effect.withSpan("process"))
```

Logs emitted inside spans are correlated with the active trace in the Maple dashboard.

## Configuration reference

Options for `Maple.layer()` on the server and browser entry points, and for `make()` on the Cloudflare entry point. The span options apply to every entry point. The Cloudflare-only options are on the [Cloudflare page](/docs/sdks/effect-cloudflare#cloudflare-specific-config).

| Option                        | Type                              | Entry points       | Description                                                                                                                                                                                                       |
| ----------------------------- | --------------------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `serviceName`                 | `string`                          | all                | Service name on traces, logs and metrics. Required in the browser. On the server and Cloudflare it falls back to `OTEL_SERVICE_NAME`, then `"unknown"`                                                            |
| `region`                      | `"us" \| "eu"`                    | all                | Region of your Maple organization. Defaults to `"us"`. Server and Cloudflare fall back to `MAPLE_REGION`. Ignored when an endpoint is set in config or env                                                        |
| `endpoint`                    | `string`                          | all                | Ingest base URL, for a proxy, collector or [Maple Local](/docs/local-mode). Overrides `region`. Server and Cloudflare fall back to `MAPLE_ENDPOINT`, then `OTEL_EXPORTER_OTLP_ENDPOINT`, then the region's ingest |
| `ingestKey`                   | `string`                          | all                | Ingest key, sent as `Authorization: Bearer`. Server and Cloudflare fall back to `MAPLE_INGEST_KEY`. Maple's hosted ingest rejects requests without one                                                            |
| `serviceVersion`              | `string`                          | all                | Service version. Server and Cloudflare fall back to the commit SHA from the environment                                                                                                                           |
| `serviceNamespace`            | `string`                          | all                | Logical group, emitted as the `service.namespace` resource attribute                                                                                                                                              |
| `environment`                 | `string`                          | all                | Deployment environment. Server and Cloudflare fall back to `MAPLE_ENVIRONMENT`, `RAILWAY_ENVIRONMENT_NAME`, `DEPLOYMENT_ENV`, then `"development"`                                                                |
| `repositoryUrl`               | `string`                          | server, Cloudflare | Repository URL, emitted as `vcs.repository.url.full`. Falls back to `MAPLE_REPOSITORY_URL`, then GitHub Actions or Vercel git metadata                                                                            |
| `attributes`                  | `Record<string, unknown>`         | all                | Extra resource attributes. They take precedence over `OTEL_RESOURCE_ATTRIBUTES` entries with the same key                                                                                                         |
| `privacy`                     | `PrivacyOptions`                  | browser            | Consent gating, visitor-id storage and email capture. See [Privacy](/docs/sdks/effect-client#privacy)                                                                                                             |
| `replay`                      | `ClientReplayConfig`              | browser            | Session replay settings. See [Session Replay & Sessions](/docs/sdks/effect-client#session-replay-and-sessions)                                                                                                    |
| `emitSessionMeta`             | `boolean`                         | browser            | Post session metadata rows for sessions without a recording. Default `true`                                                                                                                                       |
| `maxBatchSize`                | `number`                          | server, browser    | Max telemetry items per export batch                                                                                                                                                                              |
| `loggerExportInterval`        | `Duration.Input`                  | server, browser    | Export interval for logs                                                                                                                                                                                          |
| `metricsExportInterval`       | `Duration.Input`                  | server, browser    | Export interval for metrics                                                                                                                                                                                       |
| `tracerExportInterval`        | `Duration.Input`                  | server, browser    | Export interval for traces                                                                                                                                                                                        |
| `shutdownTimeout`             | `Duration.Input`                  | server, browser    | Graceful shutdown timeout                                                                                                                                                                                         |
| `excludeLogSpans`             | `boolean`                         | all                | Skip Effect log spans in OTLP log attributes. Default `false`                                                                                                                                                     |
| `dropSpanNames`               | `ReadonlyArray<string>`           | all                | Drop spans whose name starts with any of these prefixes. Only the matching span is dropped; its children still export                                                                                             |
| `dropSpanSubtrees`            | `ReadonlyArray<string>`           | all                | Drop spans whose name starts with any of these prefixes, together with every span under them. See [Dropping spans](#dropping-spans)                                                                               |
| `dropSpan`                    | `(span: FinishedSpan) => boolean` | all                | Called with each finished span (`name`, `kind`, `attributes`, `exit`). Return `true` to drop it                                                                                                                   |
| `anticipatedErrorIdentifiers` | `ReadonlyArray<string>`           | all                | `_tag` / `Error.name` values of expected failures. Spans failing only with these export as `Ok`, with no `exception` event                                                                                        |
| `isAnticipatedError`          | `(error: unknown) => boolean`     | all                | Predicate form of `anticipatedErrorIdentifiers`, for failures a tag can't single out                                                                                                                              |

### Dropping spans

Use `dropSpanNames` to remove a span on its own, for example protocol chatter. Use `dropSpanSubtrees` to remove a noisy loop entirely, such as an idle job-queue poll and every query it runs:

```typescript
const TracerLive = Maple.layer({
	serviceName: "worker",
	dropSpanSubtrees: ["JobQueue.poll"],
	anticipatedErrorIdentifiers: ["DuplicateDocument"],
})
```

A dropped subtree starts unsampled. Outgoing HTTP calls under it send `traceparent` with the sampled flag off, so downstream services that honour the flag drop their part of the trace too. Errors inside the subtree never reach error tracking, so keep real work (processing a job the poll found) outside the dropped span, or drop only the empty polls with `dropSpan`. Effect's span levels give the same result without a Maple option: `Effect.withSpan("JobQueue.poll", { level: "Debug" })` with `Tracer.MinimumTraceLevel` set to `"Info"`.

### Expected failures

`anticipatedErrorIdentifiers` and `isAnticipatedError` keep expected outcomes (a duplicate, a not-found, a rejected login) visible as traces without counting them as errors. A span still exports as `Error` if its cause contains a defect, or if it is a server span that answered with a 5xx. Spans whose failure carries Effect's `ErrorReporter.ignore` flag, such as unmatched-route 404s, are dropped.

> In Effect 3, duration fields use the `Duration.DurationInput` type instead of `Duration.Input`.

The region endpoints are `https://ingest.maple.dev` (US) and `https://ingest.eu.maple.dev` (EU). An ingest key only works in the region it was created in. See [Regions](/docs/reference/regions).

---

# Effect SDK on servers

> Instrumentation · https://maple.dev/docs/sdks/effect-server

Set up the Effect SDK on Node.js, Bun, or Deno with environment-variable auto-detection.

The server entry point of `@maple-dev/effect-sdk` runs on Node.js, Bun, and Deno. It uses Effect's `Otlp.layerJson` exporter with a background fiber that batches and ships telemetry to Maple's ingest endpoint, and reads its configuration from environment variables when it is not passed in code.

<div class="flex flex-wrap gap-2 mb-8 not-prose">
    <span class="text-[10px] uppercase tracking-wider px-2 py-1 border border-border text-fg-muted">Node.js</span>
    <span class="text-[10px] uppercase tracking-wider px-2 py-1 border border-border text-fg-muted">Bun</span>
    <span class="text-[10px] uppercase tracking-wider px-2 py-1 border border-border text-fg-muted">Deno</span>
</div>

> Already installed the SDK? If not, see the [install instructions](/docs/sdks/effect#install).

## Quick start

```typescript
import { Maple } from "@maple-dev/effect-sdk"
import { Effect } from "effect"

const TracerLive = Maple.layer({
	serviceName: "my-effect-app",
})

const program = Effect.gen(function* () {
	yield* Effect.log("Hello from Effect!")
}).pipe(Effect.withSpan("hello-maple"))

Effect.runPromise(program.pipe(Effect.provide(TracerLive)))
```

The default import (`@maple-dev/effect-sdk`) resolves to the server build under Node.js. You can also import the entry point explicitly:

```typescript
import { Maple } from "@maple-dev/effect-sdk/server"
```

Set `MAPLE_INGEST_KEY` to an ingest key from **Settings → Ingestion** (use the private key, `maple_sk_…`) and the SDK picks it up. With nothing else set, telemetry goes to the US ingest, `https://ingest.maple.dev`. For an EU organization, set `MAPLE_REGION=eu` or pass `region: "eu"`, and it goes to `https://ingest.eu.maple.dev`.

The server layer always exports; there is no disable switch. A missing ingest key does not turn export off, so a keyless app pointed at [Maple Local](/docs/local-mode) or your own OTLP collector still sends telemetry. Keyless against Maple's hosted ingest cannot work: every request is rejected with `401`, and the SDK logs a one-time warning. For local development that exports nothing, point `MAPLE_ENDPOINT` at a sink you control, or use `MapleFlush.make`, which does nothing without a key.

## Environment variables

The server layer reads these variables when the matching config option is not set:

| Variable                      | Used for                                                                             |
| ----------------------------- | ------------------------------------------------------------------------------------ |
| `MAPLE_INGEST_KEY`            | Ingest key, sent as a bearer token. Omitted from requests when unset                 |
| `MAPLE_REGION`                | `us` (default) or `eu`. Only used when no endpoint is set anywhere                   |
| `MAPLE_ENDPOINT`              | Ingest base URL, for a proxy, collector or Maple Local. Overrides the region         |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Endpoint fallback when `MAPLE_ENDPOINT` is unset                                     |
| `OTEL_SERVICE_NAME`           | Service name when `serviceName` is not passed. Without either, the name is `unknown` |
| `MAPLE_ENVIRONMENT`           | Deployment environment                                                               |
| `MAPLE_REPOSITORY_URL`        | Repository URL, emitted as `vcs.repository.url.full`                                 |
| `OTEL_RESOURCE_ATTRIBUTES`    | Extra resource attributes as `key=value` pairs                                       |

**Ingest endpoint:** `endpoint` option, then `MAPLE_ENDPOINT`, then `OTEL_EXPORTER_OTLP_ENDPOINT`, then the ingest for `region` or `MAPLE_REGION`, then `https://ingest.maple.dev`. Any explicit endpoint beats any region setting.

**Commit SHA** (first match wins):

1. `COMMIT_SHA`
2. `RAILWAY_GIT_COMMIT_SHA`
3. `VERCEL_GIT_COMMIT_SHA`
4. `CF_PAGES_COMMIT_SHA`
5. `RENDER_GIT_COMMIT`

**Deployment environment** (first match wins):

1. `MAPLE_ENVIRONMENT`
2. `RAILWAY_ENVIRONMENT_NAME`
3. `DEPLOYMENT_ENV`
4. Falls back to `"development"`

`VERCEL_ENV` and `NODE_ENV` are **not** read. On any platform outside that list, set `MAPLE_ENVIRONMENT` explicitly or your telemetry lands under `development`.

**Repository URL:** `MAPLE_REPOSITORY_URL`, then `GITHUB_SERVER_URL` + `GITHUB_REPOSITORY` (GitHub Actions), then `VERCEL_GIT_REPO_OWNER` + `VERCEL_GIT_REPO_SLUG`.

The SDK also detects the **runtime** (Node.js, Bun, Deno) and **cloud provider** (Railway, Vercel, Cloudflare, Render) and adds them as `maple.runtime` and `maple.provider` resource attributes.

## Deployment platform notes

Managed platforms expose the commit SHA automatically. The **environment** is only detected on Railway. Everywhere else, set `MAPLE_ENVIRONMENT` yourself:

| Platform         | Commit SHA env var       | Environment                                 |
| ---------------- | ------------------------ | ------------------------------------------- |
| Railway          | `RAILWAY_GIT_COMMIT_SHA` | `RAILWAY_ENVIRONMENT_NAME` (automatic)      |
| Vercel           | `VERCEL_GIT_COMMIT_SHA`  | Set `MAPLE_ENVIRONMENT`                     |
| Cloudflare Pages | `CF_PAGES_COMMIT_SHA`    | Set `MAPLE_ENVIRONMENT`                     |
| Render           | `RENDER_GIT_COMMIT`      | Set `MAPLE_ENVIRONMENT`                     |
| Other hosts      | `COMMIT_SHA` (set in CI) | Set `MAPLE_ENVIRONMENT` or `DEPLOYMENT_ENV` |

On other hosts, set `COMMIT_SHA` in your build pipeline and `MAPLE_ENVIRONMENT` at runtime.

## Server-side `track()`

Some funnel steps only happen on the backend: a `signup_completed` in a webhook handler, a `plan_started` when billing confirms a subscription. `MapleEvents` posts those to Maple's [product events endpoint](/docs/product-events/api) so they land next to the browser SDK's `track()` calls, keyed to the same person.

```typescript
import { MapleEvents } from "@maple-dev/effect-sdk/server"
import { Effect, Layer } from "effect"

const EventsLive = MapleEvents.layer({ serviceName: "billing" })

const onSubscriptionCreated = Effect.fn("onSubscriptionCreated")(function* (
	userId: string,
	orgId: string,
	plan: string,
) {
	const events = yield* MapleEvents.MapleEvents
	yield* events.track("plan_started", { userId, groupId: orgId, attributes: { plan } })
})
```

`track(name, options)` buffers the event and returns immediately; batches go out every 5 seconds, at 100 events, and when the layer's scope closes. It never fails the caller: a rejected batch is dropped with a rate-limited console warning. `options` are all optional: `userId`, `groupId`, `visitorId` (the browser cookie value, if your backend has it), `sessionId`, `timestamp`, `url`, `pagePath`, and `attributes` (coerced and capped exactly like the browser `track()`).

The endpoint and ingest key resolve the same way as the tracer (`endpoint`/`ingestKey` in config, else `MAPLE_ENDPOINT`, `OTEL_EXPORTER_OTLP_ENDPOINT` or the region, and `MAPLE_INGEST_KEY`). Without a key, events are dropped with a one-shot warning. Outside an Effect runtime, `MapleEvents.makeHandle(config)` returns a plain `{ track, flush, dispose }`. Call `dispose()` on shutdown so the last batch is sent.

## Verify

1. Start your application and send it a few requests.
2. In Maple, open **Explore → Traces**. The layer exports spans every 5 seconds and metrics every 10 seconds by default.
3. Each request should show up as one trace, with the spans you created with `Effect.withSpan` (or an HTTP server span, if you use Effect's HTTP server) at the root.

Your service also appears on the **Services** page once its first spans arrive.

## Troubleshooting

- **A warning about sending without an ingest key.** `MAPLE_INGEST_KEY` is not set in the process environment and no `ingestKey` was passed. Every request to Maple's ingest is rejected with `401`.
- **`401` responses with a key set.** The key is wrong, or it belongs to the other region. A key from an EU organization needs `MAPLE_REGION=eu` (or `region: "eu"`), and a US key needs the default region.
- **Telemetry goes somewhere else.** If `MAPLE_ENDPOINT` or `OTEL_EXPORTER_OTLP_ENDPOINT` is set, it wins over the region. A Kubernetes operator or platform can set `OTEL_EXPORTER_OTLP_ENDPOINT` for you; unset it or set `MAPLE_ENDPOINT` to the ingest you want.
- **Network.** From the machine running the app, run `curl -i https://ingest.maple.dev/v1/traces -X POST`. Any HTTP status code means the host can reach Maple.

## Next steps

- [Explore traces](/docs/explore/traces)
- [Track errors](/docs/errors/overview)
- [Create alert rules](/docs/alerting/alert-rules)
- [OpenTelemetry conventions](/docs/concepts/otel-conventions)

---

# Effect SDK in the browser

> Instrumentation · https://maple.dev/docs/sdks/effect-client

Set up the Effect SDK in browser environments with explicit configuration and auto-captured browser metadata.

The browser entry point of `@maple-dev/effect-sdk` runs in single-page apps and any other browser context. Unlike the server build, it reads no environment variables: browsers have no `process.env`, so all configuration is passed to `Maple.layer()`.

<div class="flex flex-wrap gap-2 mb-8 not-prose">
    <span class="text-[10px] uppercase tracking-wider px-2 py-1 border border-border text-fg-muted">Browsers</span>
</div>

> Already installed the SDK? If not, see the [install instructions](/docs/sdks/effect#install).

## Quick start

```typescript
import { Maple } from "@maple-dev/effect-sdk/client"
import { Effect } from "effect"

const TracerLive = Maple.layer({
	serviceName: "my-frontend",
	ingestKey: "maple_pk_...",
	region: "eu", // omit for the US region
})

const program = Effect.log("Hello!").pipe(Effect.withSpan("hello"))

Effect.runPromise(program.pipe(Effect.provide(TracerLive)))
```

`serviceName` is the only required option. Without `endpoint`, the layer sends to the ingest for `region`: `https://ingest.maple.dev` for `"us"` (the default) or `https://ingest.eu.maple.dev` for `"eu"`. Set `endpoint` only for a proxy or [Maple Local](/docs/local-mode); it overrides `region`.

Import from `/client` explicitly. Bundlers that apply the `node` export condition, such as the server side of a framework build, resolve the bare `@maple-dev/effect-sdk` import to the server build.

## Auto-captured browser attributes

The client layer reads `globalThis.navigator` and `Intl.DateTimeFormat` to fill in resource attributes:

- `user_agent.original`: `navigator.userAgent`
- `browser.language`: `navigator.language`
- `browser.timezone`: `Intl.DateTimeFormat().resolvedOptions().timeZone`
- `maple.sdk.type`: always `"client"`, so server and browser spans can be filtered apart

Attributes you pass in the `attributes` option are merged on top of these.

## Session replay and sessions

The browser presets (`Maple.layer` and `MapleFlush.make`) record **rrweb session replays by default**, without a separate browser SDK. Every span carries a `session.id`, the session appears in [Sessions](/docs/session-replay/browser-sdk) with its linked traces, and the recording plays back next to them.

```typescript
const TracerLive = Maple.layer({
	serviceName: "my-frontend",
	ingestKey: "maple_pk_...",
	replay: {
		sampleRate: 0.1, // record 10% of sessions (default 1)
	},
})
```

| Option                 | Default | Description                                                                        |
| ---------------------- | ------- | ---------------------------------------------------------------------------------- |
| `replay.enabled`       | `true`  | Record rrweb session replays.                                                      |
| `replay.sampleRate`    | `1`     | Fraction of sessions to record, 0 to 1.                                            |
| `replay.maskAllInputs` | `true`  | Mask all `<input>` values in the recording.                                        |
| `replay.maskAllText`   | `false` | Mask all text in the recording.                                                    |
| `emitSessionMeta`      | `true`  | Post session metadata rows so unrecorded sessions still appear in the Sessions UI. |

How it behaves:

- **Sampling still yields sessions.** When replay is disabled or a session isn't sampled, the SDK still posts session metadata rows. The session shows up in the Sessions UI with its linked traces, just without a recording. Set `emitSessionMeta: false` to turn that off too.
- **Tab lifecycle.** Recording suspends when the tab is hidden (flushing the tail with `keepalive`) and resumes when it becomes visible again. Sessions survive reloads within a tab and rotate after 30 minutes of inactivity (24-hour hard cap).
- **Identify users** at any point; the id is attached when the session's next metadata row is posted and stamped as `user.id` on future spans. Pass `null` or `undefined` after sign-out to make future telemetry anonymous again:

```typescript
import { identify } from "@maple-dev/effect-sdk/client"

identify(user.id)
identify(null)
```

- **Clear the identity** on logout with `clearIdentity()` (the explicit inverse of `identify()`); metadata rows and spans go back to anonymous while the session continues:

```typescript
import { clearIdentity } from "@maple-dev/effect-sdk/client"

clearIdentity()
```

- **Interop with `@maple-dev/browser`.** If the standalone browser SDK is also on the page, it owns the session. This SDK's recorder and row emission stand down automatically, and spans link to that session instead. Run replay from one SDK, not both.

## Privacy

The `privacy` option controls consent, the persistent visitor id and what identity data is sent:

| Option                         | Default     | Description                                                                                                                                                                         |
| ------------------------------ | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `privacy.requireConsent`       | `false`     | Capture nothing until `setConsent(true)` is called                                                                                                                                  |
| `privacy.persistVisitorId`     | `true`      | Store a persistent visitor id                                                                                                                                                       |
| `privacy.persistence`          | `"browser"` | `"memory"` stores nothing on the device (no cookies, Web Storage or IndexedDB, no visitor id). Each page load is a new session                                                      |
| `privacy.crossSubdomainCookie` | `true`      | Scope the visitor-id cookie to the registered domain, so sibling subdomains share one visitor                                                                                       |
| `privacy.cookieDomain`         |             | Explicit cookie `Domain` (without the leading dot). `""` forces a host-only cookie                                                                                                  |
| `privacy.captureUserEmail`     | `true`      | Send the email passed to `identify()`                                                                                                                                               |
| `privacy.respectDoNotTrack`    | `false`     | Treat `navigator.doNotTrack` like Global Privacy Control                                                                                                                            |
| `privacy.sanitizeUrl`          |             | Function that rewrites every URL before it leaves the page. Runs after the built-in redaction of credential-shaped query and fragment parameters such as `token` and `access_token` |

Global Privacy Control is honored by default: when the browser sends it, the SDK does not store the persistent visitor id. Capture itself continues without it.

With `requireConsent: true`, grant or revoke consent from your consent banner:

```typescript
import { setConsent } from "@maple-dev/effect-sdk/client"

setConsent(true)
```

With `requireConsent`, `Maple.layer` does not export metrics, because it cannot separate values recorded before consent was granted. Use `MapleFlush.make` if you need browser metrics after consent.

## Use the public ingest key

Anything in your bundle is visible to every visitor. Use the **Public key** (`maple_pk_…`) from **Settings → Ingestion**, which the page marks for browser and client-side SDKs. Never ship the private key (`maple_sk_…`) in client code. Ingest keys can only write telemetry; they cannot read data back out.

## Bundle size

The `/client` entry point leaves out the Node-only resource detector and platform helpers, so the base bundle ships the OTLP JSON exporter and Effect's tracer and logger primitives (about 13 kB). The replay engine, rrweb included, sits behind a dynamic import in a separate chunk (about 360 kB). It is only fetched when replay is enabled _and_ the session is sampled, so apps that set `replay: { enabled: false }` never download it. `effect` is a peer dependency; if your app already uses Effect on the client, the SDK adds only its own layer code.

## Configuration reference

See the full [configuration table](/docs/sdks/effect#configuration-reference) on the Effect SDK page. In the browser, `serviceName` is required and every other option is optional.

## Verify

1. Load a page of your app that runs code inside the layer.
2. In Maple, open **Explore → Traces** and filter on your service name. The layer exports spans every 5 seconds by default.
3. With replay on, open **Explore → Replays** to see the session and its linked traces.

## Troubleshooting

- **A console warning about sending without an ingest key.** Pass `ingestKey`, or point `endpoint` at a proxy that adds the key.
- **`401` responses.** The key is wrong, or it belongs to the other region. An EU key needs `region: "eu"`.
- **Requests blocked by the browser.** Check the Network tab. Maple's ingest answers CORS preflights from any origin, so a blocked request usually means an ad blocker or a Content Security Policy without `connect-src` for the ingest host.
- **Nothing sent at all.** With `privacy.requireConsent: true`, the SDK captures nothing until `setConsent(true)` is called.

## Next steps

- [Session replays](/docs/session-replay/replays)
- [Web analytics](/docs/product-events/web-analytics)
- [Explore traces](/docs/explore/traces)
- [OpenTelemetry conventions](/docs/concepts/otel-conventions)

---

# Effect SDK on Cloudflare Workers

> Instrumentation · https://maple.dev/docs/sdks/effect-cloudflare

Set up the Effect SDK on Cloudflare Workers with explicit flush() in ctx.waitUntil and in-isolate buffering.

The `/cloudflare` entry point of `@maple-dev/effect-sdk` is built for Cloudflare Workers. A Worker has no long-running process: the isolate handles a request, returns a response, and the runtime may suspend it at any moment. A background export fiber does not run reliably between invocations, so this entry point buffers telemetry inside the isolate and sends it when you call `flush(env)`, which you schedule with `ctx.waitUntil`.

<div class="flex flex-wrap gap-2 mb-8 not-prose">
    <span class="text-[10px] uppercase tracking-wider px-2 py-1 border border-border text-fg-muted">Cloudflare Workers</span>
</div>

> Already installed the SDK? If not, see the [install instructions](/docs/sdks/effect#install).

## Why Workers are different

- **No background fiber.** Spans, logs and metrics accumulate in memory inside the isolate and are only sent when you call `flush()`.
- **Configuration on first flush.** You can call `make()` at module scope without `env`. The SDK reads the endpoint, ingest key and resource attributes from `env` on the first `flush(env)` call.
- **Manual lifecycle.** `ctx.waitUntil(telemetry.flush(env))` keeps the isolate alive long enough to send the batch after the response is returned.

## Quick start

```typescript
import * as MapleCloudflareSDK from "@maple-dev/effect-sdk/cloudflare"
import { HttpRouter } from "effect/http"
import { Layer } from "effect"
import { Routes } from "./routes"

const telemetry = MapleCloudflareSDK.make({ serviceName: "my-worker" })

const { handler } = HttpRouter.toWebHandler(Routes.pipe(Layer.provideMerge(telemetry.layer)))

export default {
	async fetch(req: Request, env: Env, ctx: ExecutionContext) {
		const res = await handler(req)
		ctx.waitUntil(telemetry.flush(env))
		return res
	},
}
```

`telemetry.layer` must be merged into the same Effect runtime that runs your routes, because the tracer has to be in scope when spans are created. A separate per-request runtime does not work.

Runtimes that own a per-request scope can use `telemetry.requestLayer` instead of calling `flush` themselves. It is the same layer plus a flush when the scope closes, and it reads `env` from the `WorkerEnvironment` service exported by this entry point.

## No-op mode

When no ingest key is set (neither `ingestKey` nor `MAPLE_INGEST_KEY`), the SDK runs in no-op mode. Each `flush()` still empties the buffers so they do not grow for the life of the isolate, but nothing is sent. The first flush logs one `console.info` line starting with `[MapleCloudflareSDK] no ingest key configured`. The same code can therefore run in preview environments that have no key.

## Failure handling

A flush fails when the request throws (for example a network error) or the ingest answers with any non-2xx status, including `401` and `429`. The failed batch goes back into the buffer ahead of newer telemetry, and that signal enters a 60-second cooldown. During the cooldown, flushes for that signal skip the request and log a warning. Traces, logs and metrics each have their own cooldown. Errors are logged to `console.error` and never reach your handler.

## Cloudflare-specific config

`make()` accepts the [common options](/docs/sdks/effect#configuration-reference) marked for Cloudflare (`serviceName`, `region`, `endpoint`, `ingestKey`, `serviceVersion`, `serviceNamespace`, `environment`, `repositoryUrl`, `attributes`), the [span options](/docs/sdks/effect#dropping-spans) (`excludeLogSpans`, `dropSpanNames`, `dropSpanSubtrees`, `dropSpan`, `anticipatedErrorIdentifiers`, `isAnticipatedError`), plus these:

| Option        | Type     | Default       | Description                              |
| ------------- | -------- | ------------- | ---------------------------------------- |
| `tracesPath`  | `string` | `/v1/traces`  | OTLP traces path appended to `endpoint`  |
| `logsPath`    | `string` | `/v1/logs`    | OTLP logs path appended to `endpoint`    |
| `metricsPath` | `string` | `/v1/metrics` | OTLP metrics path appended to `endpoint` |

An error that crossed an HTTP boundary is a decoded body rather than the class that raised it, so a failure shaped `{ error: { _tag } }` (the envelope convention many APIs use) is matched on the body's `_tag`. Client-side spans classify the same as the server-side ones they mirror, with no separate identifiers to configure.

`dropSpanNames` suppresses protocol-level chatter. For example, `["McpServer/Notifications."]` drops MCP notification spans without dropping handler spans. Use `dropSpanSubtrees` instead when the children should go too.

## Endpoint resolution

The endpoint is the first of:

1. `config.endpoint`
2. `env.MAPLE_ENDPOINT`
3. `env.OTEL_EXPORTER_OTLP_ENDPOINT`
4. The ingest for `config.region`, then for `env.MAPLE_REGION`: `https://ingest.maple.dev` for `us`, `https://ingest.eu.maple.dev` for `eu`
5. `https://ingest.maple.dev`

Any explicit endpoint beats any region. On hosted Maple, a `MAPLE_INGEST_KEY` secret is usually all you need, plus `MAPLE_REGION=eu` for an EU organization. Other variables read from `env`: `OTEL_SERVICE_NAME`, `MAPLE_ENVIRONMENT`, `MAPLE_REPOSITORY_URL`, `OTEL_RESOURCE_ATTRIBUTES`, and the commit SHA variables listed on the [server page](/docs/sdks/effect-server#environment-variables).

## Verify

1. Set the key as a secret with `wrangler secret put MAPLE_INGEST_KEY` (an ingest key from **Settings → Ingestion**), then deploy with `wrangler deploy`.
2. Send a few requests to the Worker.
3. In Maple, open **Explore → Traces**. Each request should show up as one trace, rooted at the server span for the request.

## Troubleshooting

Run `wrangler tail` and look for lines starting with `[MapleCloudflareSDK]`:

- **`no ingest key configured`.** The Worker has no `MAPLE_INGEST_KEY` in `env`. Set it with `wrangler secret put MAPLE_INGEST_KEY`.
- **`flush failed; cooldown 60s`** with `OTLP 401`. The key is wrong, or it belongs to the other region. An EU key needs `MAPLE_REGION=eu`.
- **`flush skipped (cooldown ...)`.** An earlier flush failed. Look further back in the log for the failure.
- **No lines and no data.** `ctx.waitUntil(telemetry.flush(env))` is not being called, so the isolate stops before the request is sent. Or `telemetry.layer` is provided to a different runtime than your routes.

## Next steps

- [Explore traces](/docs/explore/traces)
- [Cloudflare integration](/docs/integrations/cloudflare)
- [Create alert rules](/docs/alerting/alert-rules)
- [OpenTelemetry conventions](/docs/concepts/otel-conventions)

---

# Cloudflare Workers instrumentation

> Instrumentation · https://maple.dev/docs/guides/instrumentation-cloudflare-workers

Send traces and logs from any Cloudflare Worker to Maple with Workers Observability OTLP destinations. No SDK or code changes.

Cloudflare Workers Observability can export a Worker's traces and logs over OTLP to any endpoint. This guide points that export at Maple. The Workers runtime records the spans itself, so there is no SDK to install, no exporter to flush in `ctx.waitUntil`, and no code to change. It works for any Worker, whatever language or framework it is written in.

If your Worker is written with Effect, the [Effect SDK for Cloudflare Workers](/docs/sdks/effect-cloudflare) gives you your own spans, logs and metrics as well. Both can run on the same Worker.

## What you get

Cloudflare records these automatically:

- **Handler spans** for `fetch`, `scheduled` and `queue` invocations.
- **Outbound `fetch` calls**, with timing and status code.
- **Binding calls**, such as KV and R2 operations and Durable Object invocations, including RPC to Durable Objects.
- **Logs** from `console.log`, `console.error` and the other `console` methods, plus the invocation log for each request.

Spans carry Cloudflare attributes (`cloudflare.script_name`, `cloudflare.colo`, `cloudflare.ray_id`, `faas.trigger`, CPU and wall time, and more). Maple recognizes them and shows a **Cloudflare Worker** panel on the span, with the script, version, handler, CPU and wall time, TTFB, edge location and Ray ID.

Metrics are not exported this way. For Worker request counts, error rates and CPU time, connect the [Cloudflare integration](/docs/integrations/cloudflare), which pulls them from Cloudflare's analytics API.

## Prerequisites

- A Cloudflare account on the **Workers Paid** plan. OTLP export is not available on Workers Free. Cloudflare bills exported events beyond the included allowance; see Cloudflare's [pricing for exported events](https://developers.cloudflare.com/workers/observability/exporting-opentelemetry-data/).
- Permission to edit Workers Observability settings in the Cloudflare dashboard.
- An ingest key from **Settings → Ingestion** in Maple. Use the private key (`maple_sk_…`). It is stored in Cloudflare and never reaches the browser.
- Your organization's ingest endpoint: `https://ingest.maple.dev`, or `https://ingest.eu.maple.dev` for EU organizations.

## 1. Create the destinations

A destination carries one signal, so create two: one for traces and one for logs.

1. In the Cloudflare dashboard, open **Workers & Pages → Observability**, then the **Destinations** tab.
2. Click **Add destination** and fill in:

    | Field            | Value                                     |
    | ---------------- | ----------------------------------------- |
    | Destination name | `maple-traces`                            |
    | Destination type | Traces                                    |
    | OTLP endpoint    | `https://ingest.maple.dev/v1/traces`      |
    | Custom headers   | `Authorization`: `Bearer YOUR_INGEST_KEY` |

3. Save, then add a second destination for logs:

    | Field            | Value                                     |
    | ---------------- | ----------------------------------------- |
    | Destination name | `maple-logs`                              |
    | Destination type | Logs                                      |
    | OTLP endpoint    | `https://ingest.maple.dev/v1/logs`        |
    | Custom headers   | `Authorization`: `Bearer YOUR_INGEST_KEY` |

The endpoint is the full signal URL, not the base URL. Use `https://ingest.eu.maple.dev/v1/traces` and `https://ingest.eu.maple.dev/v1/logs` for EU organizations.

Destinations belong to the Cloudflare account, not to one Worker. Every Worker in the account can reference the same two.

## 2. Point your Worker at them

Add an `observability` block to the Worker's Wrangler config. The names must match the destination names exactly.

```jsonc
// wrangler.jsonc
{
	"observability": {
		"traces": {
			"enabled": true,
			"destinations": ["maple-traces"],
		},
		"logs": {
			"enabled": true,
			"destinations": ["maple-logs"],
		},
	},
}
```

Or in `wrangler.toml`:

```toml
[observability.traces]
enabled = true
destinations = ["maple-traces"]

[observability.logs]
enabled = true
destinations = ["maple-logs"]
```

Deploy with `wrangler deploy`. Export starts with the next invocation.

### Optional settings

Both `traces` and `logs` accept two more keys:

- **`head_sampling_rate`**: a number from `0` to `1`, the share of invocations to record. `1` (every invocation) is the default. For high-traffic Workers, `0.1` keeps cost down while leaving enough traces to investigate.
- **`persist`**: defaults to `true`, which also keeps the data in Cloudflare's own dashboard. Set it to `false` to send it only to Maple and skip Cloudflare's storage charges.

```jsonc
"traces": {
	"enabled": true,
	"destinations": ["maple-traces"],
	"head_sampling_rate": 0.1,
	"persist": false
}
```

## Verify

1. Send a few requests to the Worker.
2. In Maple, open **Explore → Traces** and filter by the service. Cloudflare names the service after the Worker script.
3. Each request is one trace: a root span for the handler, with `fetch` and binding spans nested under it. Click a span to see the **Cloudflare Worker** panel.
4. Open **Explore → Logs** to see `console` output from the same invocations.

The Worker also appears on the **Services** page once its first spans arrive.

## Things to know

- **Traces stop at the Worker.** Cloudflare does not yet propagate trace context to services outside Cloudflare, so a backend called by the Worker starts its own trace instead of joining the Worker's.
- **Some spans show 0 ms.** The Workers runtime only advances the clock on I/O, as a Spectre mitigation. Pure CPU work between two I/O calls reports no duration.
- **Non-2xx responses are marked `Error`.** Cloudflare sets the span status to `Error` for any non-2xx `fetch`, including 404s from bots probing `/wp-admin`. Maple does not open error issues for a 4xx span that has no exception attached, so this noise stays out of [Errors](/docs/errors/overview). 5xx responses and thrown exceptions still count.
- **Span and attribute names can change.** Cloudflare's tracing is in beta and its names are not final.

## Troubleshooting

- **Nothing arrives.** Check that the destination names in the Wrangler config match the dashboard exactly, that the Worker was redeployed after the change, and that the destination is enabled in **Observability → Destinations**.
- **`401` from the destination.** The key is wrong, was copied from the other region, or the header is malformed. The header name is `Authorization` and the value is `Bearer ` followed by the key. See [Ingest API status codes](/docs/reference/ingest#status-codes).
- **Traces arrive but logs do not (or the reverse).** Each destination handles one signal. Check that the traces destination uses `/v1/traces` with type **Traces**, and the logs destination uses `/v1/logs` with type **Logs**.
- **The Worker has no Cloudflare edge section on the service map.** With the [Cloudflare integration](/docs/integrations/cloudflare) connected, Maple matches the traced service to the script by service name or `faas.name`. Both come from Cloudflare, so this only breaks if the script was renamed.

## Next steps

- [Cloudflare integration](/docs/integrations/cloudflare): Worker metrics, zones, Queues and Durable Objects.
- [Effect SDK on Cloudflare Workers](/docs/sdks/effect-cloudflare): your own spans, logs and metrics from inside the Worker.
- [Explore traces](/docs/explore/traces)
- [Create alert rules](/docs/alerting/alert-rules)

---

# Node.js instrumentation

> Instrumentation · https://maple.dev/docs/guides/instrumentation-nodejs

Instrument a Node.js application with OpenTelemetry and send traces, logs, and metrics to Maple.

This guide sets up the OpenTelemetry Node.js SDK so your application sends traces, logs and metrics to Maple, with automatic instrumentation for HTTP servers, frameworks and database clients.

To have a coding agent do this setup, use the [maple-onboard](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-onboard) skill, and [maple-audit](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-audit) to check an existing setup.

## Prerequisites

- Node.js 18.19+ or 20.6+ (the versions that support `node --import`)
- An ingest key from **Settings → Ingestion** in Maple. Use the private key (`maple_sk_…`) for server applications.

## Install

```bash
npm install @opentelemetry/api \
  @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/instrumentation \
  @opentelemetry/resources \
  @opentelemetry/sdk-logs \
  @opentelemetry/sdk-metrics \
  @opentelemetry/exporter-trace-otlp-proto \
  @opentelemetry/exporter-logs-otlp-proto \
  @opentelemetry/exporter-metrics-otlp-proto
```

## Configure

Create `tracing.mjs` next to your entry point. It starts the SDK before any of your application code loads.

```javascript
// tracing.mjs
import { register } from "node:module"
import { NodeSDK } from "@opentelemetry/sdk-node"
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node"
import { resourceFromAttributes } from "@opentelemetry/resources"
import { BatchLogRecordProcessor } from "@opentelemetry/sdk-logs"
import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics"
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto"
import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-proto"
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-proto"

// Lets the instrumentations patch ES modules. CommonJS apps can drop this line.
register("@opentelemetry/instrumentation/hook.mjs", import.meta.url)

const MAPLE_ENDPOINT = "https://ingest.maple.dev" // EU: https://ingest.eu.maple.dev
const MAPLE_KEY = "YOUR_INGEST_KEY"

const headers = { authorization: `Bearer ${MAPLE_KEY}` }

const sdk = new NodeSDK({
	resource: resourceFromAttributes({
		"service.name": "my-node-app",
		"deployment.environment.name": process.env.NODE_ENV || "development",
		"vcs.repository.url.full": "https://github.com/acme/my-node-app",
		"vcs.ref.head.revision": process.env.GITHUB_SHA ?? process.env.GIT_COMMIT,
	}),
	traceExporter: new OTLPTraceExporter({ url: `${MAPLE_ENDPOINT}/v1/traces`, headers }),
	logRecordProcessors: [
		new BatchLogRecordProcessor(new OTLPLogExporter({ url: `${MAPLE_ENDPOINT}/v1/logs`, headers })),
	],
	metricReaders: [
		new PeriodicExportingMetricReader({
			exporter: new OTLPMetricExporter({ url: `${MAPLE_ENDPOINT}/v1/metrics`, headers }),
		}),
	],
	instrumentations: [getNodeAutoInstrumentations()],
})

sdk.start()

process.on("SIGTERM", () => {
	sdk.shutdown().finally(() => process.exit(0))
})
```

The example puts the endpoint and key in source. An ingest key can only write telemetry to your organization. It cannot read data or call the Maple API. Keeping it in source means the SDK always starts with a complete configuration, so a deploy that is missing an environment variable cannot silently turn telemetry off. To keep the key out of source, use [environment variables](#environment-variables) instead.

Load the file before your application:

```bash
node --import ./tracing.mjs app.js
```

For a TypeScript entry point run through [tsx](https://tsx.is), load tsx first:

```bash
node --import tsx --import ./tracing.mjs app.ts
```

## Environment variables

The SDK can take its whole configuration from the standard OpenTelemetry variables. With them set, you do not need `tracing.mjs`: the `register` entry point of the auto-instrumentations package starts the SDK and reads the variables.

```bash
export OTEL_SERVICE_NAME="my-node-app"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
export OTEL_TRACES_EXPORTER="otlp"
export OTEL_METRICS_EXPORTER="otlp"
export OTEL_LOGS_EXPORTER="otlp"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production,vcs.repository.url.full=https://github.com/acme/my-node-app"

node --require @opentelemetry/auto-instrumentations-node/register app.js
```

For an ES module app, also pass `--experimental-loader=@opentelemetry/instrumentation/hook.mjs` so the instrumentations can patch ES modules.

`OTEL_EXPORTER_OTLP_ENDPOINT` takes the base URL. The exporters append `/v1/traces`, `/v1/logs` and `/v1/metrics` themselves.

## Auto-instrumentation

`getNodeAutoInstrumentations()` enables the OpenTelemetry instrumentations for common libraries, including `http`, Express, Fastify, `pg`, `mysql2`, `ioredis` and many more. Each incoming request becomes a server span, and each outgoing HTTP call or database query becomes a child span.

To turn off instrumentations you do not want:

```javascript
instrumentations: [
	getNodeAutoInstrumentations({
		"@opentelemetry/instrumentation-fs": { enabled: false },
		"@opentelemetry/instrumentation-dns": { enabled: false },
	}),
],
```

## Custom spans

Wrap your own operations in spans:

```typescript
import { trace, SpanStatusCode } from "@opentelemetry/api"

const tracer = trace.getTracer("my-app")

async function processOrder(orderId: string) {
	return tracer.startActiveSpan("process-order", async (span) => {
		try {
			span.setAttribute("order.id", orderId)
			span.setAttribute("payment.method", "card")
			return await chargePayment(orderId)
		} catch (error) {
			span.recordException(error as Error)
			span.setStatus({ code: SpanStatusCode.ERROR, message: (error as Error).message })
			throw error
		} finally {
			span.end()
		}
	})
}
```

Service map edges come from instrumented client spans that propagate `traceparent` to an instrumented callee, not from attributes such as `peer.service`. See [Service map](/docs/explore/service-map).

## Attributes Maple uses

Most attributes are optional. These unlock specific features.

### Commits and GitHub

Set `vcs.ref.head.revision` (the full 40-character commit SHA) and `vcs.repository.url.full` on the resource. They drive deploy markers, the Releases page and, with the [GitHub integration](/docs/integrations/github), commit hover cards on traces. The resource in [Configure](#configure) already sets both. Read the SHA from your CI or host at build time, such as `GITHUB_SHA` or `VERCEL_GIT_COMMIT_SHA`, not from `git` at runtime:

```bash
export OTEL_RESOURCE_ATTRIBUTES="vcs.repository.url.full=https://github.com/acme/api,vcs.ref.head.revision=$GITHUB_SHA"
```

### Databases

Database instrumentation sets these attributes for you. For a driver without instrumentation, set them on a `Client` span:

```typescript
import { SpanKind } from "@opentelemetry/api"

const user = await tracer.startActiveSpan(
	"SELECT users",
	{
		kind: SpanKind.CLIENT,
		attributes: {
			"db.system.name": "postgresql",
			"db.namespace": "app",
			"db.query.text": "SELECT * FROM users WHERE id = $1",
		},
	},
	async (span) => {
		try {
			return await client.query("SELECT * FROM users WHERE id = $1", [id])
		} finally {
			span.end()
		}
	},
)
```

`db.system.name` and `db.namespace` draw the database node on the [service map](/docs/explore/service-map). `db.query.text` groups statements into query shapes, so record it with placeholders, not bound values. See [Database queries](/docs/concepts/otel-conventions#database-queries) for the full list.

## Log correlation

Log records emitted through the OpenTelemetry log SDK during an active span carry its trace and span IDs, so Maple links each log line to its trace.

The auto-instrumentations include `pino`, `winston` and `bunyan` instrumentations that add the active trace context to each log record. Pino and bunyan records are also sent through the log exporter configured above. Winston needs the `@opentelemetry/winston-transport` package for that.

## Other Node.js frameworks

- Next.js has its own guide: [Next.js instrumentation](/docs/guides/instrumentation-nextjs).
- For Effect applications, use the [Effect SDK](/docs/sdks/effect).

## Verify

1. Start your application and send it a few requests.
2. In Maple, open **Explore → Traces**. The SDK sends spans in batches every 5 seconds by default, and metrics every 60 seconds.
3. Each request should show up as one trace with a single root server span, named after the method and route (for example `GET /api/orders`), and child spans for the queries and outgoing calls it made.

Your service also appears on the **Services** page once its first spans arrive.

## Troubleshooting

- **`401` responses.** The key is wrong, was copied from the other region, or the header is malformed. The header must be `Authorization: Bearer YOUR_INGEST_KEY`. In `OTEL_EXPORTER_OTLP_HEADERS` it is written `Authorization=Bearer YOUR_INGEST_KEY`. See [Ingest API status codes](/docs/reference/ingest#status-codes).
- **Wrong protocol or path.** Maple accepts OTLP over HTTP. Use the `-proto` (or `-http`) exporters, not `-grpc`. Endpoints set in code need the full signal path (`/v1/traces`); `OTEL_EXPORTER_OTLP_ENDPOINT` takes only the base URL.
- **Network.** From the machine running the app, run `curl -i https://ingest.maple.dev/v1/traces -X POST`. Any HTTP status code means the host can reach Maple. A timeout or DNS error means a firewall or proxy is blocking outbound HTTPS.
- **Nothing exported.** `tracing.mjs` must load before the modules it instruments, so use `--import` rather than importing it from your app. A process that exits right away can lose its last batch; call `sdk.shutdown()` before exit, as the `SIGTERM` handler above does. Set `OTEL_LOG_LEVEL=debug` to print export errors to the console.

## Next steps

- [Explore traces](/docs/explore/traces)
- [Track errors](/docs/errors/overview)
- [Create alert rules](/docs/alerting/alert-rules)
- [OpenTelemetry conventions](/docs/concepts/otel-conventions)

---

# Next.js instrumentation

> Instrumentation · https://maple.dev/docs/guides/instrumentation-nextjs

Instrument a Next.js application with @vercel/otel and send traces, logs, and metrics to Maple.

This guide sets up OpenTelemetry in a Next.js application with `@vercel/otel`, so the spans Next.js emits for requests, rendering, route handlers and middleware reach Maple, along with your logs and metrics. It works with the App Router and the Pages Router.

To have a coding agent do this setup, use the [maple-onboard](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-onboard) skill, and [maple-audit](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-audit) to check an existing setup.

## Prerequisites

- Next.js 13.4 or later. On 13.4 and 14, the instrumentation hook needs a config flag (see below). From Next.js 15 it is on by default.
- An ingest key from **Settings → Ingestion** in Maple. Use the private key (`maple_sk_…`): `instrumentation.ts` runs on the server.

## Install

```bash
npm install @vercel/otel \
  @opentelemetry/api \
  @opentelemetry/api-logs \
  @opentelemetry/instrumentation \
  @opentelemetry/resources \
  @opentelemetry/sdk-trace-base \
  @opentelemetry/sdk-logs \
  @opentelemetry/sdk-metrics \
  @opentelemetry/exporter-logs-otlp-proto \
  @opentelemetry/exporter-metrics-otlp-proto
```

`@vercel/otel` declares the other `@opentelemetry/*` packages as peer dependencies, so they are installed alongside it. It includes its own OTLP trace exporter. The log and metric exporters come from the standard OpenTelemetry packages.

## Configure

Create `instrumentation.ts` in the project root, next to `next.config.ts`. If your project keeps its code in `src/`, put the file at `src/instrumentation.ts` instead. Next.js calls `register()` once when each server runtime starts.

```typescript
// instrumentation.ts
import { registerOTel, OTLPHttpProtoTraceExporter } from "@vercel/otel"
import { BatchLogRecordProcessor } from "@opentelemetry/sdk-logs"
import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics"
import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-proto"
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-proto"

const MAPLE_ENDPOINT = "https://ingest.maple.dev" // EU: https://ingest.eu.maple.dev
const MAPLE_KEY = "YOUR_INGEST_KEY"

const headers = { authorization: `Bearer ${MAPLE_KEY}` }

export function register() {
	registerOTel({
		serviceName: "my-next-app",
		attributes: {
			"deployment.environment.name": process.env.VERCEL_ENV ?? "development",
			"vcs.repository.url.full": "https://github.com/acme/my-next-app",
			"vcs.ref.head.revision": process.env.VERCEL_GIT_COMMIT_SHA,
		},
		traceExporter: new OTLPHttpProtoTraceExporter({
			url: `${MAPLE_ENDPOINT}/v1/traces`,
			headers,
		}),
		logRecordProcessors: [
			new BatchLogRecordProcessor(new OTLPLogExporter({ url: `${MAPLE_ENDPOINT}/v1/logs`, headers })),
		],
		metricReaders: [
			new PeriodicExportingMetricReader({
				exporter: new OTLPMetricExporter({ url: `${MAPLE_ENDPOINT}/v1/metrics`, headers }),
			}),
		],
	})
}
```

`traceExporter` takes an exporter instance. `OTLPHttpProtoTraceExporter` is exported by `@vercel/otel` and sends OTLP over HTTP with protobuf encoding. It also runs in the Edge runtime.

The example puts the endpoint and key in source. An ingest key can only write telemetry to your organization. It cannot read data or call the Maple API. Keeping it in source means the SDK always starts with a complete configuration, so a deploy that is missing an environment variable cannot silently turn telemetry off. To keep the key out of source, use [environment variables](#environment-variables) instead.

### Next.js 13.4 and 14

Enable the instrumentation hook in `next.config.ts`. Skip this on Next.js 15 and later.

```typescript
// next.config.ts
export default {
	experimental: { instrumentationHook: true },
}
```

## Environment variables

When you leave out `traceExporter`, `@vercel/otel` configures an OTLP trace exporter from the standard environment variables. Set them in your hosting provider's project settings:

```bash
OTEL_SERVICE_NAME="my-next-app"
OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production,vcs.repository.url.full=https://github.com/acme/my-next-app"
```

`instrumentation.ts` then only needs `registerOTel({ serviceName: "my-next-app" })`. The log and metric exporters from the Configure step read `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_EXPORTER_OTLP_HEADERS` too, so you can construct them without a `url` or `headers`.

## Auto-instrumentation

Next.js emits spans for its own work, and `@vercel/otel` exports them:

- **Requests:** a root server span for every request to a page or route handler, named after the method and route.
- **Rendering:** spans for App Router rendering, Pages Router `getServerSideProps`, and route handler execution.
- **`fetch`:** outgoing `fetch()` calls from server code get client spans, and the trace context is propagated to the service you call.

Set `NEXT_OTEL_VERBOSE=1` to get more spans from Next.js internals. Next.js does not produce metrics of its own, so the metric reader exports the metrics you record with the `@opentelemetry/api` meter.

## Custom spans

Wrap business logic in custom spans to make it visible in the trace:

```typescript
import { trace, SpanStatusCode } from "@opentelemetry/api"

const tracer = trace.getTracer("my-next-app")

export async function processOrder(orderId: string) {
	return tracer.startActiveSpan("process-order", async (span) => {
		try {
			span.setAttribute("order.id", orderId)
			span.setAttribute("payment.method", "card")
			return await chargePayment(orderId)
		} catch (error) {
			span.recordException(error as Error)
			span.setStatus({ code: SpanStatusCode.ERROR, message: (error as Error).message })
			throw error
		} finally {
			span.end()
		}
	})
}
```

Service map edges come from instrumented client spans that propagate `traceparent` to an instrumented callee, not from attributes such as `peer.service`. See [Service map](/docs/explore/service-map).

## Attributes Maple uses

Most attributes are optional. These unlock specific features.

### Commits and GitHub

Set `vcs.ref.head.revision` (the full 40-character commit SHA) and `vcs.repository.url.full` on the resource. They drive deploy markers, the Releases page and, with the [GitHub integration](/docs/integrations/github), commit hover cards on traces. The resource in [Configure](#configure) already sets both. Read the SHA from your CI or host at build time, such as `GITHUB_SHA` or `VERCEL_GIT_COMMIT_SHA`, not from `git` at runtime:

```bash
export OTEL_RESOURCE_ATTRIBUTES="vcs.repository.url.full=https://github.com/acme/api,vcs.ref.head.revision=$GITHUB_SHA"
```

### Databases

Database instrumentation sets these attributes for you. For a driver without instrumentation, set them on a `Client` span:

```typescript
import { SpanKind } from "@opentelemetry/api"

const user = await tracer.startActiveSpan(
	"SELECT users",
	{
		kind: SpanKind.CLIENT,
		attributes: {
			"db.system.name": "postgresql",
			"db.namespace": "app",
			"db.query.text": "SELECT * FROM users WHERE id = $1",
		},
	},
	async (span) => {
		try {
			return await client.query("SELECT * FROM users WHERE id = $1", [id])
		} finally {
			span.end()
		}
	},
)
```

`db.system.name` and `db.namespace` draw the database node on the [service map](/docs/explore/service-map). `db.query.text` groups statements into query shapes, so record it with placeholders, not bound values. See [Database queries](/docs/concepts/otel-conventions#database-queries) for the full list.

## Log correlation

Log records emitted through the OpenTelemetry log SDK during an active server span carry its trace and span IDs, so Maple links each log line to its trace. To send logs from pino or winston, bridge the logger to the OpenTelemetry log SDK that `registerOTel` configured above.

## Edge runtime

`instrumentation.ts` runs in the Edge runtime as well as in Node.js. The trace exporter works in both. The metric reader exports on a 60-second timer, so metrics are only reliable from the Node.js runtime.

## Verify

1. Run `next dev` or deploy, then open a page or call a route handler a few times.
2. In Maple, open **Explore → Traces**. Spans are sent in batches every 5 seconds by default.
3. Each request should show up as one trace with a single root server span named after the method and route (for example `GET /dashboard`), with rendering and `fetch` spans nested under it.

Your service also appears on the **Services** page once its first spans arrive.

## Troubleshooting

- **`401` responses.** The key is wrong, was copied from the other region, or the header is malformed. The header must be `Authorization: Bearer YOUR_INGEST_KEY`. In `OTEL_EXPORTER_OTLP_HEADERS` it is written `Authorization=Bearer YOUR_INGEST_KEY`. See [Ingest API status codes](/docs/reference/ingest#status-codes).
- **Wrong protocol or path.** Maple accepts OTLP over HTTP. Endpoints set in code need the full signal path (`/v1/traces`); `OTEL_EXPORTER_OTLP_ENDPOINT` takes only the base URL.
- **Network.** From the machine running the app, run `curl -i https://ingest.maple.dev/v1/traces -X POST`. Any HTTP status code means the host can reach Maple. A timeout or DNS error means a firewall or proxy is blocking outbound HTTPS.
- **Nothing exported.** Check that `instrumentation.ts` is in the project root, or in `src/` for a `src` layout, and not inside `app/` or `pages/`. On Next.js 13.4 and 14, check that `experimental.instrumentationHook` is set.

## Next steps

- [Explore traces](/docs/explore/traces)
- [Track errors](/docs/errors/overview)
- [Create alert rules](/docs/alerting/alert-rules)
- [OpenTelemetry conventions](/docs/concepts/otel-conventions)

---

# Python instrumentation

> Instrumentation · https://maple.dev/docs/guides/instrumentation-python

Instrument a Python application with OpenTelemetry and send traces, logs, and metrics to Maple.

This guide sets up the OpenTelemetry Python SDK so your application sends traces, logs and metrics to Maple, with instrumentation for FastAPI, Django, Flask and common client libraries.

To have a coding agent do this setup, use the [maple-onboard](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-onboard) skill, and [maple-audit](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-audit) to check an existing setup.

## Prerequisites

- Python 3.9+
- An ingest key from **Settings → Ingestion** in Maple. Use the private key (`maple_sk_…`) for server applications.

## Install

```bash
pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
```

## Configure

Create a `telemetry.py` module that sets up traces, metrics and logs:

```python
# telemetry.py
import logging
import os

from opentelemetry import metrics, trace
from opentelemetry._logs import set_logger_provider
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

MAPLE_ENDPOINT = "https://ingest.maple.dev"  # EU: https://ingest.eu.maple.dev
MAPLE_KEY = "YOUR_INGEST_KEY"
HEADERS = {"authorization": f"Bearer {MAPLE_KEY}"}

resource = Resource.create({
    "service.name": "my-python-app",
    "deployment.environment.name": os.getenv("DEPLOYMENT_ENV", "development"),
    "vcs.repository.url.full": "https://github.com/acme/my-python-app",
    "vcs.ref.head.revision": os.getenv("GITHUB_SHA") or os.getenv("GIT_COMMIT", ""),
})

# Traces
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint=f"{MAPLE_ENDPOINT}/v1/traces", headers=HEADERS))
)
trace.set_tracer_provider(tracer_provider)

# Metrics
metric_reader = PeriodicExportingMetricReader(
    OTLPMetricExporter(endpoint=f"{MAPLE_ENDPOINT}/v1/metrics", headers=HEADERS)
)
metrics.set_meter_provider(MeterProvider(resource=resource, metric_readers=[metric_reader]))

# Logs: records from the standard logging module go to Maple
logger_provider = LoggerProvider(resource=resource)
logger_provider.add_log_record_processor(
    BatchLogRecordProcessor(OTLPLogExporter(endpoint=f"{MAPLE_ENDPOINT}/v1/logs", headers=HEADERS))
)
set_logger_provider(logger_provider)
logging.getLogger().addHandler(LoggingHandler(logger_provider=logger_provider))
```

The example puts the endpoint and key in source. An ingest key can only write telemetry to your organization. It cannot read data or call the Maple API. Keeping it in source means the SDK always starts with a complete configuration, so a deploy that is missing an environment variable cannot silently turn telemetry off. To keep the key out of source, use [environment variables](#environment-variables) instead.

Import the module first thing at startup, before the modules you want traced:

```python
import telemetry  # noqa: F401  (sets up OpenTelemetry)
from myapp import create_app

app = create_app()
```

## Environment variables

Without any code, the `opentelemetry-instrument` wrapper configures the SDK from the standard environment variables and instruments every supported library it finds. Install the distro and exporter, then install the instrumentation packages that match your dependencies:

```bash
pip install opentelemetry-distro opentelemetry-exporter-otlp-proto-http
opentelemetry-bootstrap -a install
```

Set the variables and start your app through the wrapper:

```bash
export OTEL_SERVICE_NAME="my-python-app"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
export OTEL_TRACES_EXPORTER="otlp"
export OTEL_METRICS_EXPORTER="otlp"
export OTEL_LOGS_EXPORTER="otlp"
export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED="true"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production,vcs.repository.url.full=https://github.com/acme/my-python-app"

opentelemetry-instrument python app.py
```

`OTEL_EXPORTER_OTLP_PROTOCOL` matters here: the distro defaults to gRPC, which needs a different exporter package. `OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED` attaches the log exporter to the standard `logging` module. Use either this path or `telemetry.py`, not both.

## Auto-instrumentation

With `telemetry.py`, install the instrumentation package for each library you use and enable it in code. Each incoming request becomes a server span, and outgoing HTTP calls and database queries become child spans.

### FastAPI

```bash
pip install opentelemetry-instrumentation-fastapi
```

```python
import telemetry  # noqa: F401
from fastapi import FastAPI
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

app = FastAPI()
FastAPIInstrumentor.instrument_app(app)
```

### Django

```bash
pip install opentelemetry-instrumentation-django
```

```python
# manage.py and wsgi.py / asgi.py, before Django loads
import telemetry  # noqa: F401
from opentelemetry.instrumentation.django import DjangoInstrumentor

DjangoInstrumentor().instrument()
```

Database queries need the instrumentation for your driver, for example `opentelemetry-instrumentation-psycopg2` or `opentelemetry-instrumentation-sqlite3`.

### Flask and HTTP clients

```bash
pip install opentelemetry-instrumentation-flask \
  opentelemetry-instrumentation-requests \
  opentelemetry-instrumentation-httpx
```

```python
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor

FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()
HTTPXClientInstrumentor().instrument()
```

The HTTP client instrumentations propagate trace context, so the services you call join the same trace.

## Custom spans

```python
from opentelemetry import trace
from opentelemetry.trace import StatusCode

tracer = trace.get_tracer("my-app")

def process_order(order_id: str):
    with tracer.start_as_current_span("process-order") as span:
        span.set_attribute("order.id", order_id)
        span.set_attribute("payment.method", "card")

        try:
            return charge_payment(order_id)
        except Exception as e:
            span.record_exception(e)
            span.set_status(StatusCode.ERROR, str(e))
            raise
```

Service map edges come from instrumented client spans that propagate `traceparent` to an instrumented callee, not from attributes such as `peer.service`. See [Service map](/docs/explore/service-map).

## Attributes Maple uses

Most attributes are optional. These unlock specific features.

### Commits and GitHub

Set `vcs.ref.head.revision` (the full 40-character commit SHA) and `vcs.repository.url.full` on the resource. They drive deploy markers, the Releases page and, with the [GitHub integration](/docs/integrations/github), commit hover cards on traces. The resource in [Configure](#configure) already sets both. Read the SHA from your CI or host at build time, such as `GITHUB_SHA` or `VERCEL_GIT_COMMIT_SHA`, not from `git` at runtime:

```bash
export OTEL_RESOURCE_ATTRIBUTES="vcs.repository.url.full=https://github.com/acme/api,vcs.ref.head.revision=$GITHUB_SHA"
```

### Databases

Database instrumentation sets these attributes for you. For a driver without instrumentation, set them on a `Client` span:

```python
from opentelemetry.trace import SpanKind

with tracer.start_as_current_span(
    "SELECT users",
    kind=SpanKind.CLIENT,
    attributes={
        "db.system.name": "postgresql",
        "db.namespace": "app",
        "db.query.text": "SELECT * FROM users WHERE id = %s",
    },
):
    cursor.execute("SELECT * FROM users WHERE id = %s", (user_id,))
```

`db.system.name` and `db.namespace` draw the database node on the [service map](/docs/explore/service-map). `db.query.text` groups statements into query shapes, so record it with placeholders, not bound values. See [Database queries](/docs/concepts/otel-conventions#database-queries) for the full list.

## Log correlation

`telemetry.py` attaches a `LoggingHandler` to the root logger. Records logged during an active span carry its trace and span IDs, so Maple links each log line to its trace:

```python
import logging

logging.getLogger(__name__).warning("Payment retry for order %s", order_id)
```

The root logger's level still applies. Python defaults it to `WARNING`, so call `logging.basicConfig(level=logging.INFO)` or set the level yourself to send `INFO` records.

## Verify

1. Start your application and send it a few requests.
2. In Maple, open **Explore → Traces**. The SDK sends spans in batches every 5 seconds by default, and metrics every 60 seconds.
3. Each request should show up as one trace with a single root server span, named after the method and route (for example `GET /api/orders`), and child spans for the queries and outgoing calls it made.

Your service also appears on the **Services** page once its first spans arrive.

## Troubleshooting

- **`401` responses.** The key is wrong, was copied from the other region, or the header is malformed. The header must be `Authorization: Bearer YOUR_INGEST_KEY`. In `OTEL_EXPORTER_OTLP_HEADERS` it is written `Authorization=Bearer YOUR_INGEST_KEY`. See [Ingest API status codes](/docs/reference/ingest#status-codes).
- **Wrong protocol or path.** Maple accepts OTLP over HTTP. Use `opentelemetry-exporter-otlp-proto-http`, and set `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` when using `opentelemetry-instrument`. Endpoints set in code need the full signal path (`/v1/traces`); `OTEL_EXPORTER_OTLP_ENDPOINT` takes only the base URL.
- **Network.** From the machine running the app, run `curl -i https://ingest.maple.dev/v1/traces -X POST`. Any HTTP status code means the host can reach Maple. A timeout or DNS error means a firewall or proxy is blocking outbound HTTPS.
- **Nothing exported.** Import `telemetry` before the framework. With gunicorn or uWSGI worker processes, set up the SDK in each worker (for example in gunicorn's `post_fork` hook), because batch processor threads do not survive a fork. A short script can exit before its batch is sent; call `tracer_provider.shutdown()`, `logger_provider.shutdown()` and the meter provider's `shutdown()` before exit.

## Next steps

- [Explore traces](/docs/explore/traces)
- [Track errors](/docs/errors/overview)
- [Create alert rules](/docs/alerting/alert-rules)
- [OpenTelemetry conventions](/docs/concepts/otel-conventions)

---

# Go instrumentation

> Instrumentation · https://maple.dev/docs/guides/instrumentation-go

Instrument a Go application with OpenTelemetry and send traces, logs, and metrics to Maple.

This guide sets up the OpenTelemetry Go SDK so your application sends traces, logs and metrics to Maple, and adds instrumentation for `net/http`, gRPC and `database/sql`.

To have a coding agent do this setup, use the [maple-onboard](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-onboard) skill, and [maple-audit](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-audit) to check an existing setup.

## Prerequisites

- Go 1.23+
- An ingest key from **Settings → Ingestion** in Maple. Use the private key (`maple_sk_…`) for server applications.

## Install

```bash
go get go.opentelemetry.io/otel \
  go.opentelemetry.io/otel/sdk \
  go.opentelemetry.io/otel/sdk/metric \
  go.opentelemetry.io/otel/sdk/log \
  go.opentelemetry.io/otel/log \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp \
  go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp \
  go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp
```

## Configure

Set up the three providers at startup and shut them down on exit:

```go
package main

import (
	"context"
	"errors"
	"log"
	"os"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp"
	"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/log/global"
	"go.opentelemetry.io/otel/propagation"
	sdklog "go.opentelemetry.io/otel/sdk/log"
	sdkmetric "go.opentelemetry.io/otel/sdk/metric"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

const mapleEndpoint = "https://ingest.maple.dev" // EU: https://ingest.eu.maple.dev
const mapleKey = "YOUR_INGEST_KEY"

func initTelemetry(ctx context.Context) (func(context.Context) error, error) {
	headers := map[string]string{"Authorization": "Bearer " + mapleKey}

	res, err := resource.New(ctx, resource.WithAttributes(
		attribute.String("service.name", "my-go-app"),
		attribute.String("deployment.environment.name", os.Getenv("DEPLOYMENT_ENV")),
		attribute.String("vcs.repository.url.full", "https://github.com/acme/my-go-app"),
	))
	if err != nil {
		return nil, err
	}

	traceExporter, err := otlptracehttp.New(ctx,
		otlptracehttp.WithEndpointURL(mapleEndpoint+"/v1/traces"),
		otlptracehttp.WithHeaders(headers),
	)
	if err != nil {
		return nil, err
	}
	tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(traceExporter), sdktrace.WithResource(res))
	otel.SetTracerProvider(tp)
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
		propagation.TraceContext{}, propagation.Baggage{},
	))

	metricExporter, err := otlpmetrichttp.New(ctx,
		otlpmetrichttp.WithEndpointURL(mapleEndpoint+"/v1/metrics"),
		otlpmetrichttp.WithHeaders(headers),
	)
	if err != nil {
		return nil, err
	}
	mp := sdkmetric.NewMeterProvider(
		sdkmetric.WithReader(sdkmetric.NewPeriodicReader(metricExporter)),
		sdkmetric.WithResource(res),
	)
	otel.SetMeterProvider(mp)

	logExporter, err := otlploghttp.New(ctx,
		otlploghttp.WithEndpointURL(mapleEndpoint+"/v1/logs"),
		otlploghttp.WithHeaders(headers),
	)
	if err != nil {
		return nil, err
	}
	lp := sdklog.NewLoggerProvider(
		sdklog.WithProcessor(sdklog.NewBatchProcessor(logExporter)),
		sdklog.WithResource(res),
	)
	global.SetLoggerProvider(lp)

	shutdown := func(ctx context.Context) error {
		return errors.Join(tp.Shutdown(ctx), mp.Shutdown(ctx), lp.Shutdown(ctx))
	}
	return shutdown, nil
}

func main() {
	ctx := context.Background()

	// A setup error only disables telemetry; it never stops the app.
	if shutdown, err := initTelemetry(ctx); err != nil {
		log.Printf("telemetry disabled: %v", err)
	} else {
		defer shutdown(ctx)
	}

	// Your application code here
}
```

The example puts the endpoint and key in source. An ingest key can only write telemetry to your organization. It cannot read data or call the Maple API. Keeping it in source means the SDK always starts with a complete configuration, so a deploy that is missing an environment variable cannot silently turn telemetry off. To keep the key out of source, use [environment variables](#environment-variables) instead.

## Environment variables

The Go exporters read the standard OpenTelemetry variables when you create them without endpoint or header options:

```bash
export OTEL_SERVICE_NAME="my-go-app"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production,vcs.repository.url.full=https://github.com/acme/my-go-app"
```

In `initTelemetry`, call `otlptracehttp.New(ctx)`, `otlpmetrichttp.New(ctx)` and `otlploghttp.New(ctx)` with no options, and build the resource with `resource.New(ctx, resource.WithFromEnv())` so it reads `OTEL_SERVICE_NAME` and `OTEL_RESOURCE_ATTRIBUTES`. The `*http` exporters always use OTLP over HTTP; setting the protocol variable keeps the configuration explicit for any other tool that reads it.

## Auto-instrumentation

Go has no automatic library discovery. You add an instrumentation package for each library you use.

### HTTP server

```bash
go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
```

```go
import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

mux := http.NewServeMux()
mux.HandleFunc("GET /api/orders", handleOrders)

// Every request becomes a server span
handler := otelhttp.NewHandler(mux, "server")
http.ListenAndServe(":8080", handler)
```

### HTTP client

Wrap the transport so outgoing requests get client spans and carry the trace context:

```go
client := &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}
req, _ := http.NewRequestWithContext(ctx, "GET", "https://api.example.com/data", nil)
resp, err := client.Do(req)
```

### gRPC

```bash
go get go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc
```

```go
import "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"

// Server
server := grpc.NewServer(grpc.StatsHandler(otelgrpc.NewServerHandler()))

// Client
conn, err := grpc.NewClient(addr, grpc.WithStatsHandler(otelgrpc.NewClientHandler()))
```

### Database

```bash
go get github.com/XSAM/otelsql
```

```go
import "github.com/XSAM/otelsql"

db, err := otelsql.Open("postgres", dsn)
```

Pass the request `ctx` to `db.QueryContext` and friends so query spans nest under the request span.

## Custom spans

```go
import (
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/codes"
)

var tracer = otel.Tracer("my-app")

func processOrder(ctx context.Context, orderID string) error {
	ctx, span := tracer.Start(ctx, "process-order")
	defer span.End()

	span.SetAttributes(
		attribute.String("order.id", orderID),
		attribute.String("payment.method", "card"),
	)

	if err := chargePayment(ctx, orderID); err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, err.Error())
		return err
	}
	return nil
}
```

Service map edges come from instrumented client spans that propagate `traceparent` to an instrumented callee, not from attributes such as `peer.service`. See [Service map](/docs/explore/service-map). Always pass `ctx` down the call chain so child spans link to their parent.

## Attributes Maple uses

Most attributes are optional. These unlock specific features.

### Commits and GitHub

Set `vcs.ref.head.revision` (the full 40-character commit SHA) and `vcs.repository.url.full` on the resource. They drive deploy markers, the Releases page and, with the [GitHub integration](/docs/integrations/github), commit hover cards on traces. Read the SHA from your CI or host at build time, such as `GITHUB_SHA` or `VERCEL_GIT_COMMIT_SHA`, not from `git` at runtime:

```bash
export OTEL_RESOURCE_ATTRIBUTES="vcs.repository.url.full=https://github.com/acme/api,vcs.ref.head.revision=$GITHUB_SHA"
```

### Databases

`otelsql` and `otelpgx` wrap `database/sql` and `pgx` and set these attributes for you. For a driver without instrumentation, set them on a `Client` span:

```go
ctx, span := tracer.Start(ctx, "SELECT users",
	trace.WithSpanKind(trace.SpanKindClient),
	trace.WithAttributes(
		attribute.String("db.system.name", "postgresql"),
		attribute.String("db.namespace", "app"),
		attribute.String("db.query.text", "SELECT * FROM users WHERE id = $1"),
	),
)
defer span.End()
```

`db.system.name` and `db.namespace` draw the database node on the [service map](/docs/explore/service-map). `db.query.text` groups statements into query shapes, so record it with placeholders, not bound values. See [Database queries](/docs/concepts/otel-conventions#database-queries) for the full list.

## Log correlation

The `otelslog` bridge sends `log/slog` records to the logger provider configured above. Records logged with a context that holds an active span carry its trace and span IDs:

```bash
go get go.opentelemetry.io/contrib/bridges/otelslog
```

```go
import "go.opentelemetry.io/contrib/bridges/otelslog"

logger := otelslog.NewLogger("my-app")
logger.InfoContext(ctx, "Order processed", "order_id", orderID)
```

Use the `...Context` methods. Without the context, the bridge cannot find the active span.

## Verify

1. Start your application and send it a few requests.
2. In Maple, open **Explore → Traces**. The SDK sends spans in batches every 5 seconds by default, and metrics every 60 seconds.
3. Each request should show up as one trace with a single root server span, and child spans for the queries and outgoing calls it made. `otelhttp` names the server span with the operation name you pass to `NewHandler`, and records the route in `http.route`.

Your service also appears on the **Services** page once its first spans arrive.

## Troubleshooting

- **`401` responses.** The key is wrong, was copied from the other region, or the header is malformed. The header must be `Authorization: Bearer YOUR_INGEST_KEY`. In `OTEL_EXPORTER_OTLP_HEADERS` it is written `Authorization=Bearer YOUR_INGEST_KEY`. See [Ingest API status codes](/docs/reference/ingest#status-codes).
- **Wrong protocol or path.** Maple accepts OTLP over HTTP. Use the `otlp*http` exporters, not `otlp*grpc`. `WithEndpointURL` takes the full URL including `/v1/traces`; `OTEL_EXPORTER_OTLP_ENDPOINT` takes only the base URL.
- **Network.** From the machine running the app, run `curl -i https://ingest.maple.dev/v1/traces -X POST`. Any HTTP status code means the host can reach Maple. A timeout or DNS error means a firewall or proxy is blocking outbound HTTPS.
- **Nothing exported.** Call the shutdown function before `main` returns so the batchers flush. Code that calls `os.Exit` or `log.Fatal` skips deferred calls. Register an error handler with `otel.SetErrorHandler` to print export failures.

## Next steps

- [Explore traces](/docs/explore/traces)
- [Track errors](/docs/errors/overview)
- [Create alert rules](/docs/alerting/alert-rules)
- [OpenTelemetry conventions](/docs/concepts/otel-conventions)

---

# Rust instrumentation

> Instrumentation · https://maple.dev/docs/guides/instrumentation-rust

Instrument a Rust application with OpenTelemetry and send traces, logs, and metrics to Maple.

This guide sets up the OpenTelemetry Rust SDK so your application sends traces, logs and metrics to Maple. Spans and log events come from the `tracing` crate, bridged to OpenTelemetry.

To have a coding agent do this setup, use the [maple-onboard](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-onboard) skill, and [maple-audit](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-audit) to check an existing setup.

## Prerequisites

- Rust 1.75+
- An ingest key from **Settings → Ingestion** in Maple. Use the private key (`maple_sk_…`) for server applications.

## Install

Add these dependencies to `Cargo.toml`. The `opentelemetry*` crates must share one minor version, and `tracing-opentelemetry` must be the release built for it (0.32 pairs with 0.31). This guide pins 0.31, and the `reqwest-tracing` feature further down names the same version.

```toml
[dependencies]
opentelemetry = "0.31"
opentelemetry_sdk = "0.31"
opentelemetry-otlp = "0.31"
opentelemetry-appender-tracing = "0.31"
tracing-opentelemetry = "0.32"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
tokio = { version = "1", features = ["full"] }
```

The default features of `opentelemetry-otlp` export OTLP over HTTP with protobuf encoding, using a blocking HTTP client on the SDK's own background threads.

## Configure

```rust
// src/telemetry.rs
use opentelemetry::{global, trace::TracerProvider as _, KeyValue};
use opentelemetry_appender_tracing::layer::OpenTelemetryTracingBridge;
use opentelemetry_otlp::{LogExporter, MetricExporter, SpanExporter, WithExportConfig, WithHttpConfig};
use opentelemetry_sdk::{
    logs::SdkLoggerProvider, metrics::SdkMeterProvider, propagation::TraceContextPropagator,
    trace::SdkTracerProvider, Resource,
};
use std::collections::HashMap;
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt, EnvFilter, Layer};

const MAPLE_ENDPOINT: &str = "https://ingest.maple.dev"; // EU: https://ingest.eu.maple.dev
const MAPLE_KEY: &str = "YOUR_INGEST_KEY";

pub struct Telemetry {
    tracer_provider: SdkTracerProvider,
    meter_provider: SdkMeterProvider,
    logger_provider: SdkLoggerProvider,
}

impl Telemetry {
    pub fn shutdown(&self) {
        let _ = self.tracer_provider.shutdown();
        let _ = self.meter_provider.shutdown();
        let _ = self.logger_provider.shutdown();
    }
}

pub fn init() -> Result<Telemetry, Box<dyn std::error::Error>> {
    let headers = HashMap::from([("Authorization".to_string(), format!("Bearer {MAPLE_KEY}"))]);

    let resource = Resource::builder()
        .with_service_name("my-rust-app")
        .with_attribute(KeyValue::new(
            "deployment.environment.name",
            std::env::var("DEPLOYMENT_ENV").unwrap_or_else(|_| "development".into()),
        ))
        .with_attribute(KeyValue::new(
            "vcs.repository.url.full",
            "https://github.com/acme/my-rust-app",
        ))
        .build();

    let tracer_provider = SdkTracerProvider::builder()
        .with_batch_exporter(
            SpanExporter::builder()
                .with_http()
                .with_endpoint(format!("{MAPLE_ENDPOINT}/v1/traces"))
                .with_headers(headers.clone())
                .build()?,
        )
        .with_resource(resource.clone())
        .build();
    global::set_tracer_provider(tracer_provider.clone());
    // Lets HTTP integrations read and write `traceparent` headers.
    global::set_text_map_propagator(TraceContextPropagator::new());

    let meter_provider = SdkMeterProvider::builder()
        .with_periodic_exporter(
            MetricExporter::builder()
                .with_http()
                .with_endpoint(format!("{MAPLE_ENDPOINT}/v1/metrics"))
                .with_headers(headers.clone())
                .build()?,
        )
        .with_resource(resource.clone())
        .build();
    global::set_meter_provider(meter_provider.clone());

    let logger_provider = SdkLoggerProvider::builder()
        .with_batch_exporter(
            LogExporter::builder()
                .with_http()
                .with_endpoint(format!("{MAPLE_ENDPOINT}/v1/logs"))
                .with_headers(headers)
                .build()?,
        )
        .with_resource(resource)
        .build();

    // Keep the exporter's own HTTP client logs out of the OpenTelemetry log pipeline.
    let otel_log_filter = EnvFilter::new("info")
        .add_directive("hyper=off".parse()?)
        .add_directive("h2=off".parse()?)
        .add_directive("reqwest=off".parse()?);

    tracing_subscriber::registry()
        .with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")))
        .with(tracing_subscriber::fmt::layer())
        .with(tracing_opentelemetry::layer().with_tracer(tracer_provider.tracer("my-rust-app")))
        .with(OpenTelemetryTracingBridge::new(&logger_provider).with_filter(otel_log_filter))
        .init();

    Ok(Telemetry { tracer_provider, meter_provider, logger_provider })
}
```

Call `init` from `main`, and shut down before exit so the last batches are sent:

```rust
mod telemetry;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let telemetry = telemetry::init()?;

    // Your application code here

    telemetry.shutdown();
    Ok(())
}
```

The example puts the endpoint and key in source. An ingest key can only write telemetry to your organization. It cannot read data or call the Maple API. Keeping it in source means the SDK always starts with a complete configuration, so a deploy that is missing an environment variable cannot silently turn telemetry off. To keep the key out of source, use [environment variables](#environment-variables) instead.

## Environment variables

When you build an exporter without `with_endpoint` and `with_headers`, it reads the standard OpenTelemetry variables, and `Resource::builder()` reads `OTEL_SERVICE_NAME` and `OTEL_RESOURCE_ATTRIBUTES`:

```bash
export OTEL_SERVICE_NAME="my-rust-app"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production,vcs.repository.url.full=https://github.com/acme/my-rust-app"
```

With `OTEL_EXPORTER_OTLP_ENDPOINT` set to the base URL, each exporter appends its own signal path.

## Auto-instrumentation

Rust has no automatic library instrumentation. You add a `tracing` integration for each crate you use, and the bridge above turns its spans into OpenTelemetry spans.

### Axum and Tower

```toml
[dependencies]
axum = "0.8"
tower-http = { version = "0.6", features = ["trace"] }
```

```rust
use axum::{routing::get, Router};
use tower_http::trace::TraceLayer;

let app = Router::new()
    .route("/api/orders", get(handle_orders))
    .layer(TraceLayer::new_for_http());
```

`TraceLayer` opens a `tracing` span for every request. It does not read incoming `traceparent` headers or set OpenTelemetry HTTP attributes. For that, use a crate built for it, such as `axum-tracing-opentelemetry`.

### reqwest

```toml
[dependencies]
reqwest = "0.13"
reqwest-middleware = "0.5"
reqwest-tracing = { version = "0.7", features = ["opentelemetry_0_31"] }
```

```rust
use reqwest_middleware::ClientBuilder;
use reqwest_tracing::TracingMiddleware;

let client = ClientBuilder::new(reqwest::Client::new())
    .with(TracingMiddleware::default())
    .build();
```

`reqwest-tracing` injects the trace context into outgoing requests. Its `opentelemetry_0_31` feature must match your `opentelemetry` version; change both together when you upgrade.

## Custom spans

The `#[instrument]` attribute opens a span for each call:

```rust
use tracing::instrument;

#[instrument(skip(payment_client), fields(order.id = %order_id, payment.method = "card"))]
async fn process_order(payment_client: &PaymentClient, order_id: String) -> Result<(), PaymentError> {
    payment_client.charge(&order_id).await?;
    Ok(())
}
```

Service map edges come from instrumented client spans that propagate `traceparent` to an instrumented callee, not from attributes such as `peer.service`. See [Service map](/docs/explore/service-map).

For a span around a block:

```rust
use tracing::{info_span, Instrument};

async fn process_order(order_id: String) {
    let span = info_span!("process-order", order.id = %order_id);
    async move {
        charge_payment(&order_id).await;
    }
    .instrument(span)
    .await;
}
```

## Attributes Maple uses

Most attributes are optional. These unlock specific features.

### Commits and GitHub

Set `vcs.ref.head.revision` (the full 40-character commit SHA) and `vcs.repository.url.full` on the resource. They drive deploy markers, the Releases page and, with the [GitHub integration](/docs/integrations/github), commit hover cards on traces. Read the SHA from your CI or host at build time, such as `GITHUB_SHA` or `VERCEL_GIT_COMMIT_SHA`, not from `git` at runtime:

```bash
export OTEL_RESOURCE_ATTRIBUTES="vcs.repository.url.full=https://github.com/acme/api,vcs.ref.head.revision=$GITHUB_SHA"
```

### Databases

Database instrumentation sets these attributes for you. For a driver without instrumentation, set them on a `Client` span. `tracing-opentelemetry` reads the `otel.kind` field as the span kind:

```rust
let span = info_span!(
    "SELECT users",
    otel.kind = "client",
    db.system.name = "postgresql",
    db.namespace = "app",
    db.query.text = "SELECT * FROM users WHERE id = $1",
);
let user = sqlx::query_as::<_, User>("SELECT * FROM users WHERE id = $1")
    .bind(id)
    .fetch_one(&pool)
    .instrument(span)
    .await?;
```

`db.system.name` and `db.namespace` draw the database node on the [service map](/docs/explore/service-map). `db.query.text` groups statements into query shapes, so record it with placeholders, not bound values. See [Database queries](/docs/concepts/otel-conventions#database-queries) for the full list.

## Log correlation

`tracing` events go to the `OpenTelemetryTracingBridge` layer as OpenTelemetry log records. Events emitted inside a span carry its trace and span IDs:

```rust
use tracing::{error, info};

#[tracing::instrument]
async fn process_order(order_id: String) {
    info!(order.id = %order_id, "processing order");

    if let Err(e) = charge_payment(&order_id).await {
        error!(error = %e, "payment failed");
    }
}
```

## Verify

1. Start your application and send it a few requests.
2. In Maple, open **Explore → Traces**. The SDK sends spans in batches every 5 seconds by default, and metrics every 60 seconds.
3. Each request should show up as one trace with a single root span for the request, and child spans for the functions you instrumented.

Your service also appears on the **Services** page once its first spans arrive.

## Troubleshooting

- **`401` responses.** The key is wrong, was copied from the other region, or the header is malformed. The header must be `Authorization: Bearer YOUR_INGEST_KEY`. In `OTEL_EXPORTER_OTLP_HEADERS` it is written `Authorization=Bearer YOUR_INGEST_KEY`. See [Ingest API status codes](/docs/reference/ingest#status-codes).
- **Wrong protocol or path.** Maple accepts OTLP over HTTP. Build exporters with `.with_http()`, not `.with_tonic()`. `with_endpoint` takes the full URL including `/v1/traces`; `OTEL_EXPORTER_OTLP_ENDPOINT` takes only the base URL.
- **Network.** From the machine running the app, run `curl -i https://ingest.maple.dev/v1/traces -X POST`. Any HTTP status code means the host can reach Maple. A timeout or DNS error means a firewall or proxy is blocking outbound HTTPS.
- **Nothing exported.** Shut the providers down before the process exits so buffered data is sent. If the build fails with trait errors between `tracing-opentelemetry` and `opentelemetry`, the versions do not match.

## Next steps

- [Explore traces](/docs/explore/traces)
- [Track errors](/docs/errors/overview)
- [Create alert rules](/docs/alerting/alert-rules)
- [OpenTelemetry conventions](/docs/concepts/otel-conventions)

---

# Java instrumentation

> Instrumentation · https://maple.dev/docs/guides/instrumentation-java

Instrument a Java application with OpenTelemetry and send traces, logs, and metrics to Maple.

This guide attaches the OpenTelemetry Java agent to your application so it sends traces, logs and metrics to Maple. The agent instruments most popular libraries without code changes. A manual SDK setup is covered further down for runtimes where you cannot attach an agent.

To have a coding agent do this setup, use the [maple-onboard](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-onboard) skill, and [maple-audit](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-audit) to check an existing setup.

## Prerequisites

- Java 8+
- An ingest key from **Settings → Ingestion** in Maple. Use the private key (`maple_sk_…`) for server applications.

## Install

Download the agent jar from the [opentelemetry-java-instrumentation releases](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases):

```bash
curl -L -o opentelemetry-javaagent.jar \
  https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
```

## Configure

Start your application with the agent attached and Maple's endpoint configured:

```bash
java \
  -javaagent:opentelemetry-javaagent.jar \
  -Dotel.service.name=my-java-app \
  -Dotel.exporter.otlp.endpoint=https://ingest.maple.dev \
  -Dotel.exporter.otlp.protocol=http/protobuf \
  -Dotel.exporter.otlp.headers="Authorization=Bearer YOUR_INGEST_KEY" \
  -Dotel.resource.attributes="deployment.environment.name=production,vcs.repository.url.full=https://github.com/acme/my-java-app" \
  -jar app.jar
```

For an EU organization, use `https://ingest.eu.maple.dev`. The endpoint is the base URL: the agent appends `/v1/traces`, `/v1/logs` and `/v1/metrics` itself.

The agent (version 2.x) exports traces, metrics and logs over OTLP by default, so there is no separate metrics or logs setup. `http/protobuf` is already its default protocol. Setting it explicitly does no harm and documents what you expect.

## Environment variables

Every `-Dotel.*` property has an environment variable equivalent, which suits containers:

```bash
export JAVA_TOOL_OPTIONS="-javaagent:/opt/opentelemetry-javaagent.jar"
export OTEL_SERVICE_NAME="my-java-app"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production,vcs.repository.url.full=https://github.com/acme/my-java-app"
```

## Auto-instrumentation

The agent instruments Spring (MVC, WebFlux, Boot), servlet containers, JDBC, gRPC, Kafka, JMS, the AWS SDK and [many more libraries](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md). Each incoming request becomes a server span, and JDBC queries and outgoing HTTP calls become child spans. The agent also reports JVM runtime metrics such as memory, garbage collection and thread counts.

## Manual SDK setup

Use this when you cannot attach the agent, for example in GraalVM native images or restricted runtimes.

### Install dependencies

Gradle:

```kotlin
implementation(platform("io.opentelemetry:opentelemetry-bom:1.51.0"))
implementation("io.opentelemetry:opentelemetry-api")
implementation("io.opentelemetry:opentelemetry-sdk")
implementation("io.opentelemetry:opentelemetry-exporter-otlp")
```

Maven:

```xml
<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-bom</artifactId>
      <version>1.51.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
  </dependency>
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-sdk</artifactId>
  </dependency>
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
  </dependency>
</dependencies>
```

### Configure the SDK

```java
import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.trace.propagation.W3CTraceContextPropagator;
import io.opentelemetry.context.propagation.ContextPropagators;
import io.opentelemetry.exporter.otlp.http.logs.OtlpHttpLogRecordExporter;
import io.opentelemetry.exporter.otlp.http.metrics.OtlpHttpMetricExporter;
import io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter;
import io.opentelemetry.sdk.OpenTelemetrySdk;
import io.opentelemetry.sdk.logs.SdkLoggerProvider;
import io.opentelemetry.sdk.logs.export.BatchLogRecordProcessor;
import io.opentelemetry.sdk.metrics.SdkMeterProvider;
import io.opentelemetry.sdk.metrics.export.PeriodicMetricReader;
import io.opentelemetry.sdk.resources.Resource;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.export.BatchSpanProcessor;

public final class Telemetry {
    private static final String MAPLE_ENDPOINT = "https://ingest.maple.dev"; // EU: https://ingest.eu.maple.dev
    private static final String MAPLE_AUTH = "Bearer YOUR_INGEST_KEY";

    public static OpenTelemetry init() {
        Resource resource = Resource.getDefault().merge(Resource.create(Attributes.builder()
            .put("service.name", "my-java-app")
            .put("deployment.environment.name", System.getenv().getOrDefault("DEPLOYMENT_ENV", "development"))
            .put("vcs.repository.url.full", "https://github.com/acme/my-java-app")
            .build()));

        SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
            .setResource(resource)
            .addSpanProcessor(BatchSpanProcessor.builder(OtlpHttpSpanExporter.builder()
                .setEndpoint(MAPLE_ENDPOINT + "/v1/traces")
                .addHeader("Authorization", MAPLE_AUTH)
                .build()).build())
            .build();

        SdkMeterProvider meterProvider = SdkMeterProvider.builder()
            .setResource(resource)
            .registerMetricReader(PeriodicMetricReader.builder(OtlpHttpMetricExporter.builder()
                .setEndpoint(MAPLE_ENDPOINT + "/v1/metrics")
                .addHeader("Authorization", MAPLE_AUTH)
                .build()).build())
            .build();

        SdkLoggerProvider loggerProvider = SdkLoggerProvider.builder()
            .setResource(resource)
            .addLogRecordProcessor(BatchLogRecordProcessor.builder(OtlpHttpLogRecordExporter.builder()
                .setEndpoint(MAPLE_ENDPOINT + "/v1/logs")
                .addHeader("Authorization", MAPLE_AUTH)
                .build()).build())
            .build();

        OpenTelemetrySdk sdk = OpenTelemetrySdk.builder()
            .setTracerProvider(tracerProvider)
            .setMeterProvider(meterProvider)
            .setLoggerProvider(loggerProvider)
            .setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance()))
            .buildAndRegisterGlobal();

        Runtime.getRuntime().addShutdownHook(new Thread(sdk::close));
        return sdk;
    }
}
```

The example puts the endpoint and key in source. An ingest key can only write telemetry to your organization. It cannot read data or call the Maple API. Keeping it in source means the SDK always starts with a complete configuration, so a deploy that is missing an environment variable cannot silently turn telemetry off.

Call `Telemetry.init()` once, first thing in `main`. Without the agent, library spans come from the [library instrumentation](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md) packages you add yourself.

## Custom spans

```java
import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.StatusCode;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;

private static final Tracer tracer = GlobalOpenTelemetry.getTracer("my-app");

public void processOrder(String orderId) {
    Span span = tracer.spanBuilder("process-order").startSpan();
    try (Scope scope = span.makeCurrent()) {
        span.setAttribute("order.id", orderId);
        span.setAttribute("payment.method", "card");
        chargePayment(orderId);
    } catch (Exception e) {
        span.recordException(e);
        span.setStatus(StatusCode.ERROR, e.getMessage());
        throw e;
    } finally {
        span.end();
    }
}
```

Service map edges come from instrumented client spans that propagate `traceparent` to an instrumented callee, not from attributes such as `peer.service`. See [Service map](/docs/explore/service-map). With the agent, `opentelemetry-api` is the only dependency this code needs.

## Attributes Maple uses

Most attributes are optional. These unlock specific features.

### Commits and GitHub

Set `vcs.ref.head.revision` (the full 40-character commit SHA) and `vcs.repository.url.full` on the resource. They drive deploy markers, the Releases page and, with the [GitHub integration](/docs/integrations/github), commit hover cards on traces. Read the SHA from your CI or host at build time, such as `GITHUB_SHA` or `VERCEL_GIT_COMMIT_SHA`, not from `git` at runtime:

```bash
export OTEL_RESOURCE_ATTRIBUTES="vcs.repository.url.full=https://github.com/acme/api,vcs.ref.head.revision=$GITHUB_SHA"
```

### Databases

The Java agent instruments JDBC, R2DBC, Hibernate and the Mongo and Redis clients, and sets these attributes for you. Without the agent, set them on a `Client` span:

```java
Span span = tracer.spanBuilder("SELECT users")
    .setSpanKind(SpanKind.CLIENT)
    .setAttribute("db.system.name", "postgresql")
    .setAttribute("db.namespace", "app")
    .setAttribute("db.query.text", "SELECT * FROM users WHERE id = ?")
    .startSpan();
try (Scope scope = span.makeCurrent()) {
    return findUser(id);
} finally {
    span.end();
}
```

`db.system.name` and `db.namespace` draw the database node on the [service map](/docs/explore/service-map). `db.query.text` groups statements into query shapes, so record it with placeholders, not bound values. See [Database queries](/docs/concepts/otel-conventions#database-queries) for the full list.

## Log correlation

With the agent, Logback, Log4j2 and JUL records are exported as OpenTelemetry log records with the active trace and span IDs, so Maple links each log line to its trace. The agent also puts `trace_id` and `span_id` into the MDC if you want them in your console output:

```xml
<!-- logback.xml -->
<encoder>
    <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} trace_id=%X{trace_id} span_id=%X{span_id} - %msg%n</pattern>
</encoder>
```

With the manual SDK, add the `opentelemetry-logback-appender-1.0` library, register its `OpenTelemetryAppender` in `logback.xml`, and call `OpenTelemetryAppender.install(openTelemetry)` after `Telemetry.init()`.

## Spring Boot

Spring Boot works with the agent as shown above. For GraalVM native images, use the [OpenTelemetry Spring Boot starter](https://opentelemetry.io/docs/zero-code/java/spring-boot-starter/) instead. It reads the same `otel.*` properties from `application.properties` or the environment:

```kotlin
implementation(platform("io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom:2.16.0"))
implementation("io.opentelemetry.instrumentation:opentelemetry-spring-boot-starter")
```

## Verify

1. Start your application and send it a few requests.
2. In Maple, open **Explore → Traces**. The agent sends spans in batches every 5 seconds by default, and metrics every 60 seconds.
3. Each request should show up as one trace with a single root server span, named after the method and route (for example `GET /api/orders`), and child spans for the queries and outgoing calls it made.

Your service also appears on the **Services** page once its first spans arrive.

## Troubleshooting

- **`401` responses.** The key is wrong, was copied from the other region, or the header is malformed. The header must be `Authorization: Bearer YOUR_INGEST_KEY`. In `otel.exporter.otlp.headers` it is written `Authorization=Bearer YOUR_INGEST_KEY`. See [Ingest API status codes](/docs/reference/ingest#status-codes).
- **Wrong protocol or path.** Maple's [ingest API](/docs/reference/ingest) is OTLP over HTTP, so keep `otel.exporter.otlp.protocol` at `http/protobuf`. The agent endpoint is the base URL, while `setEndpoint` in the manual SDK takes the full signal path (`/v1/traces`).
- **Network.** From the machine running the app, run `curl -i https://ingest.maple.dev/v1/traces -X POST`. Any HTTP status code means the host can reach Maple. A timeout or DNS error means a firewall or proxy is blocking outbound HTTPS.
- **Nothing exported.** Check the startup log for a line from the OpenTelemetry agent; if it is missing, the `-javaagent` flag is not reaching the JVM. Add `-Dotel.javaagent.debug=true` to print spans and export errors.

## Next steps

- [Explore traces](/docs/explore/traces)
- [Track errors](/docs/errors/overview)
- [Create alert rules](/docs/alerting/alert-rules)
- [OpenTelemetry conventions](/docs/concepts/otel-conventions)

---

# Kotlin instrumentation

> Instrumentation · https://maple.dev/docs/guides/instrumentation-kotlin

Instrument a Kotlin JVM application (Ktor, Spring Boot) with OpenTelemetry and send traces, logs, and metrics to Maple.

This guide instruments a Kotlin JVM application (Ktor, Spring Boot or plain JVM) so it sends traces, logs and metrics to Maple. Kotlin runs on the JVM, so the OpenTelemetry Java agent works without code changes. For Android, see the [opentelemetry-android](https://github.com/open-telemetry/opentelemetry-android) project.

To have a coding agent do this setup, use the [maple-onboard](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-onboard) skill, and [maple-audit](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-audit) to check an existing setup.

## Prerequisites

- Kotlin 1.9+ on JDK 11+
- An ingest key from **Settings → Ingestion** in Maple. Use the private key (`maple_sk_…`) for server applications.

## Install

Download the agent jar from the [opentelemetry-java-instrumentation releases](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases):

```bash
curl -L -o opentelemetry-javaagent.jar \
  https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
```

## Configure

Start your application with the agent attached:

```bash
java \
  -javaagent:opentelemetry-javaagent.jar \
  -Dotel.service.name=my-kotlin-app \
  -Dotel.exporter.otlp.endpoint=https://ingest.maple.dev \
  -Dotel.exporter.otlp.protocol=http/protobuf \
  -Dotel.exporter.otlp.headers="Authorization=Bearer YOUR_INGEST_KEY" \
  -Dotel.resource.attributes="deployment.environment.name=production,vcs.repository.url.full=https://github.com/acme/my-kotlin-app" \
  -jar build/libs/app.jar
```

For an EU organization, use `https://ingest.eu.maple.dev`. The endpoint is the base URL: the agent appends `/v1/traces`, `/v1/logs` and `/v1/metrics` itself.

The agent (version 2.x) exports traces, metrics and logs over OTLP by default, so there is no separate metrics or logs setup. `http/protobuf` is already its default protocol. Setting it explicitly does no harm and documents what you expect.

## Environment variables

Every `-Dotel.*` property has an environment variable equivalent, which suits containers:

```bash
export JAVA_TOOL_OPTIONS="-javaagent:/opt/opentelemetry-javaagent.jar"
export OTEL_SERVICE_NAME="my-kotlin-app"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production,vcs.repository.url.full=https://github.com/acme/my-kotlin-app"
```

## Auto-instrumentation

The agent instruments Ktor, Spring, gRPC, JDBC, Kafka and [many more libraries](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md). Each incoming request becomes a server span, and queries and outgoing HTTP calls become child spans. The agent also reports JVM runtime metrics such as memory, garbage collection and thread counts.

## Manual SDK setup

Use this when you cannot attach the agent, for example in native images or restricted runtimes.

### Install dependencies

In `build.gradle.kts`:

```kotlin
dependencies {
    implementation(platform("io.opentelemetry:opentelemetry-bom:1.51.0"))
    implementation("io.opentelemetry:opentelemetry-api")
    implementation("io.opentelemetry:opentelemetry-sdk")
    implementation("io.opentelemetry:opentelemetry-exporter-otlp")
}
```

### Configure the SDK

```kotlin
// Telemetry.kt
import io.opentelemetry.api.OpenTelemetry
import io.opentelemetry.api.common.Attributes
import io.opentelemetry.api.trace.propagation.W3CTraceContextPropagator
import io.opentelemetry.context.propagation.ContextPropagators
import io.opentelemetry.exporter.otlp.http.logs.OtlpHttpLogRecordExporter
import io.opentelemetry.exporter.otlp.http.metrics.OtlpHttpMetricExporter
import io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter
import io.opentelemetry.sdk.OpenTelemetrySdk
import io.opentelemetry.sdk.logs.SdkLoggerProvider
import io.opentelemetry.sdk.logs.export.BatchLogRecordProcessor
import io.opentelemetry.sdk.metrics.SdkMeterProvider
import io.opentelemetry.sdk.metrics.export.PeriodicMetricReader
import io.opentelemetry.sdk.resources.Resource
import io.opentelemetry.sdk.trace.SdkTracerProvider
import io.opentelemetry.sdk.trace.export.BatchSpanProcessor

private const val MAPLE_ENDPOINT = "https://ingest.maple.dev" // EU: https://ingest.eu.maple.dev
private const val MAPLE_AUTH = "Bearer YOUR_INGEST_KEY"

object Telemetry {
    fun init(): OpenTelemetry {
        val resource = Resource.getDefault().merge(
            Resource.create(
                Attributes.builder()
                    .put("service.name", "my-kotlin-app")
                    .put("deployment.environment.name", System.getenv("DEPLOYMENT_ENV") ?: "development")
                    .put("vcs.repository.url.full", "https://github.com/acme/my-kotlin-app")
                    .build()
            )
        )

        val tracerProvider = SdkTracerProvider.builder()
            .setResource(resource)
            .addSpanProcessor(
                BatchSpanProcessor.builder(
                    OtlpHttpSpanExporter.builder()
                        .setEndpoint("$MAPLE_ENDPOINT/v1/traces")
                        .addHeader("Authorization", MAPLE_AUTH)
                        .build()
                ).build()
            )
            .build()

        val meterProvider = SdkMeterProvider.builder()
            .setResource(resource)
            .registerMetricReader(
                PeriodicMetricReader.builder(
                    OtlpHttpMetricExporter.builder()
                        .setEndpoint("$MAPLE_ENDPOINT/v1/metrics")
                        .addHeader("Authorization", MAPLE_AUTH)
                        .build()
                ).build()
            )
            .build()

        val loggerProvider = SdkLoggerProvider.builder()
            .setResource(resource)
            .addLogRecordProcessor(
                BatchLogRecordProcessor.builder(
                    OtlpHttpLogRecordExporter.builder()
                        .setEndpoint("$MAPLE_ENDPOINT/v1/logs")
                        .addHeader("Authorization", MAPLE_AUTH)
                        .build()
                ).build()
            )
            .build()

        val sdk = OpenTelemetrySdk.builder()
            .setTracerProvider(tracerProvider)
            .setMeterProvider(meterProvider)
            .setLoggerProvider(loggerProvider)
            .setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance()))
            .buildAndRegisterGlobal()

        Runtime.getRuntime().addShutdownHook(Thread { sdk.close() })
        return sdk
    }
}
```

The example puts the endpoint and key in source. An ingest key can only write telemetry to your organization. It cannot read data or call the Maple API. Keeping it in source means the SDK always starts with a complete configuration, so a deploy that is missing an environment variable cannot silently turn telemetry off.

Call `Telemetry.init()` once, first thing in `main`. It registers the SDK globally, so calling it a second time throws.

### Ktor

Without the agent, add the Ktor instrumentation library for server and client spans:

```kotlin
dependencies {
    implementation("io.opentelemetry.instrumentation:opentelemetry-ktor-3.0:2.16.0-alpha")
}
```

```kotlin
import io.ktor.client.HttpClient
import io.ktor.client.engine.cio.CIO
import io.ktor.server.application.Application
import io.ktor.server.application.install
import io.opentelemetry.api.GlobalOpenTelemetry
import io.opentelemetry.instrumentation.ktor.v3_0.KtorClientTelemetry
import io.opentelemetry.instrumentation.ktor.v3_0.KtorServerTelemetry

fun Application.configureMonitoring() {
    install(KtorServerTelemetry) {
        setOpenTelemetry(GlobalOpenTelemetry.get())
    }
}

val client = HttpClient(CIO) {
    install(KtorClientTelemetry) {
        setOpenTelemetry(GlobalOpenTelemetry.get())
    }
}
```

### Spring Boot

Spring Boot apps in Kotlin work the same as Java apps. Use the agent, or for GraalVM native images use the [OpenTelemetry Spring Boot starter](https://opentelemetry.io/docs/zero-code/java/spring-boot-starter/), which reads the same `otel.*` properties:

```kotlin
implementation(platform("io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom:2.16.0"))
implementation("io.opentelemetry.instrumentation:opentelemetry-spring-boot-starter")
```

## Custom spans

```kotlin
import io.opentelemetry.api.GlobalOpenTelemetry
import io.opentelemetry.api.trace.Span
import io.opentelemetry.api.trace.StatusCode
import io.opentelemetry.api.trace.Tracer

private val tracer = GlobalOpenTelemetry.getTracer("my-app")

inline fun <T> Tracer.span(name: String, block: (Span) -> T): T {
    val span = spanBuilder(name).startSpan()
    return span.makeCurrent().use {
        try {
            block(span)
        } catch (e: Throwable) {
            span.recordException(e)
            span.setStatus(StatusCode.ERROR, e.message ?: "")
            throw e
        } finally {
            span.end()
        }
    }
}

fun processOrder(orderId: String) = tracer.span("process-order") { span ->
    span.setAttribute("order.id", orderId)
    span.setAttribute("payment.method", "card")
    chargePayment(orderId)
}
```

`makeCurrent()` binds the span to the current thread. In coroutines that switch threads, wrap the work in `withContext(Context.current().asContextElement())` from the `opentelemetry-extension-kotlin` library so the span follows the coroutine.

Service map edges come from instrumented client spans that propagate `traceparent` to an instrumented callee, not from attributes such as `peer.service`. See [Service map](/docs/explore/service-map).

## Attributes Maple uses

Most attributes are optional. These unlock specific features.

### Commits and GitHub

Set `vcs.ref.head.revision` (the full 40-character commit SHA) and `vcs.repository.url.full` on the resource. They drive deploy markers, the Releases page and, with the [GitHub integration](/docs/integrations/github), commit hover cards on traces. Read the SHA from your CI or host at build time, such as `GITHUB_SHA` or `VERCEL_GIT_COMMIT_SHA`, not from `git` at runtime:

```bash
export OTEL_RESOURCE_ATTRIBUTES="vcs.repository.url.full=https://github.com/acme/api,vcs.ref.head.revision=$GITHUB_SHA"
```

### Databases

The Java agent instruments JDBC, R2DBC and the Mongo and Redis clients, and sets these attributes for you. Without the agent, set them on a `Client` span:

```kotlin
val span = tracer.spanBuilder("SELECT users")
    .setSpanKind(SpanKind.CLIENT)
    .setAttribute("db.system.name", "postgresql")
    .setAttribute("db.namespace", "app")
    .setAttribute("db.query.text", "SELECT * FROM users WHERE id = ?")
    .startSpan()
try {
    span.makeCurrent().use { findUser(id) }
} finally {
    span.end()
}
```

`db.system.name` and `db.namespace` draw the database node on the [service map](/docs/explore/service-map). `db.query.text` groups statements into query shapes, so record it with placeholders, not bound values. See [Database queries](/docs/concepts/otel-conventions#database-queries) for the full list.

## Log correlation

With the agent, Logback, Log4j2 and JUL records are exported as OpenTelemetry log records with the active trace and span IDs, so Maple links each log line to its trace. The agent also puts `trace_id` and `span_id` into the MDC if you want them in your console output:

```xml
<!-- src/main/resources/logback.xml -->
<encoder>
    <pattern>%d{HH:mm:ss.SSS} %-5level trace_id=%X{trace_id} span_id=%X{span_id} - %msg%n</pattern>
</encoder>
```

With the manual SDK, add the `opentelemetry-logback-appender-1.0` library, register its `OpenTelemetryAppender` in `logback.xml`, and call `OpenTelemetryAppender.install(openTelemetry)` after `Telemetry.init()`.

## Verify

1. Start your application and send it a few requests.
2. In Maple, open **Explore → Traces**. The agent sends spans in batches every 5 seconds by default, and metrics every 60 seconds.
3. Each request should show up as one trace with a single root server span, named after the method and route (for example `GET /api/orders`), and child spans for the queries and outgoing calls it made.

Your service also appears on the **Services** page once its first spans arrive.

## Troubleshooting

- **`401` responses.** The key is wrong, was copied from the other region, or the header is malformed. The header must be `Authorization: Bearer YOUR_INGEST_KEY`. In `otel.exporter.otlp.headers` it is written `Authorization=Bearer YOUR_INGEST_KEY`. See [Ingest API status codes](/docs/reference/ingest#status-codes).
- **Wrong protocol or path.** Maple's [ingest API](/docs/reference/ingest) is OTLP over HTTP, so keep `otel.exporter.otlp.protocol` at `http/protobuf`. The agent endpoint is the base URL, while `setEndpoint` in the manual SDK takes the full signal path (`/v1/traces`).
- **Network.** From the machine running the app, run `curl -i https://ingest.maple.dev/v1/traces -X POST`. Any HTTP status code means the host can reach Maple. A timeout or DNS error means a firewall or proxy is blocking outbound HTTPS.
- **Nothing exported.** Check the startup log for a line from the OpenTelemetry agent; if it is missing, the `-javaagent` flag is not reaching the JVM. Add `-Dotel.javaagent.debug=true` to print spans and export errors.

## Next steps

- [Explore traces](/docs/explore/traces)
- [Track errors](/docs/errors/overview)
- [Create alert rules](/docs/alerting/alert-rules)
- [OpenTelemetry conventions](/docs/concepts/otel-conventions)

---

# C# / .NET instrumentation

> Instrumentation · https://maple.dev/docs/guides/instrumentation-csharp

Instrument a .NET application with OpenTelemetry and send traces, logs, and metrics to Maple.

This guide sets up the OpenTelemetry .NET SDK in an ASP.NET Core or generic-host application so it sends traces, logs and metrics to Maple, with instrumentation for ASP.NET Core and `HttpClient`.

To have a coding agent do this setup, use the [maple-onboard](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-onboard) skill, and [maple-audit](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-audit) to check an existing setup.

## Prerequisites

- .NET 8+ (the packages also target .NET Framework 4.6.2+)
- An ingest key from **Settings → Ingestion** in Maple. Use the private key (`maple_sk_…`) for server applications.

## Install

```bash
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
```

## Configure

Wire OpenTelemetry into the host builder in `Program.cs`. Each signal gets its own OTLP exporter:

```csharp
using OpenTelemetry;
using OpenTelemetry.Exporter;
using OpenTelemetry.Logs;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

const string MapleEndpoint = "https://ingest.maple.dev"; // EU: https://ingest.eu.maple.dev
const string MapleHeaders = "Authorization=Bearer YOUR_INGEST_KEY";

void ConfigureMaple(OtlpExporterOptions opts, string path)
{
    opts.Endpoint = new Uri($"{MapleEndpoint}{path}");
    opts.Protocol = OtlpExportProtocol.HttpProtobuf;
    opts.Headers = MapleHeaders;
}

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(serviceName: "my-dotnet-app", serviceVersion: "1.0.0")
        .AddAttributes(new Dictionary<string, object>
        {
            ["deployment.environment.name"] = builder.Environment.EnvironmentName,
            ["vcs.repository.url.full"] = "https://github.com/acme/my-dotnet-app",
        }))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddOtlpExporter(opts => ConfigureMaple(opts, "/v1/traces")))
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddOtlpExporter(opts => ConfigureMaple(opts, "/v1/metrics")))
    .WithLogging(logging => logging
        .AddOtlpExporter(opts => ConfigureMaple(opts, "/v1/logs")));

var app = builder.Build();
app.MapGet("/", () => "Hello!");
app.Run();
```

When you set `Endpoint` in code, it must include the signal path (`/v1/traces`, `/v1/metrics`, `/v1/logs`). The exporter's default protocol is gRPC, so set `HttpProtobuf` on each one.

The example puts the endpoint and key in source. An ingest key can only write telemetry to your organization. It cannot read data or call the Maple API. Keeping it in source means the SDK always starts with a complete configuration, so a deploy that is missing an environment variable cannot silently turn telemetry off. To keep the key out of source, use [environment variables](#environment-variables) instead.

For a console app or worker without a host, build the providers directly and dispose them on exit:

```csharp
using var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .ConfigureResource(r => r.AddService("my-dotnet-worker"))
    .AddSource("my-dotnet-worker")
    .AddHttpClientInstrumentation()
    .AddOtlpExporter(opts => ConfigureMaple(opts, "/v1/traces"))
    .Build();

using var meterProvider = Sdk.CreateMeterProviderBuilder()
    .ConfigureResource(r => r.AddService("my-dotnet-worker"))
    .AddMeter("my-dotnet-worker")
    .AddOtlpExporter(opts => ConfigureMaple(opts, "/v1/metrics"))
    .Build();
```

## Environment variables

`UseOtlpExporter()` sets up one OTLP exporter for every signal you enabled and reads its settings from the standard environment variables:

```bash
export OTEL_SERVICE_NAME="my-dotnet-app"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production,vcs.repository.url.full=https://github.com/acme/my-dotnet-app"
```

```csharp
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing.AddAspNetCoreInstrumentation().AddHttpClientInstrumentation())
    .WithMetrics(metrics => metrics.AddAspNetCoreInstrumentation().AddHttpClientInstrumentation())
    .WithLogging()
    .UseOtlpExporter();
```

With `http/protobuf`, the exporter appends `/v1/traces`, `/v1/metrics` and `/v1/logs` to the base endpoint. Traces, metrics and logs are all exported, because each one is enabled above.

## Auto-instrumentation

.NET instrumentation comes from one NuGet package per library, registered on the tracing or metrics builder with `.AddXxxInstrumentation()`:

| Library               | Package                                             |
| --------------------- | --------------------------------------------------- |
| ASP.NET Core          | `OpenTelemetry.Instrumentation.AspNetCore`          |
| HttpClient            | `OpenTelemetry.Instrumentation.Http`                |
| Entity Framework Core | `OpenTelemetry.Instrumentation.EntityFrameworkCore` |
| SqlClient             | `OpenTelemetry.Instrumentation.SqlClient`           |
| StackExchange.Redis   | `OpenTelemetry.Instrumentation.StackExchangeRedis`  |
| gRPC client           | `OpenTelemetry.Instrumentation.GrpcNetClient`       |
| .NET runtime metrics  | `OpenTelemetry.Instrumentation.Runtime`             |

ASP.NET Core turns each incoming request into a server span and records request duration metrics. `HttpClient` calls become client spans that carry the trace context to the service you call.

## Custom spans

In .NET, spans are `System.Diagnostics.Activity` objects. Create an `ActivitySource` and start activities from it:

```csharp
using System.Diagnostics;

public class OrderService
{
    private static readonly ActivitySource ActivitySource = new("MyApp.Orders");

    public async Task ProcessOrder(string orderId)
    {
        using var activity = ActivitySource.StartActivity("process-order");
        activity?.SetTag("order.id", orderId);
        activity?.SetTag("payment.method", "card");

        try
        {
            await ChargePayment(orderId);
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.AddException(ex);
            throw;
        }
    }
}
```

Register the source name so its activities are exported:

```csharp
.WithTracing(tracing => tracing
    .AddSource("MyApp.Orders")
    // ...
)
```

Service map edges come from instrumented client spans that propagate `traceparent` to an instrumented callee, not from attributes such as `peer.service`. See [Service map](/docs/explore/service-map).

## Attributes Maple uses

Most attributes are optional. These unlock specific features.

### Commits and GitHub

Set `vcs.ref.head.revision` (the full 40-character commit SHA) and `vcs.repository.url.full` on the resource. They drive deploy markers, the Releases page and, with the [GitHub integration](/docs/integrations/github), commit hover cards on traces. Read the SHA from your CI or host at build time, such as `GITHUB_SHA` or `VERCEL_GIT_COMMIT_SHA`, not from `git` at runtime:

```bash
export OTEL_RESOURCE_ATTRIBUTES="vcs.repository.url.full=https://github.com/acme/api,vcs.ref.head.revision=$GITHUB_SHA"
```

### Databases

Database instrumentation sets these attributes for you. For a driver without instrumentation, set them on a `Client` span:

```csharp
using var activity = ActivitySource.StartActivity("SELECT users", ActivityKind.Client);
activity?.SetTag("db.system.name", "postgresql");
activity?.SetTag("db.namespace", "app");
activity?.SetTag("db.query.text", "SELECT * FROM users WHERE id = @id");
```

`db.system.name` and `db.namespace` draw the database node on the [service map](/docs/explore/service-map). `db.query.text` groups statements into query shapes, so record it with placeholders, not bound values. See [Database queries](/docs/concepts/otel-conventions#database-queries) for the full list.

## Log correlation

With `WithLogging()` configured, `ILogger<T>` records are exported as OpenTelemetry log records. Records written during an active activity carry its trace and span IDs, so Maple links each log line to its trace:

```csharp
public class OrderService(ILogger<OrderService> logger)
{
    public async Task ProcessOrder(string orderId)
    {
        logger.LogInformation("Processing order {OrderId}", orderId);
    }
}
```

## Verify

1. Run your application (`dotnet run`) and send it a few requests.
2. In Maple, open **Explore → Traces**. The SDK sends spans in batches every 5 seconds by default, and metrics every 60 seconds.
3. Each request should show up as one trace with a single root server span, named after the method and route (for example `GET /api/orders`), and child spans for the queries and outgoing calls it made.

Your service also appears on the **Services** page once its first spans arrive.

## Troubleshooting

- **`401` responses.** The key is wrong, was copied from the other region, or the header is malformed. `OtlpExporterOptions.Headers` and `OTEL_EXPORTER_OTLP_HEADERS` both take `Authorization=Bearer YOUR_INGEST_KEY`. See [Ingest API status codes](/docs/reference/ingest#status-codes).
- **Wrong protocol or path.** Maple accepts OTLP over HTTP, and the .NET exporter defaults to gRPC. Set `OtlpExportProtocol.HttpProtobuf` in code or `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf`. An endpoint set in code needs the full signal path; the environment variable takes only the base URL.
- **Network.** From the machine running the app, run `curl -i https://ingest.maple.dev/v1/traces -X POST`. Any HTTP status code means the host can reach Maple. A timeout or DNS error means a firewall or proxy is blocking outbound HTTPS.
- **Nothing exported.** Custom spans only export when their `ActivitySource` name is registered with `AddSource`, and custom metrics when their `Meter` name is registered with `AddMeter`. In console apps, dispose the providers before exit so the last batch is sent.

## Next steps

- [Explore traces](/docs/explore/traces)
- [Track errors](/docs/errors/overview)
- [Create alert rules](/docs/alerting/alert-rules)
- [OpenTelemetry conventions](/docs/concepts/otel-conventions)

---

# Laravel instrumentation

> Instrumentation · https://maple.dev/docs/guides/instrumentation-laravel

Instrument a Laravel application with OpenTelemetry and send traces, logs, and metrics to Maple.

This guide instruments a Laravel application with the [`keepsuit/laravel-opentelemetry`](https://github.com/keepsuit/laravel-opentelemetry) package, which hooks OpenTelemetry into Laravel's HTTP kernel, database, queue, cache and logging, and sends traces, logs and metrics to Maple.

To have a coding agent do this setup, use the [maple-onboard](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-onboard) skill, and [maple-audit](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-audit) to check an existing setup.

## Prerequisites

- PHP 8.2+ and Laravel 11.31 or later (the versions the current package release supports)
- Composer
- An ingest key from **Settings → Ingestion** in Maple. Use the private key (`maple_sk_…`) for server applications.

## Install

```bash
composer require keepsuit/laravel-opentelemetry
```

Publish the config file so you can turn individual instrumentations on and off later:

```bash
php artisan vendor:publish \
  --provider="Keepsuit\LaravelOpenTelemetry\LaravelOpenTelemetryServiceProvider" \
  --tag="opentelemetry-config"
```

The package pulls in the OpenTelemetry PHP SDK and OTLP exporter, and registers its middleware and instrumentation through a service provider.

## Configure

The package reads the standard OpenTelemetry environment variables and exports traces, metrics and logs over OTLP by default. Point them at Maple's ingest endpoint with your ingest key:

```env
# .env
OTEL_SERVICE_NAME=my-laravel-app

OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=otlp
OTEL_LOGS_EXPORTER=otlp

OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"

OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production,vcs.repository.url.full=https://github.com/acme/my-laravel-app"
```

For an EU organization, use `https://ingest.eu.maple.dev`. The endpoint is the base URL; the exporters append `/v1/traces`, `/v1/metrics` and `/v1/logs`.

The package applies `OTEL_EXPORTER_OTLP_HEADERS` to every signal. If you run an older release that ignores it, set `OTEL_EXPORTER_OTLP_TRACES_HEADERS`, `OTEL_EXPORTER_OTLP_METRICS_HEADERS` and `OTEL_EXPORTER_OTLP_LOGS_HEADERS` to the same value.

An ingest key can only write telemetry to your organization. It cannot read data or call the Maple API. Still keep it in `.env` or your secret store rather than in committed config.

### `http/protobuf` or `http/json`

Maple accepts both encodings. The difference is on the PHP side:

- `http/protobuf` needs a protobuf implementation. The pure-PHP `google/protobuf` library works but is slow. For production, install the protobuf C extension (`pecl install protobuf`, then `extension=protobuf.so`).
- `http/json` needs no extension. If protobuf export fails in your environment, for example in a minimal Laravel Sail container, set `OTEL_EXPORTER_OTLP_PROTOCOL=http/json`.

## Auto-instrumentation

Once installed, the package creates telemetry for:

- **HTTP requests:** a server span per incoming request, and the `http.server.request.duration` metric
- **HTTP client:** a client span for each request made with the `Http` facade, with the trace context propagated to the service you call, and the `http.client.request.duration` metric
- **Database:** a span per Eloquent or query builder statement, and the `db.client.operation.duration` metric
- **Queue jobs:** producer and consumer spans, with context carried across the queue
- **Redis:** commands
- **Cache:** hits and misses, recorded as span events
- **Views and Livewire components:** a span per render
- **Console commands:** a span per run, for the commands you list in the `ConsoleInstrumentation` config
- **Events:** recorded as span events, with configurable exclusions

Each instrumentation can be turned off with its `OTEL_INSTRUMENTATION_*` variable (for example `OTEL_INSTRUMENTATION_QUERY=false`) or in the published `config/opentelemetry.php`.

## Custom spans

Use the `Tracer` facade. `measure()` starts the span, makes it the active span while the callback runs, and ends it:

```php
use Keepsuit\LaravelOpenTelemetry\Facades\Tracer;

Tracer::newSpan('process-order')
    ->setAttributes([
        'order.id' => $orderId,
        'payment.method' => 'card',
    ])
    ->measure(fn () => $this->chargePayment($orderId));
```

To record an exception or set the status yourself, start and activate the span by hand. The span has the OpenTelemetry PHP API (`setAttribute`, `recordException`, `setStatus`):

```php
use Keepsuit\LaravelOpenTelemetry\Facades\Tracer;
use OpenTelemetry\API\Trace\StatusCode;

$span = Tracer::newSpan('process-order')->start();
$scope = $span->activate();

try {
    return $this->chargePayment($orderId);
} catch (\Throwable $e) {
    $span->recordException($e);
    $span->setStatus(StatusCode::STATUS_ERROR, $e->getMessage());
    throw $e;
} finally {
    $scope->detach();
    $span->end();
}
```

Service map edges come from instrumented client spans that propagate `traceparent` to an instrumented callee, not from attributes such as `peer.service`. See [Service map](/docs/explore/service-map). The facade also has `Tracer::traceId()`, `Tracer::activeSpan()` and `Tracer::propagationHeaders()` for correlating logs and propagating context by hand.

For custom metrics, use the `Meter` facade:

```php
use Keepsuit\LaravelOpenTelemetry\Facades\Meter;

Meter::counter('orders.processed', 'orders', 'Orders processed')->add(1);
```

## Attributes Maple uses

Most attributes are optional. These unlock specific features.

### Commits and GitHub

Set `vcs.ref.head.revision` (the full 40-character commit SHA) and `vcs.repository.url.full` on the resource. They drive deploy markers, the Releases page and, with the [GitHub integration](/docs/integrations/github), commit hover cards on traces. Read the SHA from your CI or host at build time, such as `GITHUB_SHA` or `VERCEL_GIT_COMMIT_SHA`, not from `git` at runtime:

```bash
export OTEL_RESOURCE_ATTRIBUTES="vcs.repository.url.full=https://github.com/acme/api,vcs.ref.head.revision=$GITHUB_SHA"
```

### Databases

The package records a span for every Eloquent and query builder statement, with the standard `db.*` attributes. `db.system.name` and `db.namespace` draw the database node on the [service map](/docs/explore/service-map), and `db.query.text` groups statements into query shapes. Keep `OTEL_INSTRUMENTATION_QUERY` on. See [Database queries](/docs/concepts/otel-conventions#database-queries) for the full list.

## Log correlation

The package adds a log channel named `otlp` that sends log records to Maple with the active trace ID attached. Route logs to it with `LOG_CHANNEL=otlp`, or add it to a stack so logs also keep going to your files:

```php
// config/logging.php
'channels' => [
    'stack' => [
        'driver' => 'stack',
        'channels' => ['single', 'otlp'],
    ],
],
```

Logs then line up with the trace that produced them in Maple.

## Local mode and Docker (Laravel Sail)

To send to [Maple Local](/docs/local-mode) running on your host, the exporter inside the Sail container must reach back to the host. Maple Local binds `127.0.0.1` by default, which a container cannot reach, so start it on all interfaces:

```bash
maple start --host 0.0.0.0
```

A non-loopback bind exposes ingest, the UI and queries to your network without authentication, so do this only on a trusted network. See the [CLI reference](/docs/reference/cli#maple-start).

Then point the exporter at `host.docker.internal` and the OTLP/HTTP port `4318`:

```env
# .env (Sail container sending to Maple Local on the host)
OTEL_EXPORTER_OTLP_ENDPOINT=http://host.docker.internal:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/json
```

- Maple Local needs no `Authorization` header, so drop the headers variable.
- On Linux hosts where `host.docker.internal` does not resolve, add `extra_hosts: ["host.docker.internal:host-gateway"]` to the service in `docker-compose.yml`.

## Verify

1. Start your application and send it a few requests, or dispatch a queued job.
2. In Maple, open **Explore → Traces**. If nothing appears after a minute, see [Troubleshooting](#troubleshooting).
3. Each request should show up as one trace with a single root server span, named after the method and route (for example `GET /orders`), and child spans for the queries and jobs it triggered.
4. If you routed logs to the `otlp` channel, open **Explore → Logs** and check that log lines link to their traces.

Your service also appears on the **Services** page once its first spans arrive.

## Troubleshooting

- **`401` responses.** The key is wrong, was copied from the other region, or the header is malformed. `OTEL_EXPORTER_OTLP_HEADERS` takes `Authorization=Bearer YOUR_INGEST_KEY`. See [Ingest API status codes](/docs/reference/ingest#status-codes).
- **Protocol.** Maple accepts OTLP over HTTP as protobuf or JSON. If exports fail with protobuf errors, switch to `http/json`, or install the protobuf extension.
- **Network.** From inside the container or server, run `curl -i https://ingest.maple.dev/v1/traces -X POST`. Any HTTP status code means the host can reach Maple. A timeout or DNS error means a firewall or proxy is blocking outbound HTTPS.
- **Nothing exported.** Run `php artisan config:clear` after changing `.env`; a cached config keeps the old values. Check that `OTEL_SDK_DISABLED` is not set to `true`.

## Next steps

- [Explore traces](/docs/explore/traces)
- [Track errors](/docs/errors/overview)
- [Create alert rules](/docs/alerting/alert-rules)
- [OpenTelemetry conventions](/docs/concepts/otel-conventions)

---

# Trace your AI agent

> AI Frameworks · https://maple.dev/docs/agent-tracing

Setup guides for sending each supported agent framework and LLM gateway to Maple as one Agent Session per conversation.

import GuideGrid from "../../components/docs/GuideGrid.astro"
import { AGENT_GATEWAY_CARDS, AGENT_GUIDE_CATEGORIES } from "../../lib/agent-tracing-guides"

Each guide sets up one framework so every conversation shows up in Maple as one [Agent Session](/docs/agent-sessions/overview). You turn on the framework's own OpenTelemetry instrumentation and point it at Maple; there is no Maple SDK.

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing) skill, which detects your framework and installs the matching skill.

```text
Set up Maple agent tracing in this project.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## Frameworks and libraries

{AGENT_GUIDE_CATEGORIES.map((category) => (
	<>
		<h3 id={category.id}>{category.title}</h3>
		<GuideGrid cards={category.cards} />
	</>
))}

## Gateways

<GuideGrid cards={AGENT_GATEWAY_CARDS} />

If you use a gateway like OpenRouter or LiteLLM together with a framework, instrument only the framework, or every model call is recorded twice. The [OpenRouter guide](/docs/agent-tracing/openrouter) covers the one exception.

## Connection details

Every guide exports OTLP over HTTP with an ingest key from **Settings → Ingestion**:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"   # https://ingest.eu.maple.dev for EU organizations
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
export OTEL_SERVICE_NAME="support-agent"
```

Maple doesn't accept gRPC, so set the protocol to `http/protobuf` if your exporter defaults to it. A traces endpoint passed in code or through `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` must end in `/v1/traces`.

Keep agent traffic at 100% sampling, or sampled-out turns leave gaps in the session. See [Sampling and throughput](/docs/concepts/sampling-throughput).

## Not listed?

Anything that emits the OpenTelemetry GenAI conventions works; [Any language](/docs/agent-tracing/opentelemetry) lists what Maple reads. OpenInference and OpenLLMetry instrumentations also work, shown as **Unidentified**. Tell us what you run at [support@maple.dev](mailto:support@maple.dev) or on [Discord](https://discord.gg/BnXjKuwJqP) and we'll add a guide.

---

# Trace Vercel AI SDK agents with OpenTelemetry

> AI Frameworks · https://maple.dev/docs/agent-tracing/vercel-ai-sdk

Send the Vercel AI SDK's OpenTelemetry spans to Maple and group each chat into one Agent Session.

The Vercel AI SDK emits OpenTelemetry spans for every `generateText`, `streamText` and `ToolLoopAgent` call, and Maple reads them without an extra instrumentation package.

In AI SDK 7 nothing is traced until you call `registerTelemetry()` at startup. You also pass a conversation id on every call, or each message becomes its own session.

You need `ai` 7.0.106 or newer and Node.js 22 or newer (Bun works too). On AI SDK 5 or 6, run `npx @ai-sdk/codemod v7` first.

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing-vercel-ai-sdk](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-vercel-ai-sdk) skill and follows it.

```text
Set up Maple agent tracing for the Vercel AI SDK in this project.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing-vercel-ai-sdk -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## Install the packages

```bash
npm install ai@^7.0.106 @ai-sdk/otel @opentelemetry/sdk-node
```

## Point the exporter at Maple

```bash
export OTEL_SERVICE_NAME="support-agent"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
```

For an EU organization, use `https://ingest.eu.maple.dev`. The exporter appends `/v1/traces` itself.

## Register the AI SDK integration

Create an `instrumentation.ts` and import it as the first line of your entry point (`import "./instrumentation"`):

```ts
// instrumentation.ts
import { OpenTelemetry } from "@ai-sdk/otel"
import { NodeSDK } from "@opentelemetry/sdk-node"
import { registerTelemetry } from "ai"

// Reads OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES and OTEL_EXPORTER_OTLP_*
export const sdk = new NodeSDK()
sdk.start()

registerTelemetry(
	new OpenTelemetry({
		usage: true,
		runtimeContext: true,
	}),
)
```

Call `registerTelemetry()` exactly once. Each call adds another integration, and each one emits its own copy of every span.

### Serverless, or an app that already uses OpenTelemetry

Two setups need a handle on the span processor: a serverless handler that flushes after every invocation, and an app that already starts its own OpenTelemetry SDK (auto-instrumentation, Sentry, your own `NodeTracerProvider`), where you shouldn't start a second one. Install the exporter packages and create the processor yourself:

```bash
npm install @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-proto
```

```ts
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto"
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base"

export const spanProcessor = new BatchSpanProcessor(new OTLPTraceExporter())
```

Pass it as `new NodeSDK({ spanProcessors: [spanProcessor] })`, or add it to your existing provider instead of creating a `NodeSDK`. Keep the `registerTelemetry()` call either way.

### Next.js

In Next.js, register both in the `register()` function of `instrumentation.ts`, using `@vercel/otel` as in the [Next.js guide](/docs/guides/instrumentation-nextjs):

```ts
// instrumentation.ts (project root, or src/instrumentation.ts)
import { OpenTelemetry } from "@ai-sdk/otel"
import { OTLPHttpProtoTraceExporter, registerOTel } from "@vercel/otel"
import { registerTelemetry } from "ai"

export function register() {
	const mapleKey = process.env.MAPLE_INGEST_KEY
	if (!mapleKey) {
		// A missing key disables export; it never stops the app.
		console.warn("MAPLE_INGEST_KEY is not set; Maple telemetry export is disabled")
		return
	}
	registerOTel({
		serviceName: "support-chat",
		traceExporter: new OTLPHttpProtoTraceExporter({
			url: "https://ingest.maple.dev/v1/traces", // EU: https://ingest.eu.maple.dev/v1/traces
			headers: { authorization: `Bearer ${mapleKey}` },
		}),
	})

	registerTelemetry(
		new OpenTelemetry({
			usage: true,
			runtimeContext: true,
		}),
	)
}
```

Set `MAPLE_INGEST_KEY` to your ingest key; this exporter doesn't read the `OTEL_EXPORTER_OTLP_*` variables.

Return AI SDK streams as the response (`toUIMessageStreamResponse()` or `createAgentUIStreamResponse()`). `@vercel/otel` ends all open spans when the request ends, so a stream read later, for example in `after()`, loses its reply and token counts.

## Pass the conversation id on every call

Pass the id as `runtimeContext.conversationId`. It only reaches telemetry if you also list it in `includeRuntimeContext`:

```ts
import { streamText } from "ai"

const result = streamText({
	model,
	messages,
	runtimeContext: { conversationId: chatId },
	telemetry: { functionId: "support_agent", includeRuntimeContext: { conversationId: true } },
})
```

Use the chat or thread id your app already stores. It must stay the same for the whole conversation and differ between conversations. `functionId` becomes the agent name in Maple, so give each agent a distinct one.

`ToolLoopAgent` doesn't take `runtimeContext` per call. Declare the id as a call option and turn it into runtime context in `prepareCall`:

```ts
// agent.ts
import { ToolLoopAgent } from "ai"
import { z } from "zod"

export const assistant = new ToolLoopAgent({
	// ...model, instructions, tools
	callOptionsSchema: z.object({ conversationId: z.string() }),
	prepareCall: ({ options, ...rest }) => ({
		...rest,
		runtimeContext: { conversationId: options.conversationId },
	}),
	telemetry: { functionId: "support_agent", includeRuntimeContext: { conversationId: true } },
})

const result = await assistant.generate({ messages, options: { conversationId: chatId } })
```

With `useChat`, the request body already carries a stable chat id as `id`. Pass it through:

```ts
// app/api/chat/route.ts
import { createAgentUIStreamResponse, type UIMessage } from "ai"
import { assistant } from "@/agent"

export async function POST(req: Request) {
	const { id, messages }: { id: string; messages: UIMessage[] } = await req.json()

	return createAgentUIStreamResponse({
		agent: assistant,
		uiMessages: messages,
		options: { conversationId: id },
	})
}
```

Sub-agents called from a tool's `execute` don't need the id, only their own `functionId`.

To keep a call's prompts and replies out of Maple, set `recordInputs: false` and `recordOutputs: false` in its `telemetry`.

## Flush in scripts and serverless functions

A short-lived process can exit before its spans are exported. In a script, call `await sdk.shutdown()` in a `finally` block before exiting. In a serverless handler, call `await spanProcessor.forceFlush()` in a `finally` instead (see [the variant above](#serverless-or-an-app-that-already-uses-opentelemetry)).

Read streams to the end (`await result.consumeStream()`) before flushing, since a stream's spans only end when it has been read. On Vercel, `@vercel/otel` flushes after each request, but queue consumers and cron jobs need their own `forceFlush()`.

## Check that it works

Run a conversation with two messages and a tool call, then open **Agent Sessions** in Maple. You should see one session named after your conversation id with framework **Vercel AI SDK**, one turn per call, and a transcript with the prompts, replies and tool calls.

Cost shows as unpriced because the AI SDK doesn't report it.

## Troubleshooting

- **No AI spans at all.** `registerTelemetry()` never ran. `experimental_telemetry: { isEnabled: true }` alone does nothing in AI SDK 7.
- **Every message is its own session.** Check all three parts: `runtimeContext: true` in `registerTelemetry()`, `runtimeContext: { conversationId }` on the call (or `prepareCall`), and `includeRuntimeContext: { conversationId: true }`.
- **Every span shows up twice.** `registerTelemetry()` ran twice, or a second OpenTelemetry SDK (such as Sentry without `skipOpenTelemetrySetup: true`) exports the same spans.
- **A failed tool shows as successful.** The tool returned an error value. Throw from `execute` instead.

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview)
- [AI SDK telemetry](https://ai-sdk.dev/docs/ai-sdk-core/telemetry)

---

# Trace Mastra agents and workflows with OpenTelemetry

> AI Frameworks · https://maple.dev/docs/agent-tracing/mastra

Export Mastra's built-in spans to Maple with @mastra/otel-exporter and group each conversation into one Agent Session.

Mastra traces agent runs, model calls and tool calls itself, and `@mastra/otel-exporter` sends those spans to Maple. You don't need an OpenTelemetry SDK or an instrumentation package.

The session id is Mastra's memory thread id, so every call of a conversation must pass the same `memory: { thread }`.

You need `@mastra/core` 1.x and Node.js 22.13 or newer.

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing-mastra](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-mastra) skill and follows it.

```text
Set up Maple agent tracing for Mastra in this project.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing-mastra -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## Install the observability packages

```bash
npm install @mastra/core@latest @mastra/observability@latest @mastra/otel-exporter@latest
```

Keep `@mastra/core`, `@mastra/observability` and `@mastra/otel-exporter` on releases from the same week, or the exporter can pick the wrong span as the model call.

## Add the Maple span processor

Without this processor the transcript has no user messages, sub-agents land in separate sessions, and raw provider responses (including cookies) are exported.

```ts
// src/mastra/maple-span-processor.ts
import { SpanType, type SpanOutputProcessor } from "@mastra/core/observability"

export const mapleSpanProcessor: SpanOutputProcessor = {
	name: "maple-span-processor",
	process(span) {
		if (!span) return span
		// One conversation id per trace: sub-agents get their own thread ids otherwise.
		let root = span
		while (root.parent) root = root.parent
		const threadId = root.metadata?.threadId
		if (threadId) span.metadata = { ...span.metadata, threadId }
		// The model call span is created without its prompt: take the step's messages.
		if (span.type === SpanType.MODEL_INFERENCE && span.input === undefined && span.parent?.input !== undefined) {
			span.input = { messages: span.parent.input }
		}
		// Step spans carry the raw provider response (headers, cookies, full body) as metadata.
		if (span.type === SpanType.MODEL_STEP && span.metadata) {
			const { headers: _headers, body: _body, ...metadata } = span.metadata
			span.metadata = metadata
		}
		return span
	},
	async shutdown() {},
}
```

## Configure the exporter

Add observability to your `Mastra` instance:

```ts
// src/mastra/index.ts
import { Mastra } from "@mastra/core/mastra"
import { SpanType } from "@mastra/core/observability"
import { Observability } from "@mastra/observability"
import { OtelExporter } from "@mastra/otel-exporter"
import { supportAgent } from "./agents/support"
import { mapleSpanProcessor } from "./maple-span-processor"

// A missing key disables export; it never stops the app.
const mapleKey = process.env.MAPLE_INGEST_KEY
if (!mapleKey) console.warn("MAPLE_INGEST_KEY is not set; Maple telemetry export is disabled")

const mapleExporter = mapleKey
	? new OtelExporter({
			provider: {
				custom: {
					endpoint: "https://ingest.maple.dev",
					protocol: "http/protobuf",
					headers: { Authorization: `Bearer ${mapleKey}` },
				},
			},
			resourceAttributes: { "deployment.environment.name": "production" },
		})
	: undefined

export const mastra = new Mastra({
	agents: { supportAgent },
	observability: new Observability({
		configs: {
			maple: {
				serviceName: "support-agent",
				exporters: mapleExporter ? [mapleExporter] : [],
				// One span per streamed chunk adds nothing Maple uses
				excludeSpanTypes: [SpanType.MODEL_CHUNK],
				spanOutputProcessors: [mapleSpanProcessor],
			},
		},
	}),
})
```

Set `MAPLE_INGEST_KEY` to your ingest key from **Settings → Ingestion**. For an EU organization, use `https://ingest.eu.maple.dev`. The exporter appends `/v1/traces` itself.

Set the endpoint, protocol and key in code, since this exporter ignores the `OTEL_EXPORTER_OTLP_*` variables. `observability` must be an `Observability` instance; a plain object silently traces nothing.

Only agents and workflows registered on this `Mastra` instance are traced. Get them with `mastra.getAgent()` or `mastra.getWorkflow()`.

## Pass the thread id on every call

Pass the same thread on every call of a conversation:

```ts
const agent = mastra.getAgent("supportAgent")

export async function handleMessage(chatId: string, userId: string, text: string) {
	const result = await agent.generate(text, {
		memory: { thread: chatId, resource: userId },
	})
	return result.text
}
```

Use the chat id your app already has. It must stay the same for the whole conversation and differ between conversations. `agent.stream()` takes the same option; read the stream to the end, since the spans are exported when it finishes.

Workflow runs, and agents called without `memory`, have no thread. Put the id in the root span's metadata instead:

```ts
const run = await mastra.getWorkflow("briefingWorkflow").createRun()
const result = await run.start({
	inputData: { request },
	tracingOptions: { metadata: { threadId: conversationId } },
})
```

Give every `Agent` a distinct `name`, or sub-agents share one lane. When a workflow step calls an agent, pass it the step's `tracingContext` (`agent.generate(prompt, { tracingContext })`) so the agent joins the workflow's trace.

## Flush in scripts and serverless functions

A short-lived process can exit before its spans are sent. In a script, call `await mastra.shutdown()` in a `finally` block before exiting. In a serverless handler, call `await mastra.observability.flush()` at the end of each request, after any streamed response has finished.

## Check that it works

Run a conversation with two messages and a tool call, then open **Agent Sessions** in Maple. You should see one session named after your thread id with framework **Mastra**, one turn per `generate()` or `stream()` call, and a transcript with the prompts, replies and tool calls.

Cost shows as unpriced because Mastra doesn't report it. If nothing arrives, set `logLevel: "debug"` on `OtelExporter` to log each export as `Export completed` or `Export FAILED` with the reason.

## Troubleshooting

- **Nothing arrives and there is no error.** `observability` is a plain object instead of `new Observability(...)`, or the agent isn't registered on the `Mastra` instance.
- **`http/protobuf exporter is not installed` at startup.** The install skipped optional dependencies. Install `@opentelemetry/exporter-trace-otlp-proto`.
- **Every message is its own session.** The call has no `memory: { thread, resource }`, or the thread id changes per request. For workflows, use `tracingOptions.metadata.threadId`.
- **The transcript has no user messages, or sub-agents land in a session named `<thread id>-<uuid>`.** Add `mapleSpanProcessor` to `spanOutputProcessors`.
- **A failed tool shows as successful.** The tool returned an error value. Throw an `Error` instead.

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview)
- [Mastra: OpenTelemetry exporter](https://mastra.ai/docs/observability/tracing/exporters/otel)

---

# Trace OpenAI Agents SDK runs with OpenTelemetry

> AI Frameworks · https://maple.dev/docs/agent-tracing/openai-agents

Send OpenAI Agents SDK runs to Maple as one Agent Session per conversation, with the transcript, tool calls and tokens.

import LanguageTabs from "../../../components/docs/LanguageTabs.astro"
import LanguageTab from "../../../components/docs/LanguageTab.astro"

OpenInference's OpenAI Agents bridge exports OpenAI Agents SDK runs to Maple, in Python (`openai-agents`) and TypeScript (`@openai/agents`). Pass the conversation id with every run, or every message shows up as its own session.

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing-openai-agents](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-openai-agents) skill and follows it.

```text
Set up Maple agent tracing for the OpenAI Agents SDK in this project.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing-openai-agents -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## Install the bridge

<LanguageTabs label="Language" tabs={[{ id: "python", label: "Python" }, { id: "typescript", label: "TypeScript" }]}>
<LanguageTab id="python">

```bash
pip install "openai-agents>=0.22" "openinference-instrumentation-openai-agents>=2.5" "opentelemetry-sdk>=1.45" "opentelemetry-exporter-otlp-proto-http>=1.45"
```

</LanguageTab>
<LanguageTab id="typescript">

```bash
npm install @openai/agents @arizeai/openinference-instrumentation-openai-agents @arizeai/openinference-core @opentelemetry/api @opentelemetry/sdk-node
```

</LanguageTab>
</LanguageTabs>

## Point the exporter at Maple

```bash
export OTEL_SERVICE_NAME=support-agent
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production
export OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
```

EU organizations use `https://ingest.eu.maple.dev`. The exporter appends `/v1/traces` itself. If you pass the endpoint to an exporter in code instead, give the full URL ending in `/v1/traces`.

## Register the bridge

<LanguageTabs label="Language" tabs={[{ id: "python", label: "Python" }, { id: "typescript", label: "TypeScript" }]}>
<LanguageTab id="python">

Add a `tracing.py` and import it at the top of your entry point, before the first `Runner.run`:

```py
# tracing.py
import re

from agents import set_trace_processors
from agents.tracing import TracingProcessor
from agents.tracing.span_data import GenerationSpanData, HandoffSpanData
from openinference.instrumentation import TraceConfig
from openinference.instrumentation.openai_agents import OpenAIAgentsInstrumentor
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor


def _chat_message(response: dict) -> dict:
    """The assistant message inside a Responses-shaped dict, as a Chat Completions message."""
    text, calls = "", []
    for item in response.get("output") or []:
        if item.get("type") == "message":
            text += "".join(c.get("text", "") for c in item.get("content") or [] if c.get("type") == "output_text")
        elif item.get("type") == "function_call":
            calls.append({"id": item["call_id"], "type": "function",
                          "function": {"name": item["name"], "arguments": item["arguments"]}})
    return {"role": "assistant", "content": text or None, "tool_calls": calls or None}


class MapleSpanFixes(TracingProcessor):
    """Fills two gaps in what OpenInference exports. Must run before the OpenInference processor."""

    def on_span_end(self, span):
        data = span.span_data
        current = trace.get_current_span()  # the matching OpenTelemetry span, still open here
        if isinstance(data, HandoffSpanData) and data.to_agent:
            # Handoff spans carry no tool name.
            current.set_attribute("gen_ai.tool.name", re.sub(r"[^a-zA-Z0-9_]", "_", f"transfer_to_{data.to_agent}").lower())
        elif isinstance(data, GenerationSpanData) and data.output and data.output[0].get("object") == "response":
            # Streamed Chat Completions calls record a Responses object OpenInference can't read.
            data.output = [_chat_message(data.output[0])]

    def on_trace_start(self, t): pass
    def on_trace_end(self, t): pass
    def on_span_start(self, span): pass
    def shutdown(self): pass
    def force_flush(self): pass


provider = TracerProvider()  # reads OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

# Replaces the SDK's default processor (which uploads to OpenAI) with MapleSpanFixes,
# then appends the OpenInference processor after it.
set_trace_processors([MapleSpanFixes()])
OpenAIAgentsInstrumentor().instrument(
    tracer_provider=provider,
    config=TraceConfig(enable_genai_semconv=True),
    exclusive_processor=False,
)
```

`MapleSpanFixes` fixes handoff tool names and streamed replies. It has to run before the OpenInference processor, so keep the order shown.

`set_trace_processors` already stops the upload to OpenAI. Don't use `set_tracing_disabled(True)` or `OPENAI_AGENTS_DISABLE_TRACING=1` for that, or you get no spans.

If the app already has a `TracerProvider` (from `opentelemetry-instrument`, Logfire or another library), add the OTLP exporter to it and pass it to `instrument()` instead of creating a second one.

If an agent uses `OpenAIChatCompletionsModel` with a non-OpenAI base URL (OpenRouter, LiteLLM, vLLM, Ollama), give it `model_settings=ModelSettings(include_usage=True)`. Otherwise streamed turns report zero tokens.

</LanguageTab>
<LanguageTab id="typescript">

Create an `instrumentation.ts` and import it as the first line of your entry point (`import "./instrumentation"`):

```ts
// instrumentation.ts
import * as agents from "@openai/agents"
import { OpenAIAgentsInstrumentation } from "@arizeai/openinference-instrumentation-openai-agents"
import { NodeSDK } from "@opentelemetry/sdk-node"

// Reads OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES and OTEL_EXPORTER_OTLP_*
export const sdk = new NodeSDK()
sdk.start()

// Replaces the SDK's default processor, which uploads traces to OpenAI
new OpenAIAgentsInstrumentation().manuallyInstrument(agents)
```

Don't use `setTracingDisabled(true)` or `OPENAI_AGENTS_DISABLE_TRACING=1` to stop the upload to OpenAI, or you get no spans.

If the app already starts OpenTelemetry (auto-instrumentation, Sentry, your own `NodeTracerProvider`), skip the `NodeSDK` lines and keep the `manuallyInstrument` call. The bridge sends its spans to the global tracer provider.

</LanguageTab>
</LanguageTabs>

## Group each conversation into one session

<LanguageTabs label="Language" tabs={[{ id: "python", label: "Python" }, { id: "typescript", label: "TypeScript" }]}>
<LanguageTab id="python">

Wrap every run in `using_session` with the conversation's id:

```py
from agents import RunConfig, Runner, SQLiteSession
from openinference.instrumentation import using_session


async def handle_message(conversation_id: str, text: str) -> str:
    with using_session(conversation_id):
        result = await Runner.run(
            agent,
            text,
            session=SQLiteSession(conversation_id, "chats.db"),
            run_config=RunConfig(workflow_name="support workflow"),
        )
    return result.final_output
```

Use the id your app already stores the chat under, and give the SDK session the same one. A new UUID per request gives you one session per message.

For streaming, call `Runner.run_streamed` inside the `with` block.

Keep `tool` out of `workflow_name`, or Maple counts a phantom tool call per run.

</LanguageTab>
<LanguageTab id="typescript">

Run every call inside an OpenTelemetry context that carries the conversation id as `gen_ai.conversation.id`. The bridge copies it onto every span of the run:

```ts
import { setAttributes } from "@arizeai/openinference-core"
import { run } from "@openai/agents"
import { context } from "@opentelemetry/api"

export async function handleMessage(conversationId: string, text: string) {
	const ctx = setAttributes(context.active(), { "gen_ai.conversation.id": conversationId })
	const result = await context.with(ctx, () => run(agent, text))
	return result.finalOutput
}
```

Use the id your app already stores the chat under. A new UUID per request gives you one session per message. If you pass a `session` to `run`, key it by the same id.

For streaming, call `run(agent, text, { stream: true })` inside the `context.with` callback. You can read the stream after it returns.

</LanguageTab>
</LanguageTabs>

## Flush in short-lived processes

<LanguageTabs label="Language" tabs={[{ id: "python", label: "Python" }, { id: "typescript", label: "TypeScript" }]}>
<LanguageTab id="python">

In serverless functions and notebooks, flush explicitly:

```py
from tracing import provider

try:
    asyncio.run(handle_message("conv-42", "What's the weather in Berlin?"))
finally:
    provider.force_flush()  # serverless: before returning; notebooks: after each run
```

The SDK's `flush_traces()` doesn't flush these spans.

</LanguageTab>
<LanguageTab id="typescript">

In a script, call `await sdk.shutdown()` in a `finally` block before exiting.

A serverless handler has to flush after every invocation, so create the span processor yourself:

```bash
npm install @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-proto
```

```ts
// instrumentation.ts, replacing `new NodeSDK()`
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto"
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base"

export const spanProcessor = new BatchSpanProcessor(new OTLPTraceExporter())
export const sdk = new NodeSDK({ spanProcessors: [spanProcessor] })
```

Then call `await spanProcessor.forceFlush()` in a `finally` block before the handler returns. Read streamed runs to the end first.

</LanguageTab>
</LanguageTabs>

## Check that it works

Send two or three messages with the same conversation id, one using a tool, then open **Agent Sessions**. You should see one session with one turn per run, the tool calls, and tokens on every model call. The framework shows as **OpenAI Agents SDK**. Cost shows as unpriced.

## Troubleshooting

- **No spans at all.** Tracing is disabled somewhere (`set_tracing_disabled` or `setTracingDisabled`, `OPENAI_AGENTS_DISABLE_TRACING`, or the run config), or the tracing setup ran after the first run. In TypeScript, `NODE_ENV=test` also turns tracing off; call `setTracingDisabled(false)` in tests.
- **One session per message.** The run isn't inside `using_session(...)` or the `context.with` callback, or the id changes per request. Group ids and SDK session ids don't reach Maple.
- **Streamed turns have zero tokens (Python).** Add `ModelSettings(include_usage=True)` to agents on a non-OpenAI base URL.
- **Every model call appears twice.** Another instrumentation of the OpenAI client (OpenInference, Logfire, Langfuse, `@opentelemetry/instrumentation-openai`) is also active. Keep one.

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview): what Maple builds from these spans.
- [Tracing in the OpenAI Agents SDK](https://openai.github.io/openai-agents-python/tracing/) (Python) and [for TypeScript](https://openai.github.io/openai-agents-js/guides/tracing/): the switch that turns off prompt and reply capture.
- [OpenRouter](/docs/agent-tracing/openrouter) and [LiteLLM](/docs/agent-tracing/litellm): if your models go through either gateway.

---

# Trace LangChain and LangGraph agents with OpenTelemetry

> AI Frameworks · https://maple.dev/docs/agent-tracing/langchain

Send LangChain and LangGraph runs to Maple with OpenInference, one Agent Session per thread.

import LanguageTabs from "../../../components/docs/LanguageTabs.astro"
import LanguageTab from "../../../components/docs/LanguageTab.astro"

OpenInference's LangChain instrumentation sends LangChain and LangGraph runs to Maple, in Python (`langchain`, `langgraph`) and TypeScript (`langchain`, `@langchain/langgraph`). Pass the conversation's `thread_id` on every call so a chat becomes one session.

You need Python 3.10 or later, or Node.js 20 or later with `@langchain/core` 1.x (Bun works too).

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing-langchain](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-langchain) skill and follows it.

```text
Set up Maple agent tracing for LangChain & LangGraph in this project.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing-langchain -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## Install the instrumentor

<LanguageTabs label="Language" tabs={[{ id: "python", label: "Python" }, { id: "typescript", label: "TypeScript" }]}>
<LanguageTab id="python">

```bash
pip install "langchain>=1.4" "langgraph>=1.2" "langchain-openai>=1.6" \
  "openinference-instrumentation-langchain>=0.1.76" "openinference-instrumentation>=0.1.66" \
  "opentelemetry-sdk>=1.45" "opentelemetry-exporter-otlp-proto-http>=1.45"
```

Keep the explicit `openinference-instrumentation` pin. Older versions don't work with this setup.

</LanguageTab>
<LanguageTab id="typescript">

```bash
npm install @arizeai/openinference-instrumentation-langchain @opentelemetry/sdk-node @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-proto
```

`@opentelemetry/sdk-node` has to be 0.209 or newer.

</LanguageTab>
</LanguageTabs>

## Point the exporter at Maple

```bash
export OTEL_SERVICE_NAME=support-agent
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production
export OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
```

EU organizations use `https://ingest.eu.maple.dev`. The exporter appends `/v1/traces` itself. If you pass the endpoint to the exporter in code instead, it has to end in `/v1/traces`.

## Initialize tracing

<LanguageTabs label="Language" tabs={[{ id: "python", label: "Python" }, { id: "typescript", label: "TypeScript" }]}>
<LanguageTab id="python">

Add a `tracing.py` and import it at the top of your entry point, before the first `invoke()`:

```py
# tracing.py
from openinference.instrumentation import TraceConfig
from openinference.instrumentation.langchain import LangChainInstrumentor
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import SpanProcessor, TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# The name= you gave create_agent(), and graph nodes that act as agents
AGENT_NAMES = {"assistant"}
# Your tool node
STEP_NAMES = {"tools"}


class AgentSpans(SpanProcessor):
    def on_start(self, span, parent_context=None):
        if span.instrumentation_scope.name != "openinference.instrumentation.langchain":
            return
        if span.name in AGENT_NAMES:
            span.set_attribute("gen_ai.operation.name", "invoke_agent")
            span.set_attribute("gen_ai.agent.name", span.name)
        elif span.name in STEP_NAMES:
            span.set_attribute("gen_ai.operation.name", "invoke_workflow")


provider = TracerProvider()
provider.add_span_processor(AgentSpans())
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

LangChainInstrumentor().instrument(
    tracer_provider=provider,
    config=TraceConfig(enable_genai_semconv=True),
)
```

Put every agent's `name=` in `AGENT_NAMES`, sub-agents included, so each gets its own lane. If your tool node isn't called `tools`, add its name to `STEP_NAMES`.

If the app already has a `TracerProvider` (from `opentelemetry-instrument`, Logfire or Sentry), add `AgentSpans()` and the exporter to it and pass it to `instrument()`.

</LanguageTab>
<LanguageTab id="typescript">

The TypeScript instrumentation only writes OpenInference attributes, so add a span processor that copies them into the GenAI attributes Maple reads. Save this file as `genai-spans.ts`:

```ts
// genai-spans.ts: adds the OpenTelemetry GenAI attributes Maple reads to OpenInference's LangChain spans
import type { Span, SpanProcessor } from "@opentelemetry/sdk-trace-base"

// Hand-built StateGraph agents, by compiled name. createAgent({ name }) is detected on its own.
const AGENT_NAMES = new Set<string>()

const ROLES: Record<string, string> = { human: "user", ai: "assistant" }

function parse(value: unknown): any {
	if (typeof value !== "string") return undefined
	try {
		return JSON.parse(value)
	} catch {
		return undefined
	}
}

function text(content: unknown): string {
	if (typeof content === "string") return content
	if (!Array.isArray(content)) return ""
	return content.map((block) => (block?.type === "text" ? block.text : "")).join("")
}

// LangChain messages are serialized as { lc, id: [..., "AIMessage"], kwargs }; plain inputs are { role, content }
function toGenAiMessage(message: any) {
	const fields = message?.lc ? message.kwargs : (message ?? {})
	const type = message?.lc
		? String(message.id.at(-1)).replace(/Message(Chunk)?$/, "").toLowerCase()
		: String(fields.role ?? fields.type)
	const role = ROLES[type] ?? type
	const parts: object[] = []
	if (role === "tool") {
		parts.push({ type: "tool_call_response", id: fields.tool_call_id, response: text(fields.content) })
	} else if (text(fields.content)) {
		parts.push({ type: "text", content: text(fields.content) })
	}
	for (const call of fields.tool_calls ?? []) {
		parts.push({ type: "tool_call", id: call.id, name: call.name, arguments: call.args })
	}
	return { role, parts }
}

export class GenAiSpans implements SpanProcessor {
	// Tool call arguments by call id, from the model reply that requested them
	private toolArgs = new Map<string, unknown>()

	onEnding(span: Span) {
		if (span.instrumentationScope.name !== "@arizeai/openinference-instrumentation-langchain") return
		const attrs = span.attributes
		const set = (key: string, value: unknown) => {
			if (value === undefined || value === null || value === "") return
			span.setAttribute(key, typeof value === "object" ? JSON.stringify(value) : (value as string | number))
		}
		const metadata = parse(attrs["metadata"]) ?? {}
		const input = parse(attrs["input.value"])
		const output = parse(attrs["output.value"])
		const kind = attrs["openinference.span.kind"]

		if (kind === "LLM") {
			const generations = output?.generations?.[0] ?? []
			const reply = generations[0]?.message?.kwargs ?? {}
			const finish = generations[0]?.generationInfo?.finish_reason ?? reply.response_metadata?.finish_reason
			if (this.toolArgs.size > 1000) this.toolArgs.clear() // calls whose tool never ran
			for (const call of reply.tool_calls ?? []) this.toolArgs.set(call.id, call.args)
			set("gen_ai.operation.name", "chat")
			set("gen_ai.provider.name", metadata.ls_provider)
			set("gen_ai.request.model", attrs["llm.model_name"])
			set("gen_ai.response.model", reply.response_metadata?.model_name)
			set("gen_ai.response.id", reply.id)
			set("gen_ai.usage.input_tokens", attrs["llm.token_count.prompt"])
			set("gen_ai.usage.output_tokens", attrs["llm.token_count.completion"])
			if (finish) span.setAttribute("gen_ai.response.finish_reasons", [finish])
			set("gen_ai.input.messages", (input?.messages?.[0] ?? []).map(toGenAiMessage))
			set(
				"gen_ai.output.messages",
				generations.map((g: any) => ({ ...toGenAiMessage(g.message ?? { role: "assistant", content: g.text }), finish_reason: finish ?? "stop" })),
			)
		} else if (kind === "TOOL") {
			// A tool called by the model returns a serialized ToolMessage
			const message = output?.output?.lc ? output.output.kwargs : undefined
			const content = message ? text(message.content) : attrs["output.value"]
			const result = parse(content)
			set("gen_ai.operation.name", "execute_tool")
			set("gen_ai.tool.name", attrs["tool.name"])
			set("gen_ai.tool.call.id", message?.tool_call_id)
			set("gen_ai.tool.call.arguments", this.toolArgs.get(message?.tool_call_id))
			this.toolArgs.delete(message?.tool_call_id)
			if (content !== undefined) set("gen_ai.tool.call.result", typeof result === "object" && result !== null ? result : content)
		} else if (span.name === metadata.lc_agent_name || AGENT_NAMES.has(span.name)) {
			const messages = output?.messages ?? []
			set("gen_ai.operation.name", "invoke_agent")
			set("gen_ai.agent.name", span.name)
			set("gen_ai.input.messages", (input?.messages ?? []).map(toGenAiMessage))
			set("gen_ai.output.messages", messages.slice(-1).map(toGenAiMessage))
		} else if (kind !== "RETRIEVER" && kind !== "EMBEDDING") {
			// Graph nodes, prompts and other runnables: steps, not model or tool calls
			set("gen_ai.operation.name", "invoke_workflow")
		}
	}

	onStart() {}
	onEnd() {}
	forceFlush() {
		return Promise.resolve()
	}
	shutdown() {
		return Promise.resolve()
	}
}
```

Then create an `instrumentation.ts` and import it as the first line of your entry point (`import "./instrumentation"`):

```ts
// instrumentation.ts
import { LangChainInstrumentation } from "@arizeai/openinference-instrumentation-langchain"
import * as CallbackManagerModule from "@langchain/core/callbacks/manager"
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto"
import { NodeSDK } from "@opentelemetry/sdk-node"
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base"
import { GenAiSpans } from "./genai-spans"

// Reads OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES and OTEL_EXPORTER_OTLP_*
export const spanProcessor = new BatchSpanProcessor(new OTLPTraceExporter())
export const sdk = new NodeSDK({
	spanProcessors: [new GenAiSpans(), spanProcessor],
	// Room for long chats: OpenInference writes several attributes per message
	spanLimits: { attributeCountLimit: 1000 },
})
sdk.start()

new LangChainInstrumentation().manuallyInstrument(CallbackManagerModule)
```

Keep the `manuallyInstrument()` call. Without it, nothing from LangChain is traced.

Give every `createAgent()` a `name`, sub-agents included, so each gets its own lane. For a graph you build with `StateGraph`, compile it with a name (`.compile({ name: "planner" })`) and add that name to `AGENT_NAMES`.

If the app already starts a `NodeSDK` or another tracer provider (auto-instrumentations, Sentry), add `new GenAiSpans()` and the exporter's processor to its span processors instead of starting a second SDK.

</LanguageTab>
</LanguageTabs>

## Group a conversation with thread_id

<LanguageTabs label="Language" tabs={[{ id: "python", label: "Python" }, { id: "typescript", label: "TypeScript" }]}>
<LanguageTab id="python">

Pass your app's conversation id as `thread_id` on every `invoke()`, `stream()` and `Command(resume=...)`:

```py
import tracing  # first, before the first invoke()

from langchain.agents import create_agent
from langchain_openai import ChatOpenAI
from langgraph.checkpoint.memory import InMemorySaver

agent = create_agent(
    ChatOpenAI(model="gpt-4o-mini", stream_usage=True),
    tools=[get_weather, calculate],
    name="assistant",
    checkpointer=InMemorySaver(),
)


def handle_message(conversation_id: str, text: str) -> str:
    result = agent.invoke(
        {"messages": [{"role": "user", "content": text}]},
        {"configurable": {"thread_id": conversation_id}},
    )
    return result["messages"][-1].content
```

Use the id your app stores the chat under. A new UUID per request gives you one session per message. A plain chain (`prompt | model`, no graph) ignores `configurable`, so pass `{"metadata": {"thread_id": conversation_id}}` instead.

Keep `stream_usage=True` if `ChatOpenAI` uses a custom `base_url`, vLLM or a gateway. Without it, streamed replies have no tokens.

</LanguageTab>
<LanguageTab id="typescript">

Pass your app's conversation id as `thread_id` on every `invoke()`, `stream()` and `new Command({ resume })`:

```ts
import "./instrumentation" // first, before the first invoke()

import { ChatOpenAI } from "@langchain/openai"
import { MemorySaver } from "@langchain/langgraph"
import { createAgent } from "langchain"

const agent = createAgent({
	model: new ChatOpenAI({ model: "gpt-4o-mini" }),
	tools: [getWeather, calculate],
	name: "assistant",
	checkpointer: new MemorySaver(),
})

export async function handleMessage(conversationId: string, text: string) {
	const result = await agent.invoke(
		{ messages: [{ role: "user", content: text }] },
		{ configurable: { thread_id: conversationId } },
	)
	return result.messages.at(-1)?.content
}
```

Use the id your app stores the chat under. A new UUID per request gives you one session per message. A plain chain (`prompt.pipe(model)`, no graph) ignores `configurable`, so pass `{ metadata: { thread_id: conversationId } }` instead.

</LanguageTab>
</LanguageTabs>

## Flush in short-lived processes

<LanguageTabs label="Language" tabs={[{ id: "python", label: "Python" }, { id: "typescript", label: "TypeScript" }]}>
<LanguageTab id="python">

Long-running servers need nothing. In serverless handlers, notebooks and task workers, flush after each run:

```py
from tracing import provider

try:
    handle_message("conv-42", "What's the weather in Berlin?")
finally:
    provider.force_flush()
```

On LangGraph Server (`langgraph dev` or self-hosted), import `tracing` at the top of the graph module that `langgraph.json` points to, and set the `OTEL_*` variables in the server's environment. Server threads group into sessions without extra code.

</LanguageTab>
<LanguageTab id="typescript">

Long-running servers need nothing. At the end of a script, call `await sdk.shutdown()`. In serverless handlers and queue workers, flush after each run:

```ts
import { spanProcessor } from "./instrumentation"

try {
	await handleMessage("conv-42", "What's the weather in Berlin?")
} finally {
	await spanProcessor.forceFlush()
}
```

</LanguageTab>
</LanguageTabs>

## Check that it works

Send two or three messages with the same conversation id, one of them using a tool, then open **Agent Sessions**. Within a minute you should see one session with one turn per `invoke()`, a readable transcript, model calls with tokens, and tool calls named after your tools.

Cost shows as **unpriced**, which is expected.

## Troubleshooting

- **No spans at all.** Load the tracing file before the first `invoke()`. In Python, pass `tracer_provider=provider`; in TypeScript, keep the `manuallyInstrument()` call. Then check the logs for exporter errors.
- **One session per message.** The `thread_id` is missing or changes per request (plain chains need it in `metadata`).
- **Streamed replies have no tokens.** In Python, set `stream_usage=True` on `ChatOpenAI`. In TypeScript, remove `streamUsage: false`.
- **Every model call appears twice.** Remove `LANGSMITH_OTEL_ENABLED` and any provider instrumentor such as `openinference-instrumentation-openai`. Plain `LANGSMITH_TRACING=true` is fine.
- **No agent lane, extra tool calls, or a tool shown as an agent.** In Python, fill `AGENT_NAMES` and `STEP_NAMES`, and don't put "agent" in tool names. In TypeScript, give every `createAgent()` a `name`.

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview): what Maple builds from these spans.
- [OpenRouter](/docs/agent-tracing/openrouter): if your models go through OpenRouter.
- [LangChain docs](https://docs.langchain.com)

---

# Trace Claude Agent SDK agents and Claude Code sessions with OpenTelemetry

> AI Frameworks · https://maple.dev/docs/agent-tracing/claude-agent-sdk

Turn on Claude Code's built-in OpenTelemetry so each Agent SDK conversation or Claude Code session shows up in Maple as one Agent Session.

import LanguageTabs from "../../../components/docs/LanguageTabs.astro"
import LanguageTab from "../../../components/docs/LanguageTab.astro"

The Claude Agent SDK and Claude Code export OpenTelemetry spans for each turn, model request and tool call once you set a few environment variables. There is nothing to install.

Spans only exist with `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1`, and in the SDK every `query()` starts a new session unless you resume the conversation's session id.

Tested with `@anthropic-ai/claude-agent-sdk` 0.3.283 (TypeScript), `claude-agent-sdk` 0.2.160 (Python) and Claude Code 2.1.283.

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing-claude-agent-sdk](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-claude-agent-sdk) skill and follows it.

```text
Set up Maple agent tracing for the Claude Agent SDK in this project.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing-claude-agent-sdk -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## Pass the telemetry variables to the CLI

`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` is required, because Claude Code has no default protocol. EU organizations use `https://ingest.eu.maple.dev`.

The snippets below drop any inherited `TRACEPARENT`, which Claude Code's Bash tool and most CI systems set. Otherwise your agent's turns nest inside that outer trace.

Never set an exporter to `console` in an SDK app. It breaks the SDK's message stream.

<LanguageTabs label="Language" tabs={[{ id: "typescript", label: "TypeScript" }, { id: "python", label: "Python" }]}>
<LanguageTab id="typescript">

```bash
npm install @anthropic-ai/claude-agent-sdk zod
```

In TypeScript, `options.env` replaces the child's environment, so spread `process.env` to keep `PATH` and `ANTHROPIC_API_KEY`:

```ts
// maple-env.ts: built per query() so values loaded later (dotenv) are included
let warnedNoKey = false

export function mapleEnv(): Record<string, string | undefined> {
	const env: Record<string, string | undefined> = { ...process.env }
	delete env.TRACEPARENT
	delete env.TRACESTATE
	const key = process.env.MAPLE_INGEST_KEY
	if (!key) {
		// A missing key turns telemetry off; the agent still runs.
		if (!warnedNoKey) console.warn("MAPLE_INGEST_KEY is not set; Maple telemetry export is disabled")
		warnedNoKey = true
		return env
	}
	return {
		...env,
		CLAUDE_CODE_ENABLE_TELEMETRY: "1",
		CLAUDE_CODE_ENHANCED_TELEMETRY_BETA: "1", // spans; without it there are none
		OTEL_TRACES_EXPORTER: "otlp",
		OTEL_LOGS_EXPORTER: "otlp", // optional: cost and replies, under Logs
		OTEL_METRICS_EXPORTER: "otlp", // optional: token and cost counters
		OTEL_EXPORTER_OTLP_PROTOCOL: "http/protobuf",
		OTEL_EXPORTER_OTLP_ENDPOINT: "https://ingest.maple.dev",
		OTEL_EXPORTER_OTLP_HEADERS: `Authorization=Bearer ${key}`,
		OTEL_SERVICE_NAME: "support-agent",
		OTEL_RESOURCE_ATTRIBUTES: "deployment.environment.name=production",
		OTEL_TRACES_EXPORT_INTERVAL: "1000",
		OTEL_LOGS_EXPORT_INTERVAL: "1000",
		// Content, off by default. See "Choose what content to record" below.
		OTEL_LOG_USER_PROMPTS: "1",
		OTEL_LOG_TOOL_DETAILS: "1",
		OTEL_LOG_TOOL_CONTENT: "1",
	}
}
```

</LanguageTab>
<LanguageTab id="python">

```bash
pip install claude-agent-sdk
```

In Python, `ClaudeAgentOptions.env` is merged over the inherited environment, so pass only the telemetry variables and remove `TRACEPARENT` from `os.environ`:

```py
# maple_env.py
import logging
import os

os.environ.pop("TRACEPARENT", None)
os.environ.pop("TRACESTATE", None)

_warned_no_key = False


def maple_env() -> dict[str, str]:
    """Telemetry env for the Claude Code CLI, built per query()."""
    global _warned_no_key
    key = os.environ.get("MAPLE_INGEST_KEY")
    if not key:
        # A missing key turns telemetry off; the agent still runs.
        if not _warned_no_key:
            logging.getLogger(__name__).warning("MAPLE_INGEST_KEY is not set; Maple telemetry export is disabled")
            _warned_no_key = True
        return {}
    return {
        "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
        "CLAUDE_CODE_ENHANCED_TELEMETRY_BETA": "1",  # spans; without it there are none
        "OTEL_TRACES_EXPORTER": "otlp",
        "OTEL_LOGS_EXPORTER": "otlp",  # optional: cost and replies, under Logs
        "OTEL_METRICS_EXPORTER": "otlp",  # optional: token and cost counters
        "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
        "OTEL_EXPORTER_OTLP_ENDPOINT": "https://ingest.maple.dev",
        "OTEL_EXPORTER_OTLP_HEADERS": f"Authorization=Bearer {key}",
        "OTEL_SERVICE_NAME": "support-agent",
        "OTEL_RESOURCE_ATTRIBUTES": "deployment.environment.name=production",
        "OTEL_TRACES_EXPORT_INTERVAL": "1000",
        "OTEL_LOGS_EXPORT_INTERVAL": "1000",
        # Content, off by default. See "Choose what content to record" below.
        "OTEL_LOG_USER_PROMPTS": "1",
        "OTEL_LOG_TOOL_DETAILS": "1",
        "OTEL_LOG_TOOL_CONTENT": "1",
    }
```

</LanguageTab>
</LanguageTabs>

Pass the env on every `query()` call, as in the next section. Or set the same variables in your Dockerfile or deployment manifest and skip `env`, as long as no `TRACEPARENT` is set there.

An `env` block in `~/.claude/settings.json` or the project's `.claude/settings.json` overrides `options.env`. Server apps can pass `settingSources: []` (Python `setting_sources=[]`) to skip settings files.

### Claude Code in your terminal, IDE or desktop app

Put the variables under `env` in `~/.claude/settings.json`, then start a new `claude` session:

```json
{
	"env": {
		"CLAUDE_CODE_ENABLE_TELEMETRY": "1",
		"CLAUDE_CODE_ENHANCED_TELEMETRY_BETA": "1",
		"OTEL_TRACES_EXPORTER": "otlp",
		"OTEL_LOGS_EXPORTER": "otlp",
		"OTEL_METRICS_EXPORTER": "otlp",
		"OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
		"OTEL_EXPORTER_OTLP_ENDPOINT": "https://ingest.maple.dev",
		"OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer YOUR_INGEST_KEY",
		"OTEL_LOG_USER_PROMPTS": "1",
		"OTEL_LOG_TOOL_DETAILS": "1",
		"OTEL_LOG_TOOL_CONTENT": "1"
	}
}
```

A repository's `.claude/settings.json` can't set these variables. Use your user settings, your shell or [managed settings](https://code.claude.com/docs/en/managed-settings). Terminal sessions report the service `claude-code`.

## Resume the session on every turn

A chat backend that calls `query()` once per message without resuming gets one Maple session per message, and the agent forgets the previous message.

Store a UUID with each conversation. Pass it as `sessionId` on the first turn and as `resume` on every turn after:

<LanguageTabs label="Language" tabs={[{ id: "typescript", label: "TypeScript" }, { id: "python", label: "Python" }]}>
<LanguageTab id="typescript">

```ts
import { randomUUID } from "node:crypto"
import { query } from "@anthropic-ai/claude-agent-sdk"
import { mapleEnv } from "./maple-env"

type Conversation = { claudeSessionId?: string }

export async function reply(conversation: Conversation, text: string) {
	const firstTurn = !conversation.claudeSessionId
	const sessionId = conversation.claudeSessionId ?? randomUUID()
	conversation.claudeSessionId = sessionId

	for await (const message of query({
		prompt: text,
		options: { env: mapleEnv(), ...(firstTurn ? { sessionId } : { resume: sessionId }) },
	})) {
		if (message.type === "result") return message.subtype === "success" ? message.result : undefined
	}
}
```

</LanguageTab>
<LanguageTab id="python">

```py
import uuid
from claude_agent_sdk import ClaudeAgentOptions, ResultMessage, query
from maple_env import maple_env


async def reply(conversation: dict, text: str) -> str | None:
    first_turn = "claude_session_id" not in conversation
    session_id = conversation.setdefault("claude_session_id", str(uuid.uuid4()))
    session = {"session_id": session_id} if first_turn else {"resume": session_id}

    async for message in query(prompt=text, options=ClaudeAgentOptions(env=maple_env(), **session)):
        if isinstance(message, ResultMessage):
            return message.result
    return None
```

</LanguageTab>
</LanguageTabs>

`resume` reads the earlier turns from `~/.claude/projects/` on the same machine. If messages can land on different hosts, use the SDK's [`sessionStore`](https://code.claude.com/docs/en/agent-sdk/session-storage) option. A Python `ClaudeSDKClient`, or a TypeScript `query()` fed an async iterable, keeps one session for all its turns and needs none of this.

## Choose what content to record

Claude Code redacts content by default. `OTEL_LOG_USER_PROMPTS=1` records prompts, which title each turn. `OTEL_LOG_TOOL_DETAILS=1` records Bash commands, file paths and tool error messages. `OTEL_LOG_TOOL_CONTENT=1` records tool results, including any secrets in files Claude reads or in command output. Turn on only what your Maple organization is allowed to store.

## Let each turn finish exporting

Let every `query()` loop reach its `result` message. Breaking out early, calling `close()` or aborting kills the CLI before it exports the turn. In a script, keep the process alive about 5 seconds after the last `query()`. On serverless platforms, finish the loop before returning the response.

## Check that it works

Run a two-turn conversation through `reply()` with at least one tool call, then open **Agent Sessions**. You should see one session with the framework **Claude Agent SDK**, one turn per message titled with the prompt, model calls with their tokens, and the tool calls by name.

The transcript has no assistant replies and cost shows as unpriced. Both are on log events under **Logs** when the logs exporter is on.

## Troubleshooting

- **Metrics and logs arrive, but no sessions.** Set `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1` and `OTEL_TRACES_EXPORTER=otlp`.
- **Nothing arrives at all.** `OTEL_EXPORTER_OTLP_PROTOCOL` is unset or `grpc`. Set `http/protobuf`. `CLAUDE_CODE_OTEL_DIAG_STDERR=1` prints export errors to the SDK's `stderr` callback.
- **Every message is its own session.** Resume the conversation's session id as shown above, and don't set `forkSession`.
- **Your agent's turns appear inside another trace.** The process inherited a `TRACEPARENT`. Drop it and `TRACESTATE` from the environment.
- **The CLI ignores your values.** They are in a repository's `.claude/settings.json`, or a settings file overrides `options.env`. Use `~/.claude/settings.json`, or pass `settingSources: []`.
- **Extra turns titled `<task-notification>`.** Claude Code ran sub-agents in the background. Add `CLAUDE_CODE_DISABLE_BACKGROUND_TASKS: "1"` to the env.

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview): what Maple builds from these spans.
- [Claude API & SDK](/docs/agent-tracing/claude-api): tracing direct calls with the Claude SDK instead of the Agent SDK.
- Claude Code [Monitoring reference](https://code.claude.com/docs/en/monitoring-usage): every variable, span attribute and event.

---

# Trace Claude API and Claude SDK calls with OpenTelemetry

> AI Frameworks · https://maple.dev/docs/agent-tracing/claude-api

Turn on the Claude SDK's built-in OpenTelemetry tracing, or add Maple as a destination in the Claude Console, so each conversation shows up in Maple as one Agent Session.

The Claude SDK and the Claude API export OpenTelemetry spans (beta). SDK tracing runs in your process and nests Claude's spans under your own. API tracing runs at Anthropic once you add a destination in the Claude Console. Start with the SDK: it carries the conversation id that Agent Sessions groups by.

Pass a `conversationId` with every request, or each request becomes a session of its own.

Tested with `@anthropic-ai/sdk` 0.135.0. For the Claude Agent SDK or Claude Code, use the [Claude Agent SDK guide](/docs/agent-tracing/claude-agent-sdk).

## Set up SDK tracing

Install the Claude SDK and the OpenTelemetry Node SDK:

```bash
npm install @anthropic-ai/sdk@0.135.0 @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-proto
```

Pin the Claude SDK version. Span names and attributes can change between releases during the beta.

Register a tracer provider before the first Claude call:

```ts
// instrumentation.ts
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto"
import { NodeSDK } from "@opentelemetry/sdk-node"

// Reads OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS.
export const sdk = new NodeSDK({
	serviceName: "support-agent",
	traceExporter: new OTLPTraceExporter(),
})

sdk.start()
```

Point the exporter at Maple with a private ingest key (`maple_sk_…`) from **Settings → Ingestion**:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"   # https://ingest.eu.maple.dev for EU organizations
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer maple_sk_YOUR_INGEST_KEY"
```

## Send a conversation id with every request

Maple groups spans into a session by `gen_ai.conversation.id`, which the SDK sets from the `conversationId` request option. The id is exported in every content mode, so use an opaque one, never an email address or a name.

```ts
// agent.ts
import { sdk } from "./instrumentation.ts"

import Anthropic from "@anthropic-ai/sdk"
import { betaTool } from "@anthropic-ai/sdk/helpers/beta/json-schema"
import { trace } from "@opentelemetry/api"

const tracer = trace.getTracer("support-agent")

// "content" puts prompts, replies and tool arguments on the spans.
const client = new Anthropic({ openTelemetry: { traces: { contentMode: "content" } } })

const lookupOrder = betaTool({
	name: "lookup_order",
	description: "Look up an order by its id. Returns status and total.",
	inputSchema: {
		type: "object",
		properties: { order_id: { type: "string" } },
		required: ["order_id"],
	},
	run: ({ order_id }) => JSON.stringify({ order_id, status: "shipped", total: 89.5 }),
})

async function answer(conversationId: string, question: string) {
	return tracer.startActiveSpan("support.answer", async (span) => {
		try {
			const message = await client.beta.messages
				.toolRunner(
					{
						model: "claude-sonnet-5-5",
						max_tokens: 1024,
						system: "You are the support agent for an online shop. Use the tools to answer.",
						tools: [lookupOrder],
						messages: [{ role: "user", content: question }],
					},
					{ openTelemetry: { conversationId } },
				)
				.runUntilDone()
			return message.content
		} finally {
			span.end()
		}
	})
}

console.log(await answer("conv_7f3a", "Where is my order A-1042?"))

// Flush buffered spans before the process exits.
await sdk.shutdown()
```

Use `runUntilDone()` or `await` the runner. `runner.done()` resolves only while something else iterates the runner, so a script that calls it hangs and exports nothing.

## Choose what content to record

`contentMode: "content"` records the system prompt, messages, replies, and tool arguments and results, which the transcript needs. The default, `metadata_only`, records ids, timing, status and token counts, and the rest of the session view still works. Turn on content only if your Maple organization is allowed to store it.

The SDK keeps up to 0.75 MB of content per span and drops the oldest messages beyond that. Raise `maxContentBytes` for long contexts.

## Add API tracing in the Claude Console

API tracing adds what only the API sees: each model run inside a request, server tools such as web search, MCP connector tools and Managed Agents turns. It needs the Console admin role and covers the Claude Platform, including Claude Platform on AWS. Calls through Amazon Bedrock, Google Cloud or Microsoft Foundry are traced by the SDK only.

In the Console, open **Settings → OpenTelemetry**, click **Add destination** and pick the **Custom** provider:

- **OTLP endpoint URL:** `https://ingest.maple.dev`, or `https://ingest.eu.maple.dev` for EU organizations, without `/v1/traces`.
- **Request headers:** `Authorization` with the value `Bearer maple_sk_…`.
- **Content mode:** **Full content** for transcripts.
- **Workspaces:** the workspaces whose traffic you want in Maple.

<figure class="shot">
  <img src="/screenshots/docs/claude-api-01-console-destination.webp" alt="The Claude Console's Add destination form filled in for Maple: provider Custom, endpoint https://ingest.maple.dev, an Authorization header and Full content selected." loading="lazy" />
</figure>

Save, then click **Test** in the destination's **More actions** menu. The API sends one `anthropic.test` span with the service `claude-platform`, which you can find in **Explore → Traces**.

With SDK tracing on as well, each request is one trace. The SDK sends a `traceparent` header and the API parents its spans under the SDK's request span:

```text
support.answer                              your span           service: support-agent
└─ anthropic.messages.tool_runner           SDK, INTERNAL       service: support-agent
   ├─ anthropic.messages.create             SDK, CLIENT         service: support-agent
   │  └─ anthropic.messages.create          API, SERVER         service: claude-platform
   │     ├─ anthropic.model_request         API                 service: claude-platform
   │     └─ anthropic.tool_use web_search   API                 service: claude-platform
   └─ anthropic.custom_tool_use lookup_order  SDK, INTERNAL     service: support-agent
```

## Check that it works

Run a two-turn conversation with one tool call, then open **Agent Sessions**. You should see one session with one trace per turn, the model calls with their tokens, and the tool call by name. The framework shows as **Unidentified**.

<figure class="shot">
  <img src="/screenshots/docs/claude-api-02-traces.webp" alt="The Traces tab of a Maple Agent Session: two turns, each with anthropic.messages.tool_runner, two anthropic.messages.create calls with token counts, and a lookup_order tool span." loading="lazy" />
</figure>

Cost shows as unpriced, because Claude's spans carry token counts without a price. Anthropic's [Usage and Cost API](https://platform.claude.com/docs/en/manage-claude/usage-cost-api) has spend per API key.

## Troubleshooting

- **No spans at all.** Register the tracer provider before the first Claude call, and check that `ANTHROPIC_OPEN_TELEMETRY` isn't `false`.
- **Every request is its own session.** Pass `conversationId` on each request.
- **The last spans of a script are missing.** Call `sdk.shutdown()` before the process exits.
- **API spans don't nest under your spans, or tokens are counted twice.** Register the W3C trace context propagator (`NodeSDK` does) and keep the client's `propagation` option on. With `@opentelemetry/auto-instrumentations-node`, set `propagation: false` on the Claude client instead: its undici instrumentation adds a second `traceparent` header.
- **The Console shows Failing with 401.** The `Authorization` header is wrong, or the key belongs to the other region.
- **The transcript is empty.** The client or the destination is in metadata-only mode.

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview): what Maple builds from these spans.
- [Claude Agent SDK & Claude Code](/docs/agent-tracing/claude-agent-sdk): tracing agents built on the Agent SDK.
- Anthropic's [OpenTelemetry guide](https://platform.claude.com/docs/en/manage-claude/opentelemetry) and [span reference](https://platform.claude.com/docs/en/api/opentelemetry/spans).

---

# Trace Cloudflare Agents with OpenTelemetry

> AI Frameworks · https://maple.dev/docs/agent-tracing/cloudflare-agents

Export the AI SDK spans of a Cloudflare Agents SDK agent from its Durable Object to Maple and group each chat into one Agent Session.

Agents built with the Cloudflare Agents SDK (`AIChatAgent` or `Agent`) usually call models through the Vercel AI SDK, which emits the spans Maple reads. This guide sends those spans from the agent's Durable Object to Maple and passes the agent's instance name as the conversation id.

The Workers runtime can't run the Node.js OpenTelemetry SDK, so you create a small tracer provider that exports over `fetch` and flush it at the end of every turn. It works on the Workers Free and Paid plans.

You need `ai` 7.0.106 or newer and the `nodejs_compat` compatibility flag, which Agents SDK projects already have.

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing-cloudflare-agents](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-cloudflare-agents) skill and follows it.

```text
Set up Maple agent tracing for the Cloudflare Agents SDK in this project.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing-cloudflare-agents -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## Install the packages

```bash
npm install ai@^7.0.106 @ai-sdk/otel @opentelemetry/api @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-http @opentelemetry/resources @opentelemetry/context-async-hooks
```

What each package does:

- `ai` and `@ai-sdk/otel`: record each turn, model call and tool call.
- `@opentelemetry/sdk-trace-base`: collects those records and sends them in batches.
- `@opentelemetry/exporter-trace-otlp-http`: delivers them to Maple.
- `@opentelemetry/resources`: puts your service name on them.
- `@opentelemetry/api` and `@opentelemetry/context-async-hooks`: keep agents that a tool calls in the same conversation.

## Store the ingest key

Save the key as a secret so it isn't in your Wrangler config:

```bash
npx wrangler secret put MAPLE_INGEST_KEY
```

For `wrangler dev`, add `MAPLE_INGEST_KEY=YOUR_INGEST_KEY` to `.dev.vars`.

## Create the tracer provider

Create a `telemetry.ts`. The agent code below imports `tracerProvider` from it, which also registers the AI SDK integration:

```ts
// telemetry.ts
import { OpenTelemetry } from "@ai-sdk/otel"
import { context } from "@opentelemetry/api"
import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks"
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"
import { resourceFromAttributes } from "@opentelemetry/resources"
import { BasicTracerProvider, BatchSpanProcessor } from "@opentelemetry/sdk-trace-base"
import { registerTelemetry } from "ai"
import { env } from "cloudflare:workers"

context.setGlobalContextManager(new AsyncLocalStorageContextManager().enable())

// A missing key disables export; it never stops the Worker.
if (!env.MAPLE_INGEST_KEY) console.warn("MAPLE_INGEST_KEY is not set; Maple telemetry export is disabled")

export const tracerProvider = new BasicTracerProvider({
	resource: resourceFromAttributes({
		"service.name": "support-agent",
		"deployment.environment.name": "production",
	}),
	spanProcessors: env.MAPLE_INGEST_KEY
		? [
				new BatchSpanProcessor(
					new OTLPTraceExporter({
						url: "https://ingest.maple.dev/v1/traces", // EU: https://ingest.eu.maple.dev/v1/traces
						headers: { authorization: `Bearer ${env.MAPLE_INGEST_KEY}` },
					}),
				),
			]
		: [],
})

registerTelemetry(
	new OpenTelemetry({
		tracer: tracerProvider.getTracer("gen_ai"),
		usage: true,
		runtimeContext: true,
	}),
)
```

The context manager keeps sub-agents called from a tool inside the same trace.

## Pass the conversation id and flush each turn

Each chat is one instance of your agent, so its instance name (`this.name`) is the conversation id. Pass it on every AI SDK call, and flush the tracer provider when the turn ends. A Durable Object can be evicted between messages, and spans still in the buffer are lost with it.

In an `AIChatAgent`, flush in `onChatResponse`, which runs after every turn:

```ts
// server.ts
import { AIChatAgent } from "@cloudflare/ai-chat"
import { convertToModelMessages, streamText } from "ai"
import { tracerProvider } from "./telemetry"

export class ChatAgent extends AIChatAgent<Env> {
	async onChatMessage() {
		const result = streamText({
			model,
			messages: await convertToModelMessages(this.messages),
			tools,
			runtimeContext: { conversationId: this.name },
			telemetry: { functionId: "support_agent", includeRuntimeContext: { conversationId: true } },
		})
		return result.toUIMessageStreamResponse()
	}

	async onChatResponse() {
		await tracerProvider.forceFlush()
	}
}
```

In a plain `Agent`, flush when the method that called the model returns:

```ts
import { Agent } from "agents"
import { generateText } from "ai"
import { tracerProvider } from "./telemetry"

export class TaskAgent extends Agent<Env> {
	async onRequest(request: Request) {
		const { prompt } = await request.json<{ prompt: string }>()
		try {
			const result = await generateText({
				model,
				prompt,
				tools,
				runtimeContext: { conversationId: this.name },
				telemetry: { functionId: "task_agent", includeRuntimeContext: { conversationId: true } },
			})
			return Response.json({ text: result.text })
		} finally {
			this.ctx.waitUntil(tracerProvider.forceFlush())
		}
	}
}
```

If the method returns a `streamText` stream instead, flush once the stream has been read, with `this.ctx.waitUntil(result.consumeStream().then(() => tracerProvider.forceFlush()))` before returning the response.

Give each chat its own instance: on the client, pass a chat id as `name` to `useAgent({ agent: "ChatAgent", name: chatId })`. Without a `name`, every client connects to the `default` instance and all chats land in one session. `functionId` becomes the agent name in Maple, so give each agent a distinct one.

To keep a call's prompts and replies out of Maple, set `recordInputs: false` and `recordOutputs: false` in its `telemetry`.

## Check that it works

Run `wrangler dev`, send two messages in one chat that trigger a tool call, then open **Agent Sessions** in Maple. You should see one session named after the agent instance with framework **Vercel AI SDK**, one turn per message, and a transcript with the prompts, replies and tool calls. A second chat should show up as a second session.

## Troubleshooting

- **No AI spans at all.** Nothing imports `telemetry.ts`, or the `MAPLE_INGEST_KEY` secret isn't set (`npx wrangler secret list`).
- **Turns are missing or arrive late.** The flush isn't running. Check that `onChatResponse` (or your `finally` block) calls `forceFlush()`.
- **All chats are one session.** The client connects without a `name`, so every chat uses the `default` instance.
- **A sub-agent shows up as its own session.** The context manager isn't registered. Keep the `setGlobalContextManager` line in `telemetry.ts`.
- **Every span shows up twice.** `registerTelemetry()` ran twice, for example from two entry points that both set it up.

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview)
- [Vercel AI SDK guide](/docs/agent-tracing/vercel-ai-sdk)
- [Cloudflare Workers instrumentation](/docs/guides/instrumentation-cloudflare-workers) for request and binding spans
- [Cloudflare Agents SDK](https://developers.cloudflare.com/agents/)

---

# Trace Genkit agents with OpenTelemetry

> AI Frameworks · https://maple.dev/docs/agent-tracing/genkit

Send Genkit's OpenTelemetry spans to Maple and group each chat into one Agent Session.

Genkit traces every flow, model call and tool call with OpenTelemetry, but it records them under its own `genkit:*` attributes, which Agent Sessions doesn't read. You export those spans to Maple and add a small span processor that copies them to the GenAI attributes Maple reads. You also pass a conversation id in each flow, or each message becomes its own session.

This guide covers Genkit for Node.js. You need `genkit` 1.22 or newer and Node.js 20 or newer.

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing-genkit](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-genkit) skill and follows it.

```text
Set up Maple agent tracing for Genkit in this project.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing-genkit -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## Install the packages

```bash
npm install genkit @opentelemetry/sdk-node @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-proto
```

## Point the exporter at Maple

```bash
export OTEL_SERVICE_NAME="support-agent"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
```

For an EU organization, use `https://ingest.eu.maple.dev`. The exporter appends `/v1/traces` itself.

## Add the span processor

Create `genkit-for-maple.ts` and copy it as is. It turns each flow into an `invoke_agent` span, each model call into a `chat` span with its messages and token counts, and each tool call into an `execute_tool` span:

```ts
// genkit-for-maple.ts
import type { ReadableSpan, SpanProcessor } from "@opentelemetry/sdk-trace-base"

type Part = {
	text?: string
	reasoning?: string
	toolRequest?: { name: string; ref?: string; input?: unknown }
	toolResponse?: { name: string; ref?: string; output?: unknown }
}
type Message = { role: string; content: Part[] }

// Genkit message parts to OpenTelemetry GenAI parts. Media parts are left out.
function toPart({ text, reasoning, toolRequest, toolResponse }: Part) {
	if (text !== undefined) return { type: "text", content: text }
	if (reasoning !== undefined) return { type: "reasoning", content: reasoning }
	if (toolRequest) {
		return { type: "tool_call", id: toolRequest.ref, name: toolRequest.name, arguments: toolRequest.input }
	}
	if (toolResponse) return { type: "tool_call_response", id: toolResponse.ref, response: toolResponse.output }
	return undefined
}

const toParts = (content: Part[]) => content.map(toPart).filter((part) => part !== undefined)

function toMessages(messages: Message[]) {
	return messages.map((m) => ({ role: m.role === "model" ? "assistant" : m.role, parts: toParts(m.content) }))
}

/** Adds the gen_ai.* attributes Maple reads to Genkit's flow, model and tool spans. */
export class GenkitForMaple implements SpanProcessor {
	onStart() {}

	onEnd(span: ReadableSpan) {
		const attrs = span.attributes
		const json = (key: string) => {
			const value = attrs[key]
			return typeof value === "string" ? JSON.parse(value) : undefined
		}
		const name = String(attrs["genkit:name"])

		switch (attrs["genkit:metadata:subtype"]) {
			case "flow":
			case "agent": {
				Object.assign(attrs, {
					"gen_ai.operation.name": "invoke_agent",
					"gen_ai.agent.name": name,
				})
				// Set in your flow, or by Genkit for defineAgent() chats
				const conversationId = attrs["genkit:metadata:conversationId"] ?? attrs["genkit:metadata:agent:sessionId"]
				if (conversationId !== undefined) attrs["gen_ai.conversation.id"] = conversationId
				break
			}
			case "model": {
				const input = json("genkit:input")
				const output = json("genkit:output")
				const [provider, ...model] = name.split("/")
				const messages: Message[] = input?.messages ?? []
				const system = messages.filter((m) => m.role === "system").flatMap((m) => toParts(m.content))
				Object.assign(attrs, {
					"gen_ai.operation.name": "chat",
					"gen_ai.provider.name": provider,
					"gen_ai.request.model": model.join("/") || name,
					"gen_ai.input.messages": JSON.stringify(toMessages(messages.filter((m) => m.role !== "system"))),
				})
				if (system.length > 0) attrs["gen_ai.system_instructions"] = JSON.stringify(system)
				if (output?.message) {
					attrs["gen_ai.output.messages"] = JSON.stringify(
						toMessages([output.message]).map((m) => ({ ...m, finish_reason: output.finishReason })),
					)
				}
				if (output?.finishReason) attrs["gen_ai.response.finish_reasons"] = [output.finishReason]
				if (output?.usage?.inputTokens !== undefined) attrs["gen_ai.usage.input_tokens"] = output.usage.inputTokens
				if (output?.usage?.outputTokens !== undefined) attrs["gen_ai.usage.output_tokens"] = output.usage.outputTokens
				break
			}
			case "tool": {
				Object.assign(attrs, {
					"gen_ai.operation.name": "execute_tool",
					"gen_ai.tool.name": name,
					"gen_ai.tool.call.arguments": attrs["genkit:input"] ?? "{}",
				})
				const result = json("genkit:output")
				if (result !== undefined) {
					attrs["gen_ai.tool.call.result"] = typeof result === "string" ? result : JSON.stringify(result)
				}
				break
			}
		}
	}

	forceFlush() {
		return Promise.resolve()
	}

	shutdown() {
		return Promise.resolve()
	}
}
```

## Start OpenTelemetry

Create an `instrumentation.ts` and import it as the first line of your entry point (`import "./instrumentation"`):

```ts
// instrumentation.ts
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto"
import { NodeSDK } from "@opentelemetry/sdk-node"
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base"
import { disableGenkitOTelInitialization } from "genkit/tracing"
import { GenkitForMaple } from "./genkit-for-maple"

export const spanProcessor = new BatchSpanProcessor(new OTLPTraceExporter())

// Reads OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES and OTEL_EXPORTER_OTLP_*
export const sdk = new NodeSDK({ spanProcessors: [new GenkitForMaple(), spanProcessor] })

// `genkit start` sets GENKIT_ENV=dev: leave the Developer UI's own tracing alone there
if (process.env.GENKIT_ENV !== "dev") {
	disableGenkitOTelInitialization()
	sdk.start()
}
```

Keep `GenkitForMaple` before the exporting processor. `disableGenkitOTelInitialization()` stops Genkit from starting its own OpenTelemetry SDK, which can't export to Maple. It also turns off `enableFirebaseTelemetry()` and `enableGoogleCloudTelemetry()`, so traces stop going to Google Cloud.

Runs under `genkit start` keep their traces in the Developer UI and send nothing to Maple. To trace them in Maple too, remove the `if`, and the Developer UI shows no traces.

If your app already starts OpenTelemetry (Sentry, auto-instrumentation, your own `NodeTracerProvider`), add both processors to that provider instead of creating a `NodeSDK`, and keep the `disableGenkitOTelInitialization()` call.

## Pass the conversation id in each flow

Run each user message through a flow and call `setCustomMetadataAttribute("conversationId", ...)` at its start, with the chat or thread id your app already stores:

```ts
import { genkit, z, type MessageData } from "genkit"
import { setCustomMetadataAttribute } from "genkit/tracing"

const ai = genkit({ plugins: [/* your model plugin */], model: "googleai/gemini-2.5-flash" })

// One history per conversation. Store it in your database in a real backend.
const histories = new Map<string, MessageData[]>()

export const supportChat = ai.defineFlow(
	{ name: "supportChat", inputSchema: z.object({ chatId: z.string(), text: z.string() }), outputSchema: z.string() },
	async ({ chatId, text }) => {
		setCustomMetadataAttribute("conversationId", chatId)

		const response = await ai.generate({ messages: histories.get(chatId) ?? [], prompt: text, tools: [getWeather] })
		histories.set(chatId, response.messages)
		return response.text
	},
)
```

The id must stay the same for the whole conversation and differ between conversations. The flow name becomes the agent name in Maple, so give each agent its own flow.

Chats with an agent from `ai.defineAgent()` (in `genkit/beta`) already carry Genkit's session id, so they need no extra call.

## Flush before the process exits

`BatchSpanProcessor` exports every few seconds, so a short-lived process can exit before its last spans are sent. In a script, call `await sdk.shutdown()` in a `finally` block before exiting. In a serverless handler, call `await spanProcessor.forceFlush()` after the flow returns. In a long-running server, call `sdk.shutdown()` on `SIGTERM`.

## Check that it works

Run a conversation with two messages and a tool call, then open **Agent Sessions** in Maple. You should see one session named after your conversation id, one turn per flow run, and a transcript with the prompts, replies and tool calls. Each model call shows its token counts.

The framework shows as **Genkit**. Cost shows as unpriced because Genkit doesn't report it.

## Troubleshooting

- **No spans at all.** `instrumentation.ts` isn't the first import, or the app runs under `genkit start`.
- **Traces show up, but Agent Sessions is empty.** `GenkitForMaple` is missing from `spanProcessors`.
- **Every message is its own session.** The flow doesn't call `setCustomMetadataAttribute("conversationId", ...)`, or `ai.generate()` runs outside a flow.
- **The Developer UI shows no traces.** `disableGenkitOTelInitialization()` ran under `genkit start`. Keep the `GENKIT_ENV` check around it.

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview)
- [Genkit observability](https://genkit.dev/docs/observability/getting-started/)

---

# Trace eve agents with OpenTelemetry

> AI Frameworks · https://maple.dev/docs/agent-tracing/eve

Send your eve agent's traces to Maple and see each conversation as one Agent Session.

Send your eve agent's traces to Maple and see each conversation as an Agent Session, with its messages, model calls, tool calls, subagents and token usage. The setup is one package and two files.

To see message text from a deployed agent, you also [mark its conversations public](#show-message-text-in-production).

You need `eve` 0.62 or newer and Node.js 24 or newer. On eve 0.61 or older, [upgrade first](#upgrade-from-eve-061-or-older).

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It loads the [maple-agent-tracing-eve](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-eve) skill and follows it.

```text
Set up Maple agent tracing for eve in this project.

Load the skill with `npx skills use MapleTechLabs/maple/skills --skill maple-agent-tracing-eve`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## Install the exporter

```bash
npm install @vercel/otel
```

## Send traces to Maple

Create `agent/instrumentation/maple.ts`:

```ts
// agent/instrumentation/maple.ts
import { OTLPHttpProtoTraceExporter } from "@vercel/otel"
import { disableInstrumentation } from "eve/instrumentation"
import { otelIntegration } from "eve/instrumentation/otel"

// A missing key disables export; it never stops the agent.
const mapleKey = process.env.MAPLE_INGEST_KEY
if (!mapleKey) console.warn("MAPLE_INGEST_KEY is not set; Maple telemetry export is disabled")

export default mapleKey
	? otelIntegration({
			traceExporter: new OTLPHttpProtoTraceExporter({
				url: "https://ingest.maple.dev/v1/traces", // EU: https://ingest.eu.maple.dev/v1/traces
				headers: { authorization: `Bearer ${mapleKey}` },
			}),
			// Leave out eve's internal workflow spans
			exportPolicy: { span: ({ attributes }) => ({ emit: "gen_ai.conversation.id" in attributes }) },
		})
	: disableInstrumentation()
```

In deployments, set `MAPLE_INGEST_KEY` in your host's environment variables. For local runs, see [Check that it works](#check-that-it-works).

Keep the `exportPolicy` line. Without it, eve's internal workflow spans make up most of the data you send.

## Name the service

Without this step, the service in Maple is named after your `package.json`. To choose the name and add the environment, create `agent/instrumentation/otel.ts`:

```ts
// agent/instrumentation/otel.ts
import { otel } from "eve/instrumentation/otel"

export default otel({
	resource: {
		"service.name": "support-agent",
		// Outside Vercel, use your host's environment variable
		"deployment.environment.name": process.env.VERCEL_ENV ?? "development",
	},
})
```

If the project already has an `otel()` file, add the two attributes to it instead. eve allows only one.

## Show message text in production

In `eve dev`, Maple shows every message, reply and tool call. A deployed agent sends that text only for conversations eve treats as public. Other conversations still appear with their turns, model calls, tools and tokens, but without any text.

eve's Slack channel treats public channels as public, and DMs and private channels as private. Its HTTP channel treats no conversation as public until you mark it:

```ts
// agent/channels/eve.ts
import { localDev, vercelOidc } from "eve/channels/auth"
import { eveChannel } from "eve/channels/eve"

export default eveChannel({
	auth: [vercelOidc(), localDev()],
	audience: "public",
})
```

Keep the `auth` list your project already uses. A public channel sends its message text to every trace destination, including Vercel Agent Runs, so mark it public only if storing those conversations is acceptable. For other channels, see [eve's audience docs](https://eve.dev/docs/channels/overview#audience).

## Check that it works

1. Copy the public ingest key (`maple_pk_…`) from **Settings → Ingestion** ([US](https://app.maple.dev/settings?tab=ingestion) or [EU](https://app.eu.maple.dev/settings?tab=ingestion)) and add it to `.env.local` in your project:

   ```env
   MAPLE_INGEST_KEY=maple_pk_...
   ```

2. Start the agent with `eve dev` and send it two messages, one of which uses a tool. eve sends the traces to Maple a few seconds after each reply.
3. Open **Agent Sessions** in Maple ([US](https://app.maple.dev/agent-sessions) or [EU](https://app.eu.maple.dev/agent-sessions)).

You should see one session for the conversation with framework **eve**, titled with its first message, with one turn per message and the messages, replies and tool calls in its transcript. Subagents show up as separate lanes inside the turn that called them.

Cost shows for models that run through Vercel AI Gateway. With a model called directly at its provider, cost shows as unpriced because eve doesn't report it.

## Upgrade from eve 0.61 or older

eve 0.61 and older ignore the files in this guide without showing an error. Upgrade eve:

```bash
npm install eve@latest
```

Then:

1. Delete `agent/instrumentation.ts`. Newer eve versions refuse to build while it exists.
2. If `agent.ts` sets `experimental: { instrumentationProviders: true }`, remove it.
3. Add the two files from this guide.
4. If the old file did more than send traces to Maple, move each part:

| In the old `agent/instrumentation.ts` | After the upgrade |
| --- | --- |
| Another exporter in `registerOTel({ traceExporter })` | Its own file with `otelIntegration({ traceExporter })` |
| `serviceName` | `service.name` in [`otel.ts`](#name-the-service) |
| `functionId` | `otel({ functionId })` |
| `recordInputs`, `recordOutputs` | Not needed: message text depends on [whether the conversation is public](#show-message-text-in-production) |
| `events["step.started"]` returning `runtimeContext` | `otelIntegration({ runtimeContext })` |

## Troubleshooting

- **Nothing arrives.** eve doesn't report failed exports, so check the key. The startup log must not say `MAPLE_INGEST_KEY is not set`. Then send this request with your key in place of `YOUR_INGEST_KEY`. `200` means the key works; `401` says what is wrong, including a key from the other region:

  ```bash
  curl -s -w "%{http_code}\n" -X POST https://ingest.maple.dev/v1/traces \
    -H "Authorization: Bearer YOUR_INGEST_KEY" \
    -H "Content-Type: application/json" -d '{"resourceSpans":[]}'
  ```

- **Sessions have no message text.** The conversation isn't public. [Mark the channel public](#show-message-text-in-production). Setting a `tracePolicy` in `otel()` doesn't change this.
- **Thousands of `workflow.*`, `world.*` and `step.*` spans.** The `exportPolicy` line is missing from `maple.ts`.
- **A failed tool shows as successful.** The tool returned an error instead of throwing one. Throw from `execute`.

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview)
- [eve: OpenTelemetry](https://eve.dev/docs/observability/otel)

---

# Trace Pydantic AI agents with OpenTelemetry

> AI Frameworks · https://maple.dev/docs/agent-tracing/pydantic-ai

Send Pydantic AI's built-in OpenTelemetry spans to Maple so each conversation shows up as one Agent Session.

Pydantic AI already emits OpenTelemetry spans for runs, model calls and tool calls. You export them to Maple and pass a conversation id on every run. Without the id, each message becomes its own session.

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing-pydantic-ai](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-pydantic-ai) skill and follows it.

```text
Set up Maple agent tracing for Pydantic AI in this project.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing-pydantic-ai -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## Export spans to Maple

Install the OpenTelemetry SDK and the OTLP/HTTP exporter next to Pydantic AI. Swap `[openai]` for the extras of the providers you use.

```bash
pip install "pydantic-ai-slim[openai]>=2.51" "opentelemetry-sdk>=1.45" "opentelemetry-exporter-otlp-proto-http>=1.45"
```

Point the exporter at Maple. For an EU organization, use `https://ingest.eu.maple.dev`.

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
```

Set up tracing once, when your process starts:

```py
# tracing.py
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from pydantic_ai import Agent, InstrumentationSettings

provider = TracerProvider(
    resource=Resource.create(
        {"service.name": "support-agent", "deployment.environment.name": "production"}
    )
)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

Agent.instrument_all(
    InstrumentationSettings(
        tracer_provider=provider,
        include_content=True,
        include_binary_content=False,
    )
)
```

Import `tracing` at the top of your entry point (`main.py`, the FastAPI app module, the worker), before the first `agent.run()`.

If your app already has a `TracerProvider` (from `opentelemetry-instrument`, Sentry or your own setup), add the `BatchSpanProcessor` to it instead of creating a second one, and call `Agent.instrument_all(InstrumentationSettings(include_content=True, include_binary_content=False))` without `tracer_provider`.

### If you already use Logfire

Skip the `pip install` above, because it conflicts with Logfire's OpenTelemetry pins. Keep the three environment variables. Logfire exports to Maple whenever `OTEL_EXPORTER_OTLP_ENDPOINT` is set.

```py
import logfire

logfire.configure(service_name="support-agent", environment="production", send_to_logfire=False)
logfire.instrument_pydantic_ai()
```

Logfire scrubs tool arguments and results by default. If they arrive as `[Scrubbed due to ...]`, see Troubleshooting.

## Pass the conversation id on every run

Pass the chat or thread id your app already has as `conversation_id=` on every `run()`, `run_stream()` and `iter()`:

```py
from pydantic_ai import Agent

support = Agent("openai:gpt-4o-mini", name="support")


async def handle_message(chat_id: str, text: str, history: list) -> str:
    result = await support.run(text, conversation_id=chat_id, message_history=history)
    return result.output
```

The id must stay the same for the whole conversation and differ between conversations.

When streaming, keep the `async with` block open until the stream is finished. The run's span ends when the block exits.

```py
async def stream_reply(chat_id: str, text: str, history: list):
    async with support.run_stream(text, conversation_id=chat_id, message_history=history) as run:
        async for delta in run.stream_text(delta=True):
            yield delta
```

### Sub-agents

When a tool runs another agent, pass `conversation_id` and `usage` from the tool's context. Give every agent a `name=` so each one gets its own lane.

```py
from pydantic_ai import Agent, RunContext

weather_worker = Agent("openai:gpt-4o-mini", name="weather_worker", tools=[get_weather])
orchestrator = Agent("openai:gpt-4o-mini", name="orchestrator")


@orchestrator.tool
async def research_weather(ctx: RunContext[None], city: str) -> str:
    """Delegate to the weather worker."""
    result = await weather_worker.run(
        f"What is the current weather in {city}?",
        usage=ctx.usage,
        conversation_id=ctx.conversation_id,
    )
    return result.output
```

## Flush short-lived processes

A Lambda, a killed worker or a notebook kernel doesn't exit normally, so flush explicitly:

```py
import asyncio

from opentelemetry import trace


def handler(event, context):
    try:
        return asyncio.run(handle_message(event["chat_id"], event["text"], []))
    finally:
        trace.get_tracer_provider().force_flush()
```

In a script or CLI, call `provider.shutdown()` at the end. With Logfire, use `logfire.force_flush()` or `logfire.shutdown()`.

## Check that it works

If Pydantic AI prints an `observability: off` banner on the first run, `tracing.py` didn't run before your first `agent.run()`.

Run a conversation with two messages and a tool call, then open **Agent Sessions**. You should see one session with your conversation id and framework **Pydantic AI**, one turn per `run()`, a transcript with prompts, replies and tool calls, and token counts and cost on every model call.

## Troubleshooting

- **Every message is its own session.** Pass `conversation_id=` on every `run()`, `run_stream()` and `iter()`.
- **A multi-agent run is split into several turns or sessions.** Pass `conversation_id=ctx.conversation_id` to every nested `run()`.
- **Tool arguments or results read `[Scrubbed due to ...]`.** Logfire's scrubbing matched a word like `session` or `auth`. Pass `scrubbing=logfire.ScrubbingOptions(callback=...)` that keeps `gen_ai.tool.call.arguments` and `gen_ai.tool.call.result`, or `scrubbing=False`.
- **A failed tool shows as successful.** The tool returned an error value. Raise `ToolFailed("...")` so the call is marked failed and the model still sees the message.
- **Spans show up twice.** Another instrumentor (Logfire's `instrument_openai()`, OpenInference, OpenLLMetry) also traces the model client. Remove it and keep Pydantic AI's.

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview)
- [Instrument a Python application](/docs/guides/instrumentation-python)

---

# Trace CrewAI crews and flows with OpenTelemetry

> AI Frameworks · https://maple.dev/docs/agent-tracing/crewai

Send CrewAI crews and flows to Maple with OpenInference, one Agent Session per conversation.

CrewAI needs two OpenInference instrumentors: `openinference-instrumentation-crewai` for crews, agents and tools, and one for your model provider, which records prompts and tokens. You also wrap every `kickoff()` in a conversation id so a chat becomes one session.

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing-crewai](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-crewai) skill and follows it.

```text
Set up Maple agent tracing for CrewAI in this project.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing-crewai -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## Install the instrumentors

```bash
pip install "crewai>=1.15" "openinference-instrumentation-crewai>=1.1.18" \
  "openinference-instrumentation-openai>=0.1.61" \
  "opentelemetry-sdk>=1.45" "opentelemetry-exporter-otlp-proto-http>=1.45"
```

Pick the model instrumentor by the model string you pass to `LLM(...)`:

| Model string | Instrumentor |
| --- | --- |
| `openai/…`, `openrouter/…`, `deepseek/…`, `ollama/…`, `custom_openai=True`, or a bare name like `gpt-4.1-mini` | `openinference-instrumentation-openai` |
| `anthropic/…` or a bare `claude-…` | `openinference-instrumentation-anthropic` |
| `gemini/…` or a bare `gemini-…` | `openinference-instrumentation-google-genai` |
| `bedrock/…` | `openinference-instrumentation-bedrock` |
| Anything else (needs `crewai[litellm]`) | `openinference-instrumentation-litellm` |

Install only the ones your crews use. The LiteLLM instrumentor records nothing for the first four rows.

## Point the exporter at Maple

```bash
export OTEL_SERVICE_NAME=support-crew
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production
export OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
export CREWAI_DISABLE_TELEMETRY=true
export CREWAI_TRACING_ENABLED=false
```

EU organizations use `https://ingest.eu.maple.dev`. If you pass `endpoint=` to `OTLPSpanExporter` in code instead, it has to end in `/v1/traces`.

The last two turn off CrewAI's own telemetry and a first-run prompt that blocks the process at exit. Don't use `OTEL_SDK_DISABLED=true` instead, because it disables your Maple traces too.

## Initialize tracing

Add a `tracing.py` and import it at the top of your entry point, before the first `kickoff()`:

```py
# tracing.py
from openinference.instrumentation import TraceConfig
from openinference.instrumentation.crewai import CrewAIInstrumentor
from openinference.instrumentation.openai import OpenAIInstrumentor
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import SpanProcessor, TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor


class CrewAIAgentNames(SpanProcessor):
    def on_start(self, span, parent_context=None):
        parent = trace.get_current_span(parent_context)
        attrs = getattr(parent, "attributes", None) or {}
        role = attrs.get("graph.node.id")
        if role and "gen_ai.agent.name" not in attrs and parent.is_recording():
            parent.set_attribute("gen_ai.agent.name", role)


provider = TracerProvider()
provider.add_span_processor(CrewAIAgentNames())
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

config = TraceConfig(enable_genai_semconv=True)
CrewAIInstrumentor().instrument(tracer_provider=provider, config=config, skip_dep_check=True)
OpenAIInstrumentor().instrument(tracer_provider=provider, config=config, skip_dep_check=True)
```

Pass `config` to every instrumentor. Keep `skip_dep_check=True`, or an instrumentor can silently skip itself.

If the app already has a `TracerProvider` (from `opentelemetry-instrument`, Logfire or Sentry), add `CrewAIAgentNames()` and the exporter to it and pass it to `instrument()`.

## Group a conversation into one session

CrewAI has no conversation id, so wrap every `kickoff()` in OpenInference's `using_session` with the id your app stores the chat under:

```py
import tracing  # noqa: F401  (first import)

from crewai import LLM, Agent, Crew, Task
from openinference.instrumentation import using_session

llm = LLM(model="openai/gpt-4o-mini", temperature=0)


def build_crew(text: str, history: str, stream: bool = False) -> Crew:
    assistant = Agent(
        role="assistant",
        goal="Answer the user's questions",
        backstory="You are a concise, helpful assistant.",
        llm=llm,
        tools=[get_weather, calculate],
    )
    task = Task(
        description=f"{text}\n\nConversation so far:\n{history}",
        expected_output="A short, direct reply to the user.",
        agent=assistant,
        name="reply",
    )
    return Crew(name="support", agents=[assistant], tasks=[task], stream=stream)


def handle_message(conversation_id: str, text: str, history: str) -> str:
    with using_session(conversation_id):
        return build_crew(text, history).kickoff().raw
```

Put the user's message first in the task description, because Maple labels each turn with its first line. Give the crew a `name=`. `crew_id` and `crew_key` don't work as conversation ids.

For conversational flows, wrap `flow.handle_turn(text, session_id=conversation_id)` in the same `using_session` and set `name = "support_flow"` on the flow class.

Use `kickoff()` or `await crew.kickoff_async()`, never `akickoff()`, which isn't traced as one run.

## Streaming crews

`Crew(stream=True)` adds an empty turn to every message. Wrap the streamed turn in one span of your own:

```py
from opentelemetry import trace

tracer = trace.get_tracer("chat")


def stream_message(conversation_id: str, text: str, history: str, send) -> None:
    with using_session(conversation_id), tracer.start_as_current_span(
        "invoke_agent support",
        attributes={
            "gen_ai.operation.name": "invoke_agent",
            "gen_ai.conversation.id": conversation_id,
        },
    ):
        for chunk in build_crew(text, history, stream=True).kickoff():
            send(chunk.content)
```

`LLM(stream=True)` without `Crew(stream=True)` doesn't need this.

## Flush in short-lived processes

Servers and `crewai run` need nothing. In serverless handlers and notebooks, import `provider` from `tracing` and call `provider.force_flush()` in a `finally` after each run.

## Check that it works

Send two or three messages with the same conversation id, one of them using a tool, then open **Agent Sessions**. Within a minute you should see one session labeled **CrewAI**, with one turn per `kickoff()` starting at `support.kickoff`, `ChatCompletion` model calls with tokens, tool calls like `get_weather.run`, and one lane per agent role.

Cost shows as **unpriced** unless your models go through LiteLLM, which is expected.

## Troubleshooting

- **Agent and tool spans, but no model calls or tokens.** The instrumentor for your model's SDK is missing. Match it to the model string with the table above.
- **No spans at all.** Import `tracing` before the first kickoff, keep `skip_dep_check=True`, make sure `OTEL_SDK_DISABLED` isn't set, and check the logs for exporter errors.
- **One session per message.** The kickoff isn't inside `using_session(...)`, or the id changes per request.
- **Every model and tool call is its own trace.** Replace `akickoff()` with `kickoff()` or `kickoff_async()`.
- **The process hangs at exit asking about traces.** Set `CREWAI_TRACING_ENABLED=false`.

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview): what Maple builds from these spans.
- [LiteLLM](/docs/agent-tracing/litellm) and [OpenRouter](/docs/agent-tracing/openrouter): if your models go through either gateway.

---

# Trace Google ADK agents with OpenTelemetry

> AI Frameworks · https://maple.dev/docs/agent-tracing/google-adk

Send Google Agent Development Kit (ADK) traces to Maple with the transcript, tool calls and tokens, one session per ADK session.

import LanguageTabs from "../../../components/docs/LanguageTabs.astro"
import LanguageTab from "../../../components/docs/LanguageTab.astro"

Google's Agent Development Kit (ADK) emits OpenTelemetry spans on its own, in Python (`google-adk`) and TypeScript (`@google/adk`), so you need no instrumentation package. You register a tracer provider and add a few lines that put the transcript and tool calls in the format Maple reads.

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing-google-adk](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-google-adk) skill and follows it.

```text
Set up Maple agent tracing for Google ADK in this project.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing-google-adk -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## Install ADK and the OTLP exporter

<LanguageTabs label="Language" tabs={[{ id: "python", label: "Python" }, { id: "typescript", label: "TypeScript" }]}>
<LanguageTab id="python">

```bash
pip install "google-adk>=2.10" litellm opentelemetry-exporter-otlp-proto-http
```

`litellm` is only needed for non-Gemini models. Don't pin a newer OpenTelemetry version: ADK 2.10 caps `opentelemetry-sdk` at 1.42.1.

</LanguageTab>
<LanguageTab id="typescript">

```bash
npm install @google/adk@^2.1 zod @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-proto
```

`zod` is for the tool in the example below.

</LanguageTab>
</LanguageTabs>

## Configure the export

<LanguageTabs label="Language" tabs={[{ id: "python", label: "Python" }, { id: "typescript", label: "TypeScript" }]}>
<LanguageTab id="python">

```bash
export OTEL_SERVICE_NAME=support-agent
export OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
# Put prompts, replies and tool calls on span attributes, in the format Maple reads
export OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY
# Drop ADK's own copies of the same content, which Maple doesn't read
export ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false
```

EU organizations use `https://ingest.eu.maple.dev`. Use `SPAN_ONLY` exactly, because `true` leaves the transcript empty.

These settings store every prompt and tool result in Maple. To keep structure and tokens without content, leave `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` unset and delete the two `gen_ai.tool.call.*` lines from the plugin below.

</LanguageTab>
<LanguageTab id="typescript">

```bash
export OTEL_SERVICE_NAME="support-agent"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
```

EU organizations use `https://ingest.eu.maple.dev`. The exporter appends `/v1/traces` itself.

These settings store every prompt and tool result in Maple. To keep structure and tokens without content, set `ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false`.

</LanguageTab>
</LanguageTabs>

## Register a tracer provider

<LanguageTabs label="Language" tabs={[{ id: "python", label: "Python" }, { id: "typescript", label: "TypeScript" }]}>
<LanguageTab id="python">

A `Runner` in your own app, worker or script exports nothing until you register a tracer provider. Add a `telemetry.py`:

```py
# telemetry.py
import json

from google.adk.plugins.base_plugin import BasePlugin
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor


class SkipDuplicateToolSpans(BatchSpanProcessor):
    """Drops two ADK tool spans that would count a call twice: the
    `execute_tool (merged)` summary of parallel calls, and the span of a call
    paused for confirmation (it runs again, in its own span, once approved)."""

    def on_end(self, span):
        if span.name != "execute_tool (merged)" and not span.attributes.get("adk.awaiting_confirmation"):
            super().on_end(span)


class ToolCallAttributes(BasePlugin):
    """Records each tool call's arguments and result on its `execute_tool` span,
    and marks the span of a call that is waiting for confirmation."""

    def __init__(self):
        super().__init__(name="tool_call_attributes")

    async def before_tool_callback(self, *, tool, tool_args, tool_context):
        trace.get_current_span().set_attribute("gen_ai.tool.call.arguments", json.dumps(tool_args, default=str))

    async def after_tool_callback(self, *, tool, tool_args, tool_context, result):
        span = trace.get_current_span()
        if tool_context.actions.requested_tool_confirmations:
            span.set_attribute("adk.awaiting_confirmation", True)
        span.set_attribute("gen_ai.tool.call.result", json.dumps(result, default=str))


# Reads OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS
provider = TracerProvider(resource=Resource.create())
provider.add_span_processor(SkipDuplicateToolSpans(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
```

Import `telemetry` as the first line of your entry point and register the plugin on the runner:

```py
# main.py
import telemetry  # first, so the provider exists before ADK runs

from google.adk.agents import LlmAgent
from google.adk.models.lite_llm import LiteLlm
from google.adk.runners import Runner
from google.adk.sessions import InMemorySessionService
from google.genai import types


def get_weather(city: str) -> dict:
    """Get the current weather for a city."""
    return {"city": city, "temperature_c": 21, "condition": "partly cloudy"}


agent = LlmAgent(
    name="assistant",
    model=LiteLlm(model="openrouter/openai/gpt-4o-mini"),
    instruction="You are a concise assistant.",
    tools=[get_weather],
)

runner = Runner(
    app_name="support",
    agent=agent,
    session_service=InMemorySessionService(),
    plugins=[telemetry.ToolCallAttributes()],
    auto_create_session=True,
)
```

If your app already sets up OpenTelemetry (Logfire, Sentry, a platform agent), add the `SkipDuplicateToolSpans(OTLPSpanExporter())` processor to that provider instead.

`adk web` and `adk api_server` create the provider themselves. There, skip it and register the plugin on your `App`: `App(name="support", root_agent=agent, plugins=[ToolCallAttributes()])`.

</LanguageTab>
<LanguageTab id="typescript">

ADK records each model request, reply and tool call on its spans in its own format. The span processor below rewrites them into the attributes Maple reads. Create an `instrumentation.ts` and import it as the first line of your entry point (`import "./instrumentation"`):

```ts
// instrumentation.ts
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto"
import { NodeSDK, tracing } from "@opentelemetry/sdk-node"

type Part = {
	text?: string
	thought?: boolean
	functionCall?: { id?: string; name?: string; args?: unknown }
	functionResponse?: { id?: string; response?: unknown }
}
type Content = { role?: string; parts?: Part[] }

const parse = (value: unknown) => (typeof value === "string" ? JSON.parse(value) : {})

// Gemini content -> the GenAI message format Maple reads
const toParts = (parts: Part[] = []) =>
	parts.flatMap((part): object[] => {
		if (part.functionCall) {
			const { id, name, args } = part.functionCall
			return [{ type: "tool_call", id, name, arguments: args }]
		}
		if (part.functionResponse) {
			const { id, response } = part.functionResponse
			return [{ type: "tool_call_response", id, response }]
		}
		if (typeof part.text !== "string") return []
		return [{ type: part.thought ? "reasoning" : "text", content: part.text }]
	})

const toMessage = ({ role, parts }: Content) => ({
	role: parts?.some((part) => part.functionResponse) ? "tool" : role === "model" ? "assistant" : "user",
	parts: toParts(parts),
})

/** Copies what ADK records on its spans into the GenAI attributes Maple reads. */
class AdkSpanProcessor extends tracing.BatchSpanProcessor {
	override onEnd(span: tracing.ReadableSpan) {
		// ADK's summary of parallel tool calls, which are also traced one by one
		if (span.name === "execute_tool (merged)") return
		const attributes = span.attributes

		if (span.name === "call_llm") {
			const request = parse(attributes["gcp.vertex.agent.llm_request"])
			const response = parse(attributes["gcp.vertex.agent.llm_response"])
			const usage = response.usageMetadata ?? {}
			const system = request.config?.systemInstruction
			attributes["gen_ai.operation.name"] = "chat"
			attributes["gen_ai.provider.name"] = "gcp.gemini"
			if (usage.thoughtsTokenCount) {
				attributes["gen_ai.usage.reasoning.output_tokens"] = usage.thoughtsTokenCount
			}
			if (usage.cachedContentTokenCount) attributes["gen_ai.usage.cache_read.input_tokens"] = usage.cachedContentTokenCount
			if (request.contents) attributes["gen_ai.input.messages"] = JSON.stringify(request.contents.map(toMessage))
			if (response.content?.parts?.length) {
				const finish_reason = response.finishReason?.toLowerCase()
				attributes["gen_ai.output.messages"] = JSON.stringify([{ ...toMessage(response.content), finish_reason }])
			}
			if (system) {
				const parts = typeof system === "string" ? [{ type: "text", content: system }] : toParts(system.parts)
				attributes["gen_ai.system_instructions"] = JSON.stringify(parts)
			}
		} else if (span.name.startsWith("execute_tool ")) {
			if (attributes["gen_ai.operation.name"] === undefined) {
				// ADK leaves the span bare when the tool threw or doesn't exist
				attributes["gen_ai.operation.name"] = "execute_tool"
				attributes["gen_ai.tool.name"] = span.name.slice("execute_tool ".length)
				attributes["error.type"] = "tool_error"
			} else {
				const result = attributes["gcp.vertex.agent.tool_response"]
				attributes["gen_ai.tool.call.arguments"] = attributes["gcp.vertex.agent.tool_call_args"]
				attributes["gen_ai.tool.call.result"] = result
				if (parse(result).error) attributes["error.type"] = "tool_error"
			}
		}

		// ADK's own copies of the same content, which Maple doesn't read
		for (const key of ["llm_request", "llm_response", "tool_call_args", "tool_response"]) {
			delete attributes[`gcp.vertex.agent.${key}`]
		}
		super.onEnd(span)
	}
}

// Reads OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS
export const spanProcessor = new AdkSpanProcessor(new OTLPTraceExporter())

// Reads OTEL_SERVICE_NAME
export const sdk = new NodeSDK({ spanProcessors: [spanProcessor] })
sdk.start()
```

If your app already starts OpenTelemetry (Sentry, auto-instrumentation, your own `NodeTracerProvider`), skip the `NodeSDK` lines and add `spanProcessor` to that provider's span processors instead.

`npx adk web` and `npx adk api_server` create their own provider without this processor, so sessions traced there show no transcript. Trace your own `Runner` instead.

</LanguageTab>
</LanguageTabs>

## Use one ADK session per conversation

<LanguageTabs label="Language" tabs={[{ id: "python", label: "Python" }, { id: "typescript", label: "TypeScript" }]}>
<LanguageTab id="python">

Each `runner.run_async()` call is one turn. Pass your conversation id as `session_id` on every turn:

```py
async def chat(conversation_id: str, user_id: str, text: str) -> str:
    reply = ""
    async for event in runner.run_async(
        user_id=user_id,
        session_id=conversation_id,  # the same id on every turn of this conversation
        new_message=types.Content(role="user", parts=[types.Part(text=text)]),
    ):
        if event.is_final_response() and event.content and event.content.parts:
            reply = "".join(part.text or "" for part in event.content.parts)
    return reply
```

With `auto_create_session=True`, the runner creates the session on the first turn. Don't call `create_session()` without a `session_id`, or every message becomes its own session.

To call an agent like a tool, add it to `sub_agents` with `mode="single_turn"` instead of using `AgentTool`, which can move the turn into a separate session.

</LanguageTab>
<LanguageTab id="typescript">

Each `runner.runAsync()` call is one turn. Pass your conversation id as `sessionId` on every turn:

```ts
// agent.ts
import { FunctionTool, InMemorySessionService, LlmAgent, Runner } from "@google/adk"
import { z } from "zod"

const getWeather = new FunctionTool({
	name: "get_weather",
	description: "Get the current weather for a city.",
	parameters: z.object({ city: z.string() }),
	execute: async ({ city }) => ({ city, temperature_c: 21, condition: "partly cloudy" }),
})

const agent = new LlmAgent({
	name: "assistant",
	model: "gemini-2.5-flash",
	instruction: "You are a concise assistant.",
	tools: [getWeather],
})

const sessionService = new InMemorySessionService()
const runner = new Runner({ appName: "support", agent, sessionService })

export async function chat(conversationId: string, userId: string, text: string) {
	// The same id on every turn of this conversation
	await sessionService.getOrCreateSession({ appName: "support", userId, sessionId: conversationId })

	let reply = ""
	for await (const event of runner.runAsync({
		userId,
		sessionId: conversationId,
		newMessage: { role: "user", parts: [{ text }] },
	})) {
		if (event.content?.parts && !event.partial) {
			reply = event.content.parts.map((part) => part.text ?? "").join("")
		}
	}
	return reply
}
```

`getOrCreateSession()` creates the session under your id on the first turn. Don't call `createSession()` without a `sessionId`, or every message becomes its own session.

</LanguageTab>
</LanguageTabs>

## Flush before a short-lived process exits

<LanguageTabs label="Language" tabs={[{ id: "python", label: "Python" }, { id: "typescript", label: "TypeScript" }]}>
<LanguageTab id="python">

A script, notebook cell or job that exits within 5 seconds of its last turn loses that turn. Flush when the work ends:

```py
# script.py
import telemetry  # first

import asyncio

from main import chat


async def main():
    try:
        await chat("support-4821", "user-17", "What's the weather in Berlin?")
    finally:
        telemetry.provider.force_flush()
        telemetry.provider.shutdown()


asyncio.run(main())
```

In a server, call `provider.shutdown()` from your shutdown hook. On Cloud Run or another platform that freezes the CPU between requests, call `provider.force_flush()` before each response returns.

</LanguageTab>
<LanguageTab id="typescript">

A script or job that exits right after its last turn loses that turn. Shut the SDK down when the work ends:

```ts
// script.ts
import { sdk } from "./instrumentation"
import { chat } from "./agent"

try {
	console.log(await chat("support-4821", "user-17", "What's the weather in Berlin?"))
} finally {
	await sdk.shutdown()
}
```

In a server, call `await sdk.shutdown()` from your shutdown hook. In a serverless handler, or on a platform that freezes the CPU between requests, call `await spanProcessor.forceFlush()` before each response returns.

</LanguageTab>
</LanguageTabs>

## Check that it works

Run a conversation of two or three turns, one calling a tool, then open **Agent Sessions**. You should see one session with the framework **Google ADK**, one turn per run, tool calls with their arguments and results, and tokens on each model call. Cost shows as unpriced.

## Troubleshooting

- **No spans at all.** No tracer provider is registered when the runner runs. Import `telemetry.py` or `instrumentation.ts` first.
- **Tokens but an empty transcript.** In Python, `OTEL_SEMCONV_STABILITY_OPT_IN` or `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY` is missing, or the capture variable is `true`. In TypeScript, `AdkSpanProcessor` isn't on the provider, or `ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS` is `false`.
- **Every message is its own session.** Pass the same conversation id as the session id on every turn.
- **A failing tool shows as successful.** It returned `{"status": "error", ...}`. Return an object with an `"error"` key, or raise an error.
- **Every model call appears twice.** Another instrumentor wraps the same calls (`litellm.callbacks = ["otel"]`, `openinference-instrumentation-google-adk`), or a second OpenTelemetry SDK exports the same spans. Remove it.

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview): how Maple builds sessions, turns and checks.
- [ADK agent activity traces](https://google.github.io/adk-docs/observability/traces/): ADK's span reference and export setup.

---

# Trace LlamaIndex agents with OpenTelemetry

> AI Frameworks · https://maple.dev/docs/agent-tracing/llamaindex

Send LlamaIndex agent and workflow runs to Maple with OpenInference, one Agent Session per conversation.

OpenInference's `openinference-instrumentation-llama-index` sends LlamaIndex agents and workflows to Maple. You add a small span processor and wrap every `agent.run()` in a conversation id so a chat becomes one session.

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing-llamaindex](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-llamaindex) skill and follows it.

```text
Set up Maple agent tracing for LlamaIndex in this project.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing-llamaindex -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## Install the instrumentor

```bash
pip install "llama-index-core>=0.14.25" "openinference-instrumentation-llama-index>=4.5.2" \
  "opentelemetry-sdk>=1.45" "opentelemetry-exporter-otlp-proto-http>=1.45"
```

Add your model package (`llama-index-llms-openai`, `llama-index-llms-openrouter`, ...) as usual.

If the app uses LlamaIndex's own `llama-index-observability-otel`, remove it. Maple can't read its transcripts, and running both doubles every span.

## Point the exporter at Maple

```bash
export OTEL_SERVICE_NAME=support-agent
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production
export OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
```

EU organizations use `https://ingest.eu.maple.dev`. If you pass `endpoint=` to `OTLPSpanExporter` in code instead, it has to end in `/v1/traces`.

## Initialize tracing

Add a `tracing.py` and import it at the top of your entry point, before the first `agent.run()`:

```py
# tracing.py
from llama_index.core.instrumentation.dispatcher import active_instrument_tags
from openinference.instrumentation import TraceConfig
from openinference.instrumentation.llama_index import LlamaIndexInstrumentor
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import SpanProcessor, TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

LLM_METHODS = (".chat", ".achat", ".stream_chat", ".astream_chat",
               ".complete", ".acomplete", ".stream_complete", ".astream_complete")


class LlamaIndexForMaple(SpanProcessor):
    def __init__(self, exporter_processor: SpanProcessor):
        self._next = exporter_processor
        self._open_llm_spans = {}

    def on_start(self, span, parent_context=None):
        agent_name = active_instrument_tags.get().get("gen_ai.agent.name")
        if agent_name:
            span.set_attribute("gen_ai.agent.name", agent_name)
        if span.name.endswith((".call_tool", ".aggregate_tool_results")):
            span.set_attribute("gen_ai.operation.name", "invoke_workflow")
        if span.name.endswith(LLM_METHODS):
            self._open_llm_spans[span.context.span_id] = span
        self._next.on_start(span, parent_context)

    def on_end(self, span):
        self._open_llm_spans.pop(span.context.span_id, None)
        if span.name.endswith("._prepare_chat_with_tools"):
            return
        if (span.status.description or "").startswith("WaitingForEvent"):
            return
        outer = self._open_llm_spans.get(span.parent.span_id) if span.parent else None
        if outer is not None and outer.name == span.name:
            outer.set_attributes(span.attributes)
            return
        self._next.on_end(span)

    def shutdown(self):
        self._next.shutdown()

    def force_flush(self, timeout_millis=30000):
        return self._next.force_flush(timeout_millis)


provider = TracerProvider()
provider.add_span_processor(LlamaIndexForMaple(BatchSpanProcessor(OTLPSpanExporter())))
trace.set_tracer_provider(provider)

LlamaIndexInstrumentor().instrument(
    tracer_provider=provider,
    config=TraceConfig(enable_genai_semconv=True),
)
```

Always add the exporter through `LlamaIndexForMaple`, never directly.

If the app already has a `TracerProvider` (from `opentelemetry-instrument`, Logfire or Sentry), add `LlamaIndexForMaple(BatchSpanProcessor(OTLPSpanExporter()))` to it and pass it to `instrument()`.

## Group a conversation into one session

Wrap each `agent.run()` call in `using_session` with the conversation id your app stores the chat under, and tag it with the agent's name:

```py
from llama_index.core.agent.workflow import AgentStream, FunctionAgent
from llama_index.core.instrumentation.dispatcher import instrument_tags
from llama_index.core.workflow import Context
from openinference.instrumentation import using_session

agent = FunctionAgent(name="assistant", llm=llm, tools=[get_weather, calculate],
                      system_prompt="You are a helpful assistant.")
contexts: dict[str, Context] = {}


async def handle_message(conversation_id: str, text: str):
    if conversation_id not in contexts:
        contexts[conversation_id] = Context(agent)
    ctx = contexts[conversation_id]
    with using_session(conversation_id), instrument_tags({"gen_ai.agent.name": agent.name}):
        handler = agent.run(user_msg=text, ctx=ctx)
    async for event in handler.stream_events():
        if isinstance(event, AgentStream):
            yield event.delta
    await handler
```

Only the `agent.run()` call needs to be inside the `with`. Consume the stream outside it.

Keep one `Context` per conversation. A new UUID per request gives you one session per message.

For multi-agent workflows, run the whole workflow inside `using_session(conversation_id)` and wrap each sub-agent's `run()` in its own `instrument_tags({"gen_ai.agent.name": agent.name})` to give each agent its own lane. `AgentWorkflow` handoffs show as a single agent.

## Get tokens on streamed calls

`FunctionAgent` streams its model calls, and OpenAI only reports tokens on a stream when asked. Pass `stream_options` on OpenAI and OpenAI-compatible models:

```py
from llama_index.llms.openai import OpenAI

llm = OpenAI(model="gpt-4o-mini", additional_kwargs={"stream_options": {"include_usage": True}})
```

With `OpenAILike` or `OpenRouter`, also pass `is_function_calling_model=True`, or the agent never calls tools.

## Flush in short-lived processes

Long-running servers need nothing. In serverless handlers, notebooks and task workers, import `provider` from `tracing` and call `provider.force_flush()` in a `finally` after each run.

## Check that it works

Send two or three messages with the same conversation id, one of them using a tool, then open **Agent Sessions**. Within a minute you should see one session labeled **LlamaIndex**, with one turn per `agent.run()`, a transcript, one model call per request with tokens, and `FunctionTool.acall` tool calls.

Cost shows as **unpriced** and streamed model calls last about 1 ms. Both are expected.

## Troubleshooting

- **No spans at all.** Import `tracing` before the first `agent.run()`, check the logs for `DependencyConflict` (upgrade llama-index-core) and exporter errors.
- **One session per message.** `agent.run()` isn't inside `using_session(...)`, or the id changes per request.
- **Each model or tool call counted two or three times.** The exporter was added directly. Add it through `LlamaIndexForMaple`.
- **No tokens on streamed calls.** Add `stream_options={"include_usage": True}` through `additional_kwargs`.
- **No lanes or agent names.** Wrap each agent's `run()` in `instrument_tags({"gen_ai.agent.name": agent.name})`.

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview): what Maple builds from these spans.
- [OpenRouter](/docs/agent-tracing/openrouter): cost per call if your models go through OpenRouter.

---

# Trace Strands Agents with OpenTelemetry

> AI Frameworks · https://maple.dev/docs/agent-tracing/strands

Send Strands Agents traces to Maple with the full transcript and one Agent Session per conversation.

import LanguageTabs from "../../../components/docs/LanguageTabs.astro"
import LanguageTab from "../../../components/docs/LanguageTab.astro"

Strands Agents has a built-in OpenTelemetry tracer, and Maple reads its spans without an extra instrumentation library. You set one environment variable so the transcript is recorded, and pass your conversation id as `session.id` on each agent.

You need `strands-agents` 1.51 or newer, or the TypeScript SDK `@strands-agents/sdk` 1.19 or newer.

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing-strands](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-strands) skill and follows it.

```text
Set up Maple agent tracing for Strands Agents in this project.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing-strands -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## Install and configure the exporter

<LanguageTabs label="Language" tabs={[{ id: "python", label: "Python" }, { id: "typescript", label: "TypeScript" }]}>
<LanguageTab id="python">

Replace `openai` with your model provider's extra (`anthropic`, `litellm`; Bedrock needs none).

```bash
pip install 'strands-agents[otel,openai]>=1.57'
```

</LanguageTab>
<LanguageTab id="typescript">

Install the SDK with its OpenTelemetry packages:

```bash
npm install @strands-agents/sdk @opentelemetry/api @opentelemetry/sdk-trace-base @opentelemetry/sdk-trace-node @opentelemetry/resources @opentelemetry/exporter-trace-otlp-http @opentelemetry/sdk-metrics @opentelemetry/exporter-metrics-otlp-http
```

</LanguageTab>
</LanguageTabs>

Set these environment variables:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_SERVICE_NAME="support-agent"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production"
export OTEL_SEMCONV_STABILITY_OPT_IN="gen_ai_latest_experimental,gen_ai_span_attributes_only"
```

For an EU organization, use `https://ingest.eu.maple.dev`. The exporter appends `/v1/traces` itself.

Without `OTEL_SEMCONV_STABILITY_OPT_IN` the transcript is empty. Strands reads it when the first `Agent` is created, so set it in the environment rather than in code.

To redact message content, append `gen_ai_unredacted_attributes=` with a `;`-separated allowlist of attributes to keep; everything else becomes `[REDACTED]`.

<LanguageTabs label="Language" tabs={[{ id: "python", label: "Python" }, { id: "typescript", label: "TypeScript" }]}>
<LanguageTab id="python">

Then start the tracer once, before your first agent runs:

```py
# telemetry.py: import this from your entry point before creating any Agent
from strands.telemetry import StrandsTelemetry

telemetry = StrandsTelemetry().setup_otlp_exporter()
```

If your app already sets up a global `TracerProvider` (for example `opentelemetry-instrument` or the ADOT distro on AgentCore), skip `StrandsTelemetry()` and add a `BatchSpanProcessor(OTLPSpanExporter(...))` pointing at Maple to that provider.

</LanguageTab>
<LanguageTab id="typescript">

Use the same `OTEL_EXPORTER_OTLP_*` variables, with `OTEL_SEMCONV_STABILITY_OPT_IN="gen_ai_latest_experimental,gen_ai_span_attributes_only"` (the SDK doesn't support the third value). Then:

```ts
import { Agent, FileStorage, SessionManager } from "@strands-agents/sdk"
import { OpenAIModel } from "@strands-agents/sdk/models/openai"
import { setupTracer } from "@strands-agents/sdk/telemetry"

const provider = setupTracer({ exporters: { otlp: true } }) // reads OTEL_EXPORTER_OTLP_* env vars
```

</LanguageTab>
</LanguageTabs>

## Pass the conversation id as session.id

<LanguageTabs label="Language" tabs={[{ id: "python", label: "Python" }, { id: "typescript", label: "TypeScript" }]}>
<LanguageTab id="python">

Pass the conversation id in `trace_attributes`:

```py
from strands import Agent
from strands.models.openai import OpenAIModel
from strands.session.file_session_manager import FileSessionManager

model = OpenAIModel(model_id="gpt-4o-mini", params={"max_tokens": 600})


def handle_message(conversation_id: str, text: str) -> str:
    agent = Agent(
        name="support_agent",
        model=model,
        tools=[get_weather, calculate],
        system_prompt="You are a concise support assistant.",
        session_manager=FileSessionManager(session_id=conversation_id, storage_dir="./sessions"),
        trace_attributes={"session.id": conversation_id},
        callback_handler=None,
    )
    return str(agent(text))
```

The session manager restores history but doesn't put its `session_id` on the spans, so you need both arguments. Use the conversation id your app already has, never a fresh UUID per request.

Create the agent per request, as above. A shared module-level `Agent` would carry one user's id into everyone's traces. Give every agent a `name`, or sub-agents merge into one lane.

With `agent.as_tool()` sub-agents, only the orchestrator needs `session.id`. For a `Swarm`, pass `trace_attributes={"session.id": conversation_id}` to the `Swarm`. For a `Graph`, set `graph.trace_attributes = {"session.id": conversation_id}` after `builder.build()`.

</LanguageTab>
<LanguageTab id="typescript">

```ts
const model = new OpenAIModel({ modelId: "gpt-4o-mini", params: { max_tokens: 600 } })

async function handleMessage(conversationId: string, text: string): Promise<string> {
	const agent = new Agent({
		name: "support_agent",
		model,
		tools: [getWeather],
		systemPrompt: "You are a concise support assistant.",
		traceAttributes: { "session.id": conversationId },
		sessionManager: new SessionManager({
			sessionId: conversationId,
			storage: { snapshot: new FileStorage("./sessions") },
		}),
		printer: false,
	})
	return String(await agent.invoke(text))
}
```

Create the agent per request here too.

</LanguageTab>
</LanguageTabs>

## Flush in scripts and jobs

A long-running server needs nothing extra. Scripts, notebooks and jobs lose their last spans unless they flush:

<LanguageTabs label="Language" tabs={[{ id: "python", label: "Python" }, { id: "typescript", label: "TypeScript" }]}>
<LanguageTab id="python">

```py
from telemetry import telemetry

try:
    handle_message(conversation_id, "What's the weather in Berlin?")
finally:
    telemetry.tracer_provider.force_flush()
    telemetry.tracer_provider.shutdown()
```

On AWS Lambda, call `telemetry.tracer_provider.force_flush()` at the end of each invocation and don't call `shutdown()`.

</LanguageTab>
<LanguageTab id="typescript">

```ts
try {
	await handleMessage(conversationId, "What's the weather in Berlin?")
} finally {
	await provider.forceFlush()
	await provider.shutdown()
}
```

</LanguageTab>
</LanguageTabs>

## Check that it works

Run a conversation of two or three messages, including one that calls a tool, then open **Agent Sessions** in Maple. You should see one session per conversation id with framework **Strands Agents**, one turn per `agent(...)` call, and a transcript with the user messages, replies and tool calls.

Cost shows as unpriced because Strands doesn't report it.

## Troubleshooting

- **Transcript is empty, but tokens and tools show up.** Add `gen_ai_span_attributes_only` and `gen_ai_latest_experimental` to `OTEL_SEMCONV_STABILITY_OPT_IN`, set before the first `Agent` is created.
- **Every message is its own session.** Pass `trace_attributes={"session.id": conversation_id}` to the agent, or to the `Swarm` or `Graph` that runs it.
- **Two users' messages land in one session.** A shared `Agent` carries one `trace_attributes` dict for everyone. Create the agent per request.
- **Every model call appears twice.** Another instrumentation (OpenLIT, OpenLLMetry, OpenInference, an OpenAI or Bedrock instrumentor) wraps the same calls. Remove it.
- **Nothing arrives from a script.** The process exited before the batch was exported. Call `force_flush()` and `shutdown()` in a `finally` block.

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview)
- [Strands Agents traces documentation](https://strandsagents.com/docs/user-guide/observability-evaluation/traces/)

---

# Trace Hugging Face smolagents with OpenTelemetry

> AI Frameworks · https://maple.dev/docs/agent-tracing/smolagents

Send smolagents runs to Maple through the OpenInference instrumentor so each conversation shows up as one Agent Session.

smolagents is traced with OpenInference's `openinference-instrumentation-smolagents`. Turn on its GenAI attributes and wrap each run in `using_session(...)`. Without `using_session(...)`, every message becomes its own session.

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing-smolagents](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-smolagents) skill and follows it.

```text
Set up Maple agent tracing for smolagents in this project.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing-smolagents -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## Install the instrumentor and export to Maple

```bash
pip install "smolagents[openai]>=1.26" "openinference-instrumentation-smolagents>=0.1.40" \
  "opentelemetry-sdk>=1.45" "opentelemetry-exporter-otlp-proto-http>=1.45"
```

Use `[litellm]` instead of `[openai]` if you run `LiteLLMModel`. Skip the `smolagents[telemetry]` extra, which installs the Arize Phoenix server.

Point the exporter at Maple. For an EU organization, use `https://ingest.eu.maple.dev`. Set the base URL only; the exporter appends `/v1/traces`.

```bash
export OTEL_SERVICE_NAME=support-agent
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production
export OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
```

Add a `tracing.py` and import it at the top of your entry point, before the first `agent.run()`:

```py
# tracing.py
import json

from openinference.instrumentation import TraceConfig
from openinference.instrumentation.smolagents import SmolagentsInstrumentor
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import SpanProcessor, TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor


class SmolagentsForMaple(SpanProcessor):
    """Fixes agent names, tool names and tool arguments for Maple."""

    def on_start(self, span, parent_context=None):
        if span.instrumentation_scope.name != "openinference.instrumentation.smolagents":
            return
        attrs = span.attributes
        if span.name.endswith(".run"):
            span.set_attribute("gen_ai.agent.name", span.name.removesuffix(".run"))
        elif "tool.name" in attrs:
            span.update_name(f"execute_tool {attrs['tool.name']}")
            if attrs.get("input.value", "").startswith("{"):
                call = json.loads(attrs["input.value"])
                span.set_attribute("gen_ai.tool.call.arguments", json.dumps(call["kwargs"] or call["args"]))


provider = TracerProvider()
provider.add_span_processor(SmolagentsForMaple())
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

SmolagentsInstrumentor().instrument(
    tracer_provider=provider,
    config=TraceConfig(enable_genai_semconv=True),
)
```

Copy `SmolagentsForMaple` as is. It gives each agent its own lane and fixes tool names and arguments.

If your app already has a `TracerProvider` (from `opentelemetry-instrument`, Logfire or another library), add `SmolagentsForMaple()` and the exporter to that provider and pass it to `instrument()` instead of creating a second one.

## Wrap each run in the conversation id

Wrap every `agent.run()` in OpenInference's `using_session` with the id your app already stores the chat under:

```py
from openinference.instrumentation import using_session
from smolagents import OpenAIServerModel, ToolCallingAgent

# One agent per conversation. In a long-running server, evict idle ones or rebuild them from stored history.
agents: dict[str, ToolCallingAgent] = {}


def handle_message(conversation_id: str, text: str) -> str:
    agent = agents.get(conversation_id)
    if agent is None:
        agent = agents[conversation_id] = ToolCallingAgent(
            tools=[get_weather, calculate],
            model=OpenAIServerModel(model_id="gpt-4o-mini"),
            name="assistant",
        )
    with using_session(conversation_id):
        return str(agent.run(text, reset=False))
```

The id must stay the same across the conversation and differ between conversations.

Keep one agent object per conversation, as above. A single shared agent with `reset=False` mixes every user's memory into one conversation.

Give every agent, including managed agents, a `name`. Unnamed agents share one lane.

## Flush before the process exits

Serverless functions, killed workers and notebooks don't exit normally, so flush after each run:

```py
from tracing import provider

try:
    handle_message("conv-42", "What's the weather in Berlin?")
finally:
    provider.force_flush()
```

## Check that it works

Run two or three messages through `handle_message` with the same conversation id, including one that uses a tool, then open **Agent Sessions**. You should see one session with framework **smolagents**, one turn per `agent.run()`, a transcript, `OpenAIModel.generate` model calls with token counts, and `execute_tool <name>` tool calls. Each run ends with an `execute_tool final_answer` call.

Cost shows as unpriced.

## Troubleshooting

- **One session per message.** Wrap every `agent.run()` in `using_session(...)` with the stored conversation id.
- **Exports fail with 404.** `OTLPSpanExporter(endpoint=...)` doesn't append `/v1/traces`. Use the environment variable with the base URL, or pass the full path.
- **Every model call appears twice.** Remove `openinference-instrumentation-openai` or `openinference-instrumentation-litellm`; the smolagents instrumentor already covers model calls.
- **No tool spans with `CodeAgent`.** A remote executor (`e2b`, `docker`, `modal`) runs tools outside your process. Only the local executor produces tool spans.

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview)
- [LiteLLM](/docs/agent-tracing/litellm) and [OpenRouter](/docs/agent-tracing/openrouter), if your models go through either gateway

---

# Trace Agno agents and teams with OpenTelemetry

> AI Frameworks · https://maple.dev/docs/agent-tracing/agno

Send Agno agent, team and tool spans to Maple as one Agent Session per conversation, with transcript, tokens and failed tools.

This guide sends Agno's OpenInference spans to Maple with an OTLP exporter. Agno's own `setup_tracing()` writes to your AgentOS database, not to Maple.

Pass `session_id` on every run. Without it, a server with one shared `Agent` puts every user's conversation into the same Maple session.

Tested with Agno 3.0.11 and `openinference-instrumentation-agno` 1.0.10 on Python 3.10 to 3.14.

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing-agno](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-agno) skill and follows it.

```text
Set up Maple agent tracing for Agno in this project.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing-agno -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## Install and point the exporter at Maple

```bash
pip install -U "agno>=3.0" "openinference-instrumentation-agno>=1.0.10" \
  opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
```

```bash
export OTEL_SERVICE_NAME=support-agent
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production
export OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export AGNO_TELEMETRY=false
```

EU organizations use `https://ingest.eu.maple.dev`. Set the base URL only; the exporter appends `/v1/traces`. `AGNO_TELEMETRY=false` turns off Agno's anonymous usage pings.

## Initialize tracing

Create one tracer provider in `tracing.py` and import it at the top of your entry point, before you build agents or an `AgentOS`:

```py
# tracing.py
from openinference.instrumentation import TraceConfig
from openinference.instrumentation.agno import AgnoInstrumentor
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

AgnoInstrumentor().instrument(
    tracer_provider=provider,
    config=TraceConfig(enable_genai_semconv=True),
)
```

If you can't change the `instrument()` call, set `OPENINFERENCE_ENABLE_GENAI_SEMCONV=true` instead.

### Keep the AgentOS traces view

Once `tracing.py` runs, `AgentOS(tracing=True)` and `setup_tracing(db=...)` stop filling the AgentOS traces view. To keep it, add Agno's database exporter to your provider, with the same `db` you give AgentOS:

```py
from agno.tracing.exporter import DatabaseSpanExporter

provider.add_span_processor(BatchSpanProcessor(DatabaseSpanExporter(db=db)))
```

## Pass session_id on every run

Each `run()` or `arun()` is its own trace. Pass `session_id` to group them into one session:

```py
from agno.agent import Agent
from agno.db.sqlite import SqliteDb
from agno.models.openrouter import OpenRouter

# Built once at import time and shared by every request
agent = Agent(
    name="support_agent",
    model=OpenRouter(id="openai/gpt-4o-mini"),
    db=SqliteDb(db_file="tmp/agno.db"),
    tools=[get_weather, calculate],
    add_history_to_context=True,
)


def chat(conversation_id: str, user_id: str, message: str) -> str:
    response = agent.run(message, session_id=conversation_id, user_id=user_id)
    return response.content


async def chat_stream(conversation_id: str, user_id: str, message: str):
    async for event in agent.arun(
        message, stream=True, session_id=conversation_id, user_id=user_id
    ):
        if getattr(event, "content", None):
            yield event.content
```

Use the conversation id your app already stores, and pass it to `team.run()`, workflows and `continue_run()` as well.

## Teams, failed tools and content

Give every `Agent` and `Team` a `name=`. Unnamed ones show up as `Agent.run` or `Team.run` with no lane of their own.

A tool that raises is marked failed. A tool that returns an error string counts as a success.

To keep content out of Maple, set `OPENINFERENCE_HIDE_INPUT_MESSAGES`, `OPENINFERENCE_HIDE_OUTPUT_MESSAGES`, `OPENINFERENCE_HIDE_INPUTS` and `OPENINFERENCE_HIDE_OUTPUTS` to `true` before the instrumentor starts. Tool arguments are still exported; removing them needs a `redaction` processor in an OpenTelemetry Collector.

## Flush before short-lived processes exit

Scripts, notebooks, workers and serverless handlers need an explicit flush:

```py
from tracing import provider

try:
    agent.run("Summarize today's tickets", session_id=conversation_id)
finally:
    provider.force_flush()  # serverless: flush at the end of every invocation
    provider.shutdown()     # scripts: flush and stop at the end of the process
```

## Check that it works

Run a conversation of two or three turns with one `session_id`, including a tool call, and open **Agent Sessions** filtered by your service name. You should see one session with your `session_id` as its id, framework **Agno**, one turn per `run()`, a transcript, and model calls such as `OpenRouter.invoke` with tokens. A session named `trace:...` means the run had no `session_id`. Cost shows in the sessions list only when your provider returns one, as OpenRouter does.

## Troubleshooting

- **Nothing arrives in Maple.** `setup_tracing()` or `AgentOS(tracing=True)` ran before `tracing.py` and the spans went to the AgentOS database. Import `tracing` first.
- **Every user is in one giant session.** The shared `Agent` runs without `session_id=`. Pass it on every `run()`, `arun()` and `continue_run()`.
- **Every turn is its own session.** You pass a new id per request, often a fresh `uuid4()`. Use the stored conversation id.
- **An agent run lands inside the previous team run's trace.** In scripts and workers, run each team run in its own context with `asyncio.create_task(...)` or `contextvars.copy_context().run(...)`.
- **Every model call appears twice.** Remove the second instrumentor, usually `openinference-instrumentation-openai`, OpenLIT or Phoenix's `register(auto_instrument=True)`.

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview)

---

# Trace DSPy programs and ReAct agents with OpenTelemetry

> AI Frameworks · https://maple.dev/docs/agent-tracing/dspy

Send DSPy programs to Maple as one Agent Session per conversation, with transcript, model and tool calls, tokens and cost.

This guide traces DSPy with OpenInference's DSPy instrumentor, adds tokens and tool names with a small DSPy callback, and groups each conversation into one session with `using_session`.

Tested with DSPy 3.4 and `openinference-instrumentation-dspy` 0.1.45 on Python 3.10 or later.

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing-dspy](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-dspy) skill and follows it.

```text
Set up Maple agent tracing for DSPy in this project.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing-dspy -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## Install and point the exporter at Maple

```bash
pip install "dspy>=3.4" "openinference-instrumentation-dspy>=0.1.45" "openinference-instrumentation>=0.1.66" \
  "opentelemetry-sdk>=1.45" "opentelemetry-exporter-otlp-proto-http>=1.45" \
  "opentelemetry-instrumentation-threading>=0.66b0"
```

```bash
export OTEL_SERVICE_NAME=support-agent
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production
export OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
```

EU organizations use `https://ingest.eu.maple.dev`. Set the base URL only; the exporter appends `/v1/traces`.

## Initialize tracing

Add a `tracing.py` and import it at the top of your entry point, before your DSPy modules are defined:

```py
# tracing.py
from openinference.instrumentation import TraceConfig
from openinference.instrumentation.dspy import DSPyInstrumentor
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.threading import ThreadingInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider()  # reads OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

DSPyInstrumentor().instrument(tracer_provider=provider, config=TraceConfig(enable_genai_semconv=True))
ThreadingInstrumentor().instrument()
```

`ThreadingInstrumentor` keeps `dspy.Parallel` and thread pool workers in the caller's session.

If the app already has a `TracerProvider` (from `opentelemetry-instrument`, Logfire or another library), add the OTLP exporter to it and pass it to `instrument()` instead of creating a second one.

## Add the Maple callback

The callback adds tokens, cost, tool names and agent spans. Save it as `maple_dspy.py`:

```py
# maple_dspy.py
import json

import dspy
from dspy.utils.callback import BaseCallback
from openinference.instrumentation import TraceConfig
from opentelemetry import trace

_config = TraceConfig()


def _message(role, values):
    text = "\n".join(v for v in values if isinstance(v, str))
    return json.dumps([{"role": role, "parts": [{"type": "text", "content": text}]}]) if text else None


class MapleCallback(BaseCallback):
    def __init__(self):
        self._agents = set()
        self._lms = {}

    def on_module_start(self, call_id, instance, inputs):
        if type(instance).__module__.startswith("dspy."):
            return  # Predict, ChainOfThought, ReAct: building blocks, not agents
        self._agents.add(call_id)
        span = trace.get_current_span()
        span.set_attribute("gen_ai.operation.name", "invoke_agent")
        span.set_attribute("gen_ai.agent.name", type(instance).__name__)
        user = _message("user", [*inputs.get("args", ()), *inputs.get("kwargs", {}).values()])
        if user and not _config.hide_inputs:
            span.set_attribute("gen_ai.input.messages", user)

    def on_module_end(self, call_id, outputs, exception):
        if call_id not in self._agents:
            return
        self._agents.discard(call_id)
        if isinstance(outputs, dspy.Prediction) and not _config.hide_outputs:
            reply = _message("assistant", [v for k, v in outputs.items() if k != "reasoning"])
            if reply:
                trace.get_current_span().set_attribute("gen_ai.output.messages", reply)

    def on_tool_start(self, call_id, instance, inputs):
        span = trace.get_current_span()
        span.set_attribute("gen_ai.tool.name", instance.name)
        span.set_attribute("gen_ai.tool.description", instance.desc or "")
        if not _config.hide_inputs:
            span.set_attribute("gen_ai.tool.call.arguments", json.dumps(inputs.get("kwargs", {}), default=str))

    def on_lm_start(self, call_id, instance, inputs):
        self._lms[call_id] = instance

    def on_lm_end(self, call_id, outputs, exception):
        lm = self._lms.pop(call_id, None)
        entry = next((e for e in reversed(lm.history[-16:]) if e["outputs"] is outputs), None) if lm else None
        if entry is None or getattr(entry["response"], "cache_hit", False):
            return  # history is off, or a cache hit that cost nothing
        usage = entry["usage"] or {}
        attributes = {
            "gen_ai.response.id": getattr(entry["response"], "id", None),
            "gen_ai.response.model": entry.get("response_model"),
            "gen_ai.usage.input_tokens": usage.get("prompt_tokens"),
            "gen_ai.usage.output_tokens": usage.get("completion_tokens"),
            "gen_ai.usage.cache_read.input_tokens": (usage.get("prompt_tokens_details") or {}).get("cached_tokens"),
            "gen_ai.usage.reasoning.output_tokens": (usage.get("completion_tokens_details") or {}).get("reasoning_tokens"),
            "gen_ai.usage.cost": entry.get("cost"),
        }
        trace.get_current_span().set_attributes({k: v for k, v in attributes.items() if v is not None})
```

Register it with the rest of your DSPy configuration:

```py
import tracing  # must come first

import dspy
from maple_dspy import MapleCallback

dspy.configure(lm=dspy.LM("openai/gpt-4o-mini", temperature=0), callbacks=[MapleCallback()])
```

Every `dspy.Module` subclass you write shows up as an agent named after its class, so give worker modules descriptive names like `WeatherWorker`.

## Group a conversation into one session

Each call to your module is its own trace. Wrap every call in `using_session` with the conversation id to group them into one session:

```py
import dspy
from openinference.instrumentation import using_session


class ChatAssistant(dspy.Module):
    def __init__(self):
        super().__init__()
        self.respond = dspy.ReAct(
            "question: str, conversation: dspy.History -> answer: str",
            tools=[get_weather, calculate],
        )

    def forward(self, question, conversation):
        return self.respond(question=question, conversation=conversation)


assistant = ChatAssistant()


def handle_message(conversation_id: str, question: str, turns: list[dict]) -> str:
    with using_session(conversation_id):
        answer = assistant(question=question, conversation=dspy.History(messages=turns)).answer
    turns.append({"question": question, "answer": answer})
    return answer
```

Use the id your app stores the chat under. A new UUID per request gives one session per message, and a constant puts every user in one session.

If you stream with `dspy.streamify`, call it after `dspy.configure(callbacks=[MapleCallback()])`, and put `using_session` around the loop that reads the stream.

To keep prompts and outputs out of your traces, set `OPENINFERENCE_HIDE_INPUTS=true` and `OPENINFERENCE_HIDE_OUTPUTS=true` before `tracing.py` runs. The callback follows both.

## Flush before short-lived processes exit

Serverless handlers, notebooks and processes that get killed need an explicit flush:

```py
from tracing import provider

try:
    handle_message("conv-42", "What's the weather in Berlin?", turns=[])
finally:
    provider.force_flush()  # serverless: before returning; notebooks: after each run
```

## Check that it works

Set `cache=False` on the LM, then run two or three messages through `handle_message` with one conversation id, including one tool call. In **Agent Sessions** you should see one session with framework **DSPy**, one turn per call, and `LM.__call__` model calls with tokens. `finish.__call__` is `dspy.ReAct`'s built-in end-of-loop tool.

## Troubleshooting

- **No spans, or exports fail with 404.** Import `tracing` first. An `endpoint=` passed in code must end in `/v1/traces`; the env variable takes the base URL.
- **No tokens anywhere.** `MapleCallback` isn't registered, a later `dspy.configure(callbacks=[...])` replaced it, or the calls were cache hits.
- **One session per message.** The call runs outside `using_session(...)`, or the id changes per request.
- **Every model call counted twice.** Remove `openinference-instrumentation-litellm` or `-openai`; the callback already records model calls.
- **`TypeError: object of type 'History' has no len()`.** A module attribute is named `history`. Rename it, or pass the `dspy.History` as an input field.

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview)
- [LiteLLM](/docs/agent-tracing/litellm) and [OpenRouter](/docs/agent-tracing/openrouter): if your models go through either gateway.

---

# Trace Haystack agents with OpenTelemetry

> AI Frameworks · https://maple.dev/docs/agent-tracing/haystack

Send Haystack 3 Agent and pipeline runs to Maple as one Agent Session per conversation, with transcript, model, tokens, cost and failed tool calls.

This guide adds `maple_haystack.py`, a Haystack tracer that records model, tokens, messages and failed tool calls for Maple, and a `conversation()` block that groups runs into one session.

Tested with `haystack-ai` 3.2 and `opentelemetry-haystack` 1.0 on Python 3.10 or later.

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing-haystack](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-haystack) skill and follows it.

```text
Set up Maple agent tracing for Haystack in this project.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing-haystack -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## Install and add the Maple tracer

```bash
pip install "haystack-ai>=3.2" "opentelemetry-haystack>=1.0" \
  "opentelemetry-sdk>=1.45" "opentelemetry-exporter-otlp-proto-http>=1.45"
```

Save this file next to your app as `maple_haystack.py`:

```py
import json
import logging
from collections.abc import Iterator
from contextlib import contextmanager
from contextvars import ContextVar
from typing import Any

from haystack.dataclasses import ChatMessage
from haystack_integrations.tracing.opentelemetry import OpenTelemetrySpan, OpenTelemetryTracer
from opentelemetry import trace
from opentelemetry.trace import StatusCode

logger = logging.getLogger(__name__)
_conversation_id: ContextVar[str | None] = ContextVar("maple_conversation_id", default=None)


@contextmanager
def conversation(conversation_id: str) -> Iterator[None]:
    token = _conversation_id.set(conversation_id)
    try:
        yield
    finally:
        _conversation_id.reset(token)


def _messages(messages: list[ChatMessage]) -> str:
    out = []
    for m in messages:
        parts: list[dict[str, Any]] = [{"type": "text", "content": t} for t in m.texts]
        parts += [{"type": "tool_call", "id": c.id, "name": c.tool_name, "arguments": c.arguments} for c in m.tool_calls]
        parts += [{"type": "tool_call_response", "id": r.origin.id, "response": r.result} for r in m.tool_call_results]
        out.append({"role": m.role.value, "parts": parts})
    return json.dumps(out, default=str)


class MapleSpan(OpenTelemetrySpan):
    def __init__(self, span: trace.Span, operation: str | None, content: bool) -> None:
        super().__init__(span)
        self._operation = operation
        self._content = content

    def set_content_tag(self, key: str, value: Any) -> None:
        try:
            if self._operation == "chat":
                self._chat(key, value)
            elif self._operation == "execute_tool":
                self._tool(key, value)
        except Exception:  # a tracing bug must never fail the agent run
            logger.exception("maple_haystack: could not map %s", key)
        if self._content:
            self.set_tag(key, value)

    def _chat(self, key: str, value: Any) -> None:
        if key.endswith(".input") and self._content:
            messages = value["messages"]
            system = [{"type": "text", "content": m.text} for m in messages if m.is_from("system")]
            if system:
                self._span.set_attribute("gen_ai.system_instructions", json.dumps(system))
            self._span.set_attribute("gen_ai.input.messages", _messages([m for m in messages if not m.is_from("system")]))
        elif key.endswith(".output"):
            replies = value["replies"]
            meta = replies[0].meta
            usage = meta.get("usage") or {}
            prompt_details = usage.get("prompt_tokens_details") or {}
            attributes = {
                "gen_ai.response.model": meta.get("model"),
                "gen_ai.response.finish_reasons": meta.get("finish_reason"),
                "gen_ai.usage.input_tokens": usage.get("prompt_tokens"),
                "gen_ai.usage.output_tokens": usage.get("completion_tokens"),
                "gen_ai.usage.cache_read.input_tokens": prompt_details.get("cached_tokens"),
                "gen_ai.usage.cache_write.input_tokens": prompt_details.get("cache_write_tokens"),
                "gen_ai.usage.reasoning.output_tokens": (usage.get("completion_tokens_details") or {}).get("reasoning_tokens"),
                "gen_ai.usage.cost": usage.get("cost"),
            }
            if self._content:
                attributes["gen_ai.output.messages"] = _messages(replies)
            self._span.set_attributes({k: v for k, v in attributes.items() if v is not None})

    def _tool(self, key: str, value: Any) -> None:
        if key.endswith(".output") and isinstance(value, dict) and "error" in value:
            self._span.set_status(StatusCode.ERROR, str(value["error"]) if self._content else "Tool invocation failed")
            self._span.set_attribute("error.type", "ToolInvocationError")
        if self._content:
            attribute = "gen_ai.tool.call.arguments" if key.endswith(".input") else "gen_ai.tool.call.result"
            self._span.set_attribute(attribute, value if isinstance(value, str) else json.dumps(value, default=str))


class MapleHaystackTracer(OpenTelemetryTracer):
    def __init__(self, tracer: trace.Tracer, *, content: bool = True) -> None:
        super().__init__(tracer)
        self._content = content

    @contextmanager
    def trace(self, operation_name: str, tags: dict[str, Any] | None = None, parent_span: Any = None) -> Iterator[MapleSpan]:
        tags = dict(tags or {})
        attributes: dict[str, str] = {}
        operation = None
        if operation_name == "haystack.agent.run":
            parent = getattr(trace.get_current_span(), "attributes", None) or {}
            attributes["gen_ai.operation.name"] = "invoke_agent"
            attributes["gen_ai.agent.name"] = parent.get("haystack.component.name") or parent.get("gen_ai.tool.name") or "agent"
        elif operation_name == "haystack.agent.step.llm" or str(tags.get("haystack.component.type", "")).endswith("ChatGenerator"):
            operation = attributes["gen_ai.operation.name"] = "chat"
        elif operation_name == "haystack.agent.step.tool":
            operation = attributes["gen_ai.operation.name"] = "execute_tool"
            attributes["gen_ai.tool.name"] = tags["haystack.tool.name"]
        if conversation_id := _conversation_id.get():
            attributes["gen_ai.conversation.id"] = conversation_id
        if not self._content:
            tags.pop("haystack.pipeline.input_data", None)

        with self._tracer.start_as_current_span(operation_name, attributes=attributes) as raw_span:
            span = MapleSpan(raw_span, operation, self._content)
            span.set_tags(tags)
            yield span
```

## Export to Maple

Configure OpenTelemetry once at startup and hand Haystack the tracer:

```py
# telemetry.py
from haystack import tracing
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

from maple_haystack import MapleHaystackTracer

provider = TracerProvider(
    resource=Resource.create({"service.name": "support-agent", "deployment.environment.name": "production"})
)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

tracing.enable_tracing(MapleHaystackTracer(trace.get_tracer("haystack")))
```

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
```

EU organizations use `https://ingest.eu.maple.dev`. Set the base URL only; the exporter appends `/v1/traces`.

Import `telemetry` before the first `pipeline.run()` or `agent.run()`. Keep the tracer name `"haystack"`, since Maple labels the sessions by it. If the app already has a `TracerProvider`, skip the provider lines and pass `trace.get_tracer("haystack")` from the existing one.

## Group turns into one session

Each run is its own trace. Wrap every run in `conversation()` with your chat's id:

```py
from haystack import Pipeline
from haystack.components.agents import Agent
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.openrouter import OpenRouterChatGenerator

from maple_haystack import conversation

chat_generator = OpenRouterChatGenerator(model="openai/gpt-4o-mini")
agent = Agent(chat_generator=chat_generator, tools=[get_weather, calculate], system_prompt=SYSTEM_PROMPT)
pipeline = Pipeline()
pipeline.add_component("assistant", agent)


def handle_message(chat_id: str, history: list[ChatMessage], text: str) -> list[ChatMessage]:
    with conversation(chat_id):
        result = pipeline.run({"assistant": {"messages": [*history, ChatMessage.from_user(text)]}})
    # The Agent re-adds its system prompt on every run, so don't store it
    return [m for m in result["assistant"]["messages"] if not m.is_from("system")]
```

Use the id your app already stores for the chat. A new UUID per request gives one session per message, and one id for the whole process merges every user into one session.

## Agent names, content and tokens

Each agent is named after its pipeline component (`assistant` above) or the `name=` of the `AgentTool` that wraps it. An Agent run directly with `agent.run()` is called `agent`.

To keep prompts and responses out of Maple, pass `content=False`:

```py
tracing.enable_tracing(MapleHaystackTracer(trace.get_tracer("haystack"), content=False))
```

`HAYSTACK_CONTENT_TRACING_ENABLED` has no effect with this tracer.

A streamed `OpenAIChatGenerator` reply carries no token counts unless you ask for them (OpenRouter always sends usage):

```py
OpenAIChatGenerator(model="gpt-4o-mini", generation_kwargs={"stream_options": {"include_usage": True}})
```

Cost only appears with `OpenRouterChatGenerator`. Other providers show tokens and read as **unpriced**.

## Flush before short-lived processes exit

Scripts, notebooks, cron jobs and serverless handlers need an explicit flush before they exit:

```py
try:
    handle_message(chat_id, history, text)
finally:
    provider.force_flush()
    provider.shutdown()
```

Long-running servers only need `provider.shutdown()` in their shutdown hook.

## Check that it works

Run a conversation of at least two turns, one with a tool call, and open **Agent Sessions** filtered by your service name. You should see one session per conversation id, labelled Haystack, with one turn per `pipeline.run()`, model calls with tokens, and tool calls with failed ones marked.

## Troubleshooting

- **Every request is its own session.** Wrap the `pipeline.run()` call itself in `conversation()`, and pass the chat's stored id.
- **Spans but no model calls, tokens or transcript.** `enable_tracing()` got the plain `OpenTelemetryTracer`, or a later call replaced `MapleHaystackTracer`.
- **No Haystack spans at all.** Haystack 3 doesn't trace until you call `tracing.enable_tracing(...)`. Call it before the first run.
- **Every model call appears twice.** Remove `openinference-instrumentation-haystack` or OpenLLMetry's `opentelemetry-instrumentation-haystack`.
- **Tokens are zero with a non-OpenAI generator.** Print `result["replies"][0].meta["usage"]` once and add its keys to `_chat()`.
- **A failed tool isn't counted.** The tool caught its own exception. Let it raise, or return `{"error": ...}`.

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview)

---

# Trace Microsoft Agent Framework and Semantic Kernel agents with OpenTelemetry

> AI Frameworks · https://maple.dev/docs/agent-tracing/microsoft-agent-framework

Send Microsoft Agent Framework and Semantic Kernel traces from Python or .NET to Maple as one Agent Session per conversation.

import LanguageTabs from "../../../components/docs/LanguageTabs.astro"
import LanguageTab from "../../../components/docs/LanguageTab.astro"

Microsoft Agent Framework (MAF) emits OpenTelemetry spans in Python and .NET. You point them at Maple and add a short span processor that sets the conversation id, or every turn shows up as its own session.

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing-microsoft-agent-framework](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-microsoft-agent-framework) skill and follows it.

```text
Set up Maple agent tracing for Microsoft Agent Framework in this project.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing-microsoft-agent-framework -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## Export traces

<LanguageTabs label="Language" tabs={[{ id: "python", label: "Python" }, { id: "csharp", label: ".NET" }]}>
<LanguageTab id="python">

Install MAF with the OTLP/HTTP exporter:

```bash
pip install "agent-framework-core>=1.19.0" "agent-framework-openai>=1.14.4" opentelemetry-exporter-otlp-proto-http
```

Call `configure_otel_providers()` once at startup, before you create agents. Without `enable_sensitive_data=True` the transcript is empty.

```py
# telemetry.py
import logging
import os

from agent_framework.observability import configure_otel_providers
from opentelemetry import trace

from maple_tracing import ConversationIdProcessor

key = os.environ.get("MAPLE_INGEST_KEY")
if key:
    configure_otel_providers(
        service_name="support-agent",
        resource_attributes={"deployment.environment.name": "production"},
        otlp_endpoint="https://ingest.maple.dev",  # EU: https://ingest.eu.maple.dev
        otlp_protocol="http/protobuf",
        otlp_headers={"Authorization": f"Bearer {key}"},
        enable_sensitive_data=True,   # prompts, replies, tool arguments and results
        enable_message_events=False,  # skip the duplicate copy of the content in OTLP logs
    )
    trace.get_tracer_provider().add_span_processor(ConversationIdProcessor())
else:
    # A missing key disables export; it never stops the app.
    logging.getLogger(__name__).warning("MAPLE_INGEST_KEY is not set; Maple telemetry export is disabled")
```

Keep `otlp_protocol="http/protobuf"`. MAF defaults to gRPC, which Maple doesn't accept.

If your app already has a `TracerProvider`, don't call `configure_otel_providers()`. Add Maple's exporter and `ConversationIdProcessor` to your provider, then call `enable_instrumentation(enable_sensitive_data=True, enable_message_events=False)` from `agent_framework.observability`.

</LanguageTab>
<LanguageTab id="csharp">

```bash
dotnet add package Microsoft.Agents.AI --version 1.22.0
dotnet add package Microsoft.Agents.AI.OpenAI --version 1.22.0
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol --version 1.19.1
```

```csharp
using System.ClientModel;
using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;
using OpenAI;
using OpenTelemetry;
using OpenTelemetry.Exporter;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

using var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .ConfigureResource(r => r.AddService("support-agent"))
    .AddSource("*Microsoft.Agents.AI*")    // agent, chat and workflow spans
    .AddSource("*Microsoft.Extensions.AI") // chat clients you instrument yourself
    .AddProcessor(new ConversationIdProcessor())
    .AddOtlpExporter(o =>
    {
        o.Endpoint = new Uri("https://ingest.maple.dev/v1/traces"); // EU: ingest.eu.maple.dev
        o.Protocol = OtlpExportProtocol.HttpProtobuf;
        o.Headers = "Authorization=Bearer YOUR_INGEST_KEY";
    })
    .Build();

var openAi = new OpenAIClient(
    new ApiKeyCredential(Environment.GetEnvironmentVariable("OPENAI_API_KEY")!));

AIAgent agent = openAi.GetChatClient("gpt-4o-mini").AsIChatClient()
    .AsAIAgent(
        instructions: "You are a helpful assistant.",
        name: "support_agent",
        tools: [AIFunctionFactory.Create(GetWeather, name: "get_weather")])
    .AsBuilder()
    .UseOpenTelemetry(configure: a => a.EnableSensitiveData = true)
    .Build();
```

Keep the leading `*` in `AddSource` (the source names start with `Experimental.`), `/v1/traces` in the endpoint, and the `HttpProtobuf` line. Pass each tool a `name`, or local functions show up under compiler-generated names.

</LanguageTab>
</LanguageTabs>

## Group each conversation into one session

Maple groups turns into a session by `gen_ai.conversation.id`.

<LanguageTabs label="Language" tabs={[{ id: "python", label: "Python" }, { id: "csharp", label: ".NET" }]}>
<LanguageTab id="python">

This processor sets it on every span started inside a `conversation()` block:

```py
# maple_tracing.py
from contextlib import contextmanager
from contextvars import ContextVar

from opentelemetry.sdk.trace import SpanProcessor

_conversation_id: ContextVar[str | None] = ContextVar("conversation_id", default=None)


class ConversationIdProcessor(SpanProcessor):
    """Puts gen_ai.conversation.id on every span started inside `conversation()`."""

    def on_start(self, span, parent_context=None):
        if (conversation_id := _conversation_id.get()) is not None:
            span.set_attribute("gen_ai.conversation.id", conversation_id)


@contextmanager
def conversation(conversation_id: str):
    token = _conversation_id.set(conversation_id)
    try:
        yield
    finally:
        _conversation_id.reset(token)
```

Wrap each request in it. Use `session.session_id` as the id, or create the session with your own chat id: `agent.create_session(session_id=chat_id)`.

```py
from agent_framework import Agent, AgentSession

from maple_tracing import conversation


async def handle_message(agent: Agent, session: AgentSession, text: str) -> str:
    with conversation(session.session_id):
        response = await agent.run(text, session=session)
    return response.text
```

When streaming, keep the whole `async for` loop inside the block. Don't pass the `conversation_id` chat option instead; it turns off MAF's in-memory history.

</LanguageTab>
<LanguageTab id="csharp">

In .NET, use an `Activity` processor with an `AsyncLocal` and set it before `RunAsync`:

```csharp
using System.Diagnostics;
using OpenTelemetry;

sealed class ConversationIdProcessor : BaseProcessor<Activity>
{
    public static readonly AsyncLocal<string?> Current = new();

    public override void OnStart(Activity activity)
    {
        if (Current.Value is { } id) activity.SetTag("gen_ai.conversation.id", id);
    }
}
```

```csharp
AgentSession session = await agent.CreateSessionAsync();
ConversationIdProcessor.Current.Value = chatId; // once per request, before RunAsync
var response = await agent.RunAsync(userMessage, session);
```

</LanguageTab>
</LanguageTabs>

## Flush before a script exits

Scripts, CLIs and notebooks that exit without flushing lose their last turns.

<LanguageTabs label="Language" tabs={[{ id: "python", label: "Python" }, { id: "csharp", label: ".NET" }]}>
<LanguageTab id="python">

Shut the providers down in a `finally`:

```py
from opentelemetry import _logs, metrics, trace


def shutdown_telemetry() -> None:
    for provider in (trace.get_tracer_provider(), metrics.get_meter_provider(), _logs.get_logger_provider()):
        provider.shutdown()


try:
    asyncio.run(main())
finally:
    shutdown_telemetry()
```

In a serverless handler, call `trace.get_tracer_provider().force_flush()` before returning.

</LanguageTab>
<LanguageTab id="csharp">

In .NET, `using var tracerProvider` flushes when `Main` ends.

</LanguageTab>
</LanguageTabs>

## Semantic Kernel

Semantic Kernel (SK) reads its telemetry switches at import time, so set them before the first `import semantic_kernel`. Configure your own provider with the same `ConversationIdProcessor`:

```bash
pip install "semantic-kernel>=1.44.1" opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
```

```py
# telemetry.py: import this before anything that imports semantic_kernel
import logging
import os

os.environ["SEMANTICKERNEL_EXPERIMENTAL_GENAI_ENABLE_OTEL_DIAGNOSTICS"] = "true"
os.environ["SEMANTICKERNEL_EXPERIMENTAL_GENAI_ENABLE_OTEL_DIAGNOSTICS_SENSITIVE"] = "true"

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

from maple_tracing import ConversationIdProcessor

provider = TracerProvider(resource=Resource.create({"service.name": "support-agent"}))
provider.add_span_processor(ConversationIdProcessor())
key = os.environ.get("MAPLE_INGEST_KEY")
if key:
    provider.add_span_processor(
        BatchSpanProcessor(
            OTLPSpanExporter(
                endpoint="https://ingest.maple.dev/v1/traces",  # EU: https://ingest.eu.maple.dev/v1/traces
                headers={"Authorization": f"Bearer {key}"},
            )
        )
    )
else:
    # A missing key disables export; it never stops the app.
    logging.getLogger(__name__).warning("MAPLE_INGEST_KEY is not set; Maple telemetry export is disabled")
trace.set_tracer_provider(provider)
```

Wrap each turn in `with conversation(thread.id):`. Pass messages positionally, as in `await agent.get_response(text, thread=thread)`; the `messages=` keyword records an empty input. Only `ChatCompletionAgent` calls produce a transcript; calling the kernel directly doesn't.

## Check that it works

Run a conversation of two or three turns where one turn calls a tool. Within about a minute, **Agent Sessions** shows one session for it, labeled **Microsoft Agent Framework** or **Semantic Kernel**, with one turn per `agent.run()`, the transcript, and tool calls with their arguments and results. Cost shows as unpriced; MAF doesn't emit cost.

## Troubleshooting

- **Nothing arrives, or `ImportError: opentelemetry-exporter-otlp-proto-grpc is required`.** The protocol defaulted to gRPC. Set `otlp_protocol="http/protobuf"` or `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf`.
- **Every turn is its own session.** Register `ConversationIdProcessor` and wrap each call, including the whole streaming loop, in `conversation()`.
- **Spans but an empty transcript.** Sensitive data is off, or `OTEL_SEMCONV_STABILITY_OPT_IN` is set without `gen_ai_latest_experimental` (use `http,gen_ai_latest_experimental`).
- **.NET: no spans at all.** Use `AddSource("*Microsoft.Agents.AI*")` with the leading `*`.
- **Semantic Kernel: no `chat` or `invoke_agent` spans.** The `SEMANTICKERNEL_EXPERIMENTAL_GENAI_*` variables were set after `semantic_kernel` was imported.

## Related

- [Agent Sessions](/docs/agent-sessions/overview): reading a session in Maple.
- [Python instrumentation](/docs/guides/instrumentation-python) and [.NET instrumentation](/docs/guides/instrumentation-csharp): tracing the rest of the service.
- [Agent Framework observability](https://learn.microsoft.com/en-us/agent-framework/agents/observability): Microsoft's reference for the settings above.
- [Semantic Kernel telemetry](https://learn.microsoft.com/en-us/semantic-kernel/concepts/enterprise-readiness/observability/): the SK diagnostics switches.

---

# Trace Spring AI agents with OpenTelemetry

> AI Frameworks · https://maple.dev/docs/agent-tracing/spring-ai

Send Spring AI ChatClient, model and tool spans to Maple, with one session per chat memory conversation.

Spring AI already emits spans for `ChatClient`, model and tool calls, with token counts. You add Spring Boot's OpenTelemetry starter, set sampling to 100%, add one configuration class for the transcript, tool names and failed tools, and pass the conversation id on every call.

Tested with Spring AI 2.0.1 on Spring Boot 4.1.1 and Java 21. Spring AI 1.1 on Boot 3.5 works too, with different dependencies and property names listed in the [skill](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-spring-ai).

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing-spring-ai](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-spring-ai) skill and follows it.

```text
Set up Maple agent tracing for Spring AI in this project.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing-spring-ai -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## Install the OpenTelemetry starter

Next to the `spring-ai-bom` import (2.0.1) and your model starter, such as `spring-ai-starter-model-openai`, add Boot's OpenTelemetry starter:

```xml
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-opentelemetry</artifactId>
</dependency>
```

## Export to Maple

Add to `application.properties`:

```properties
spring.application.name=support-agent

management.opentelemetry.tracing.export.otlp.endpoint=https://ingest.maple.dev/v1/traces
management.opentelemetry.tracing.export.otlp.headers.Authorization=Bearer ${MAPLE_INGEST_KEY:}
management.tracing.sampling.probability=1.0
management.opentelemetry.resource-attributes.deployment.environment.name=production

# The starter also exports metrics, to localhost:4318 unless told otherwise
management.otlp.metrics.export.url=https://ingest.maple.dev/v1/metrics
management.otlp.metrics.export.headers.Authorization=Bearer ${MAPLE_INGEST_KEY:}

# Read by the configuration class below
maple.ai.capture-content=true
```

The empty default in `${MAPLE_INGEST_KEY:}` keeps the app starting when the key is missing. To avoid sending requests without a key, turn export off at the top of `main` in that case:

```java
public static void main(String[] args) {
	// A missing key disables export; it never stops the app.
	var mapleKey = System.getenv("MAPLE_INGEST_KEY");
	if (mapleKey == null || mapleKey.isEmpty()) {
		System.err.println("MAPLE_INGEST_KEY is not set; Maple telemetry export is disabled");
		System.setProperty("management.tracing.export.enabled", "false");
		System.setProperty("management.otlp.metrics.export.enabled", "false");
	}
	SpringApplication.run(Application.class, args);
}
```

Spring Boot doesn't read `.env` files, so set `MAPLE_INGEST_KEY` in the environment that starts the JVM.

For an EU organization, use `https://ingest.eu.maple.dev`. This property takes the full URL, so keep `/v1/traces` on the end. To skip metrics, replace the two metrics lines with `management.otlp.metrics.export.enabled=false`.

Keep `management.tracing.sampling.probability=1.0`. Boot's default of `0.1` silently drops nine turns out of ten.

## Add the attributes Maple reads

Add this class in a package your application scans:

```java
package com.example.agent;

import java.util.ArrayList;
import java.util.List;
import java.util.Map;

import io.micrometer.common.KeyValue;
import io.micrometer.observation.Observation;
import io.micrometer.observation.ObservationFilter;
import io.micrometer.observation.ObservationRegistry;
import tools.jackson.databind.json.JsonMapper;

import org.springframework.ai.chat.client.observation.ChatClientObservationContext;
import org.springframework.ai.chat.messages.AssistantMessage;
import org.springframework.ai.chat.messages.Message;
import org.springframework.ai.chat.messages.ToolResponseMessage;
import org.springframework.ai.chat.observation.ChatModelObservationContext;
import org.springframework.ai.tool.execution.DefaultToolExecutionExceptionProcessor;
import org.springframework.ai.tool.execution.ToolExecutionExceptionProcessor;
import org.springframework.ai.tool.observation.ToolCallingObservationContext;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration(proxyBeanMethods = false)
public class MapleAiObservationConfig {

	/** Agent name for a ChatClient: `.defaultAdvisors(a -> a.param(AGENT_NAME, "support_agent"))`. */
	public static final String AGENT_NAME = "gen_ai.agent.name";

	@Bean
	ObservationFilter mapleGenAiAttributes(@Value("${maple.ai.capture-content:false}") boolean captureContent) {
		return context -> {
			if (context instanceof ChatClientObservationContext client) {
				client.addLowCardinalityKeyValue(KeyValue.of("gen_ai.operation.name", "invoke_agent"));
				if (client.getRequest().context().get(AGENT_NAME) instanceof String agent) {
					client.addLowCardinalityKeyValue(KeyValue.of("gen_ai.agent.name", agent));
				}
			}
			else if (context instanceof ToolCallingObservationContext tool) {
				tool.addLowCardinalityKeyValue(KeyValue.of("gen_ai.tool.name", tool.getToolDefinition().name()));
				if (tool.getToolCallId() != null) {
					tool.addHighCardinalityKeyValue(KeyValue.of("gen_ai.tool.call.id", tool.getToolCallId()));
				}
				if (captureContent) {
					tool.addHighCardinalityKeyValue(KeyValue.of("gen_ai.tool.call.arguments", tool.getToolCallArguments()));
					if (tool.getToolCallResult() != null) {
						tool.addHighCardinalityKeyValue(KeyValue.of("gen_ai.tool.call.result", tool.getToolCallResult()));
					}
				}
			}
			else if (captureContent && context instanceof ChatModelObservationContext chat) {
				chat.addHighCardinalityKeyValue(KeyValue.of("gen_ai.input.messages",
						JsonMapper.shared().writeValueAsString(chat.getRequest().getInstructions().stream().map(MapleAiObservationConfig::message).toList())));
				if (chat.getResponse() != null) {
					chat.addHighCardinalityKeyValue(KeyValue.of("gen_ai.output.messages",
							JsonMapper.shared().writeValueAsString(chat.getResponse().getResults().stream().map(g -> message(g.getOutput())).toList())));
				}
			}
			return context;
		};
	}

	// Spring AI hands a failing tool's message back to the model and ends the tool span OK.
	// Mark the span failed first, then keep the default behaviour.
	@Bean
	ToolExecutionExceptionProcessor toolExecutionExceptionProcessor(ObservationRegistry registry) {
		ToolExecutionExceptionProcessor fallback = DefaultToolExecutionExceptionProcessor.builder().build();
		return exception -> {
			Observation toolCall = registry.getCurrentObservation();
			if (toolCall != null) {
				toolCall.error(exception);
			}
			return fallback.process(exception);
		};
	}

	/** One message in the OpenTelemetry GenAI shape: {role, parts: [...]}. */
	private static Map<String, Object> message(Message message) {
		List<Map<String, Object>> parts = new ArrayList<>();
		if (message instanceof ToolResponseMessage toolResponse) {
			toolResponse.getResponses().forEach(r -> parts.add(
					Map.of("type", "tool_call_response", "id", r.id(), "response", r.responseData())));
		}
		else {
			if (message.getText() != null && !message.getText().isEmpty()) {
				parts.add(Map.of("type", "text", "content", message.getText()));
			}
			if (message instanceof AssistantMessage assistant) {
				assistant.getToolCalls().forEach(call -> parts.add(
						Map.of("type", "tool_call", "id", call.id(), "name", call.name(), "arguments", call.arguments())));
			}
		}
		return Map.of("role", message.getMessageType().getValue(), "parts", parts);
	}

}
```

If your app already defines a `ToolExecutionExceptionProcessor`, add the `error()` call to it instead of adding a second bean.

The transcript comes from `maple.ai.capture-content=true`. Spring AI's own `log-prompt` and `log-completion` settings only write to the application log. Set it to `false` to keep message and tool content out of Maple; everything else still shows up, with an empty transcript. A failed tool's exception message is still sent.

## Group turns into one session

Pass the `ChatMemory.CONVERSATION_ID` advisor parameter on every request. Spring AI writes it to the `spring.ai.chat.client.conversation.id` attribute, which Maple groups sessions by:

```java
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.client.advisor.MessageChatMemoryAdvisor;
import org.springframework.ai.chat.memory.ChatMemory;
import org.springframework.stereotype.Service;

@Service
public class ChatService {

	private final ChatClient chatClient;

	public ChatService(ChatClient.Builder builder, ChatMemory chatMemory, SupportTools tools) {
		this.chatClient = builder
			.defaultSystem("You are a concise support assistant.")
			.defaultTools(tools)
			.defaultAdvisors(MessageChatMemoryAdvisor.builder(chatMemory).build())
			.defaultAdvisors(a -> a.param(MapleAiObservationConfig.AGENT_NAME, "support_agent"))
			.build();
	}

	public String reply(String conversationId, String userMessage) {
		return chatClient.prompt()
			.user(userMessage)
			.advisors(a -> a.param(ChatMemory.CONVERSATION_ID, conversationId))
			.call()
			.content();
	}

}
```

Use the conversation id your app already stores. Set it on the request, as above, and never with `defaultAdvisors` on the builder, which puts every user in one session. The parameter works without a chat memory advisor. Sub-agent calls inside tools don't need it.

## Exit command-line apps explicitly

A web app needs nothing extra: Boot flushes pending spans on shutdown. In a `CommandLineRunner` app, exit explicitly, or the OpenAI client's threads hold the JVM and the unsent spans for about 60 seconds:

```java
public static void main(String[] args) {
	System.exit(SpringApplication.exit(SpringApplication.run(Application.class, args)));
}
```

On serverless platforms, inject `SdkTracerProvider` and call `tracerProvider.forceFlush().join(10, TimeUnit.SECONDS)` at the end of each invocation.

## Check that it works

Send two or three messages with the same conversation id, including one that calls a tool. After about a minute, **Agent Sessions** in Maple shows one session for that id with framework **Spring AI**, one turn per `ChatClient` call, the transcript, and tokens on every model call. Cost shows as unpriced, because Spring AI doesn't report it.

## Troubleshooting

- **Only some turns arrive.** Sampling is at Boot's default of 10%. Set `management.tracing.sampling.probability=1.0`.
- **Every message is its own session.** Pass `.advisors(a -> a.param(ChatMemory.CONVERSATION_ID, id))` on every `prompt()` call.
- **The transcript is empty.** `maple.ai.capture-content` isn't `true`, or `MapleAiObservationConfig` isn't in a scanned package.
- **Every span is its own trace.** The OpenTelemetry Java agent is attached next to the starter. Remove it, or follow the Java agent steps in the [skill](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-spring-ai).

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview)
- [Spring AI observability reference](https://docs.spring.io/spring-ai/reference/observability/index.html)
- [Spring Boot tracing reference](https://docs.spring.io/spring-boot/reference/actuator/tracing.html)

---

# Trace agents built on the OpenAI, Anthropic and Gemini SDKs

> AI Frameworks · https://maple.dev/docs/agent-tracing/provider-sdks

Trace your own agent loop on the OpenAI, Anthropic or Google Gen AI SDK so each conversation becomes one Maple Agent Session, in Python or TypeScript.

import LanguageTabs from "../../../components/docs/LanguageTabs.astro"
import LanguageTab from "../../../components/docs/LanguageTab.astro"

Use this guide when your agent is your own loop around `client.chat.completions.create`, `client.messages.create` or `client.models.generate_content`. An instrumentation library records each model call, and you add the turn and tool spans and put the conversation id on the turn span. If you use an agent framework on top of these SDKs, use [that framework's guide](/docs/agent-tracing) instead.

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing-provider-sdks](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-provider-sdks) skill and follows it.

```text
Set up Maple agent tracing for the OpenAI, Anthropic or Gemini SDK in this project.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing-provider-sdks -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## Configure the exporter

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT="SPAN_ONLY"
```

For an EU organization, use `https://ingest.eu.maple.dev`. `SPAN_ONLY` records prompts and replies for the transcript. Leave it unset to keep content out of Maple. `EVENT_ONLY` or `true` leaves the transcript empty.

## Set up tracing

<LanguageTabs label="Language" tabs={[{ id: "python", label: "Python" }, { id: "typescript", label: "TypeScript" }]}>
<LanguageTab id="python">

Install the `genai` packages below, not `opentelemetry-instrumentation-openai` or `opentelemetry-instrumentation-openai-v2`.

```bash
pip install "opentelemetry-sdk>=1.45" "opentelemetry-exporter-otlp-proto-http>=1.45" \
  "opentelemetry-instrumentation-genai-openai>=1.2b0"
# Anthropic: opentelemetry-instrumentation-genai-anthropic>=1.2b0
# Gemini:    opentelemetry-instrumentation-google-genai>=1.2b0
```

```py
# tracing.py
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.genai.openai import OpenAIInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider(resource=Resource.create({"service.name": "support-agent"}))
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

OpenAIInstrumentor().instrument()
# from opentelemetry.instrumentation.genai.anthropic import AnthropicInstrumentor
# from opentelemetry.instrumentation.google_genai import GoogleGenAiSdkInstrumentor
```

Import `tracing` first in your entry point. If your app already has a `TracerProvider` (Sentry, Logfire, Datadog), add the `BatchSpanProcessor` to it instead.

</LanguageTab>
<LanguageTab id="typescript">

No OpenTelemetry instrumentation works for `openai` 7 in TypeScript, so this helper records the turn, each tool call and each model call.

```bash
npm install @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-proto
```

```ts
// instrumentation.ts
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto"
import { NodeSDK, tracing } from "@opentelemetry/sdk-node"

export const spanProcessor = new tracing.BatchSpanProcessor(new OTLPTraceExporter())

export const sdk = new NodeSDK({ serviceName: "support-agent", spanProcessors: [spanProcessor] })
sdk.start()
```

```ts
// agent-tracing.ts
import { type Attributes, type Span, SpanKind, SpanStatusCode, trace } from "@opentelemetry/api"
import type OpenAI from "openai"

const tracer = trace.getTracer("support-agent")

const captureContent = ["SPAN_ONLY", "SPAN_AND_EVENT"].includes(
	(process.env.OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT ?? "").toUpperCase(),
)

export function agentSpan<T>(agentName: string, conversationId: string | undefined, fn: () => Promise<T>) {
	const attributes: Attributes = { "gen_ai.operation.name": "invoke_agent", "gen_ai.agent.name": agentName }
	if (conversationId) attributes["gen_ai.conversation.id"] = conversationId
	return withSpan(`invoke_agent ${agentName}`, SpanKind.INTERNAL, attributes, () => fn())
}

export function runTool(callId: string, name: string, args: string, tool: (args: any) => unknown) {
	const attributes = { "gen_ai.operation.name": "execute_tool", "gen_ai.tool.name": name, "gen_ai.tool.call.id": callId }
	return withSpan(`execute_tool ${name}`, SpanKind.INTERNAL, attributes, async (span) => {
		if (captureContent) span.setAttribute("gen_ai.tool.call.arguments", args)
		let result: string
		try {
			result = JSON.stringify(await tool(JSON.parse(args)))
		} catch (error) {
			markFailed(span, error)
			result = JSON.stringify({ error: String(error) })
		}
		if (captureContent) span.setAttribute("gen_ai.tool.call.result", result)
		return result
	})
}

type ChatParams = Omit<OpenAI.Chat.ChatCompletionCreateParamsNonStreaming, "stream">

/** One model call. Pass `onText` to stream; usage still arrives. */
export function tracedChat(client: OpenAI, params: ChatParams, onText?: (delta: string) => void) {
	const attributes = { "gen_ai.operation.name": "chat", "gen_ai.provider.name": "openai", "gen_ai.request.model": params.model }
	return withSpan(`chat ${params.model}`, SpanKind.CLIENT, attributes, async (span) => {
		if (captureContent) {
			span.setAttribute("gen_ai.input.messages", JSON.stringify(params.messages.map(toGenAiMessage)))
		}
		let completion: OpenAI.Chat.ChatCompletion
		if (onText) {
			const started = performance.now()
			let firstChunkAt: number | undefined
			// Without include_usage, a streamed call reports no tokens at all.
			const stream = client.chat.completions.stream({ ...params, stream_options: { include_usage: true } })
			stream.on("content", (delta) => {
				firstChunkAt ??= performance.now()
				onText(delta)
			})
			completion = await stream.finalChatCompletion()
			if (firstChunkAt !== undefined) {
				span.setAttribute("gen_ai.response.time_to_first_chunk", (firstChunkAt - started) / 1000)
			}
		} else {
			completion = await client.chat.completions.create(params)
		}
		span.setAttributes({
			"gen_ai.response.id": completion.id,
			"gen_ai.response.model": completion.model,
			"gen_ai.response.finish_reasons": completion.choices.map((c) => c.finish_reason),
		})
		if (completion.usage) {
			span.setAttributes({
				"gen_ai.usage.input_tokens": completion.usage.prompt_tokens,
				"gen_ai.usage.output_tokens": completion.usage.completion_tokens,
				"gen_ai.usage.cache_read.input_tokens": completion.usage.prompt_tokens_details?.cached_tokens ?? 0,
				"gen_ai.usage.reasoning.output_tokens": completion.usage.completion_tokens_details?.reasoning_tokens ?? 0,
			})
			// OpenRouter adds the call's price in USD to usage. Other providers don't send one.
			const cost = (completion.usage as { cost?: number }).cost
			if (cost !== undefined) span.setAttribute("gen_ai.usage.cost", cost)
		}
		if (captureContent) {
			const output = completion.choices.map((c) => ({ ...toGenAiMessage(c.message), finish_reason: c.finish_reason }))
			span.setAttribute("gen_ai.output.messages", JSON.stringify(output))
		}
		return completion
	})
}

// Converts OpenAI messages to Maple's transcript format. Text and tool calls only.
function toGenAiMessage(message: OpenAI.Chat.ChatCompletionMessageParam | OpenAI.Chat.ChatCompletionMessage) {
	if (message.role === "tool") {
		return { role: "tool", parts: [{ type: "tool_call_response", id: message.tool_call_id, response: message.content }] }
	}
	const parts: object[] = []
	if (typeof message.content === "string" && message.content) parts.push({ type: "text", content: message.content })
	if (message.role === "assistant") {
		for (const call of message.tool_calls ?? []) {
			if (call.type === "function") {
				parts.push({ type: "tool_call", id: call.id, name: call.function.name, arguments: call.function.arguments })
			}
		}
	}
	return { role: message.role, parts }
}

function withSpan<T>(name: string, kind: SpanKind, attributes: Attributes, fn: (span: Span) => Promise<T>) {
	return tracer.startActiveSpan(name, { kind, attributes }, async (span) => {
		try {
			return await fn(span)
		} catch (error) {
			markFailed(span, error)
			throw error
		} finally {
			span.end()
		}
	})
}

function markFailed(span: Span, error: unknown) {
	const err = error instanceof Error ? error : new Error(String(error))
	span.recordException(err)
	span.setStatus({ code: SpanStatusCode.ERROR, message: err.message })
	span.setAttribute("error.type", err.name)
}
```

</LanguageTab>
</LanguageTabs>

## Wrap each turn and tool call

<LanguageTabs label="Language" tabs={[{ id: "python", label: "Python" }, { id: "typescript", label: "TypeScript" }]}>
<LanguageTab id="python">

Wrap each turn in an `invoke_agent` span that carries `gen_ai.conversation.id`. Use the id your app already stores for the conversation (thread id, ticket id), not a new UUID per request.

```py
# agent_tracing.py
import json
import os
from contextlib import contextmanager

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("support-agent")

CAPTURE_CONTENT = os.environ.get(
    "OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT", ""
).upper() in ("SPAN_ONLY", "SPAN_AND_EVENT")


@contextmanager
def agent_span(agent_name: str, conversation_id: str | None = None):
    attributes = {"gen_ai.operation.name": "invoke_agent", "gen_ai.agent.name": agent_name}
    if conversation_id:
        attributes["gen_ai.conversation.id"] = conversation_id
    with tracer.start_as_current_span(f"invoke_agent {agent_name}", attributes=attributes) as span:
        yield span


def run_tool(call_id: str, name: str, arguments: str, tool) -> str:
    with tracer.start_as_current_span(
        f"execute_tool {name}",
        attributes={
            "gen_ai.operation.name": "execute_tool",
            "gen_ai.tool.name": name,
            "gen_ai.tool.call.id": call_id,
        },
    ) as span:
        if CAPTURE_CONTENT:
            span.set_attribute("gen_ai.tool.call.arguments", arguments)
        try:
            result = json.dumps(tool(**json.loads(arguments)))
        except Exception as exc:
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR, str(exc)))
            span.set_attribute("error.type", type(exc).__qualname__)
            result = json.dumps({"error": str(exc)})
        if CAPTURE_CONTENT:
            span.set_attribute("gen_ai.tool.call.result", result)
        return result
```

In your loop, the tracing is the `with agent_span(...)` line and the `run_tool(...)` call:

```py
# agent.py
import tracing  # noqa: F401  (first import)
from openai import OpenAI

from agent_tracing import agent_span, run_tool

client = OpenAI()
MODEL = "gpt-4o-mini"
# get_weather, fetch_transport_data and TOOL_SCHEMAS are your own tools and their JSON schemas.
TOOLS = {"get_weather": get_weather, "fetch_transport_data": fetch_transport_data}


def chat_turn(conversation_id: str, history: list, user_text: str) -> str:
    with agent_span("support_agent", conversation_id):
        history.append({"role": "user", "content": user_text})
        while True:
            response = client.chat.completions.create(model=MODEL, messages=history, tools=TOOL_SCHEMAS)
            message = response.choices[0].message
            history.append(message.model_dump(include={"role", "content", "tool_calls"}, exclude_none=True))
            if not message.tool_calls:
                return message.content
            for call in message.tool_calls:
                result = run_tool(call.id, call.function.name, call.function.arguments, TOOLS[call.function.name])
                history.append({"role": "tool", "tool_call_id": call.id, "content": result})
```

With Anthropic, run each `tool_use` block with `run_tool(block.id, block.name, json.dumps(block.input), ...)`. With Gemini's automatic function calling, the instrumentation records the tool spans itself, so wrap the turn in `agent_span` and skip `run_tool`.

</LanguageTab>
<LanguageTab id="typescript">

Pass the conversation id your app already stores to `agentSpan`, and route model and tool calls through `tracedChat` and `runTool`:

```ts
// agent.ts
import "./instrumentation.ts"
import OpenAI from "openai"
import { agentSpan, runTool, tracedChat } from "./agent-tracing.ts"

const client = new OpenAI()
const model = "gpt-4o-mini"
// tools: Record<string, (args: any) => unknown> and toolSchemas: OpenAI.Chat.ChatCompletionTool[] are your own.

export function chatTurn(
	conversationId: string,
	history: OpenAI.Chat.ChatCompletionMessageParam[],
	userText: string,
	onText?: (delta: string) => void,
) {
	return agentSpan("support_agent", conversationId, async () => {
		history.push({ role: "user", content: userText })
		while (true) {
			const completion = await tracedChat(client, { model, messages: history, tools: toolSchemas }, onText)
			const message = completion.choices[0].message
			history.push(message)
			if (!message.tool_calls?.length) return message.content ?? ""
			for (const call of message.tool_calls) {
				if (call.type !== "function") continue
				const result = await runTool(call.id, call.function.name, call.function.arguments, tools[call.function.name])
				history.push({ role: "tool", tool_call_id: call.id, content: result })
			}
		}
	})
}
```

`@anthropic-ai/sdk` has tracing built in; see [Claude API & SDK](/docs/agent-tracing/claude-api). For `@google/genai`, copy `tracedChat` and map that SDK's response fields. The [skill's TypeScript reference](https://github.com/MapleTechLabs/maple/blob/main/skills/maple-agent-tracing-provider-sdks/references/typescript.md) has the attribute table.

</LanguageTab>
</LanguageTabs>

## Sub-agents, streaming and cost

Run a sub-agent inside `run_tool` and wrap its loop in `agent_span` with its own name and no conversation id.

When you stream OpenAI Chat Completions in Python, pass `stream_options={"include_usage": True}`, or the call shows 0 tokens. The TypeScript helper sets it for you.

Python sessions show as unpriced. The TypeScript helper records cost only when you call through OpenRouter.

## Flush before a short-lived process exits

In Python, a serverless handler or notebook should call `provider.force_flush()` after each turn. In TypeScript, call `await sdk.shutdown()` before a script exits, or `await spanProcessor.forceFlush()` before a serverless handler returns.

## Check that it works

Run a conversation of two or three messages with a tool call, flush, and open **Agent Sessions** in Maple. You should see one session with your conversation id, one turn per user message, and a transcript with the replies and tool calls. The framework is **Unidentified** for this setup.

## Troubleshooting

- **Every model call is its own session.** The call ran outside `agent_span`. Wrap the whole turn, including the tool loop.
- **One session per turn.** The conversation id changes per request. Pass the id stored with the conversation.
- **The transcript is empty.** Set `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY` in the process that makes the calls.
- **No model spans in Python.** `tracing` wasn't imported first, or you installed `opentelemetry-instrumentation-openai` instead of `opentelemetry-instrumentation-genai-openai`.
- **Every model call appears twice.** A second instrumentation (OpenLLMetry, OpenInference, `logfire.instrument_openai()`, Sentry's OpenAI integration) wraps the same SDK. Uninstall the extra one.

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview)
- [OpenRouter](/docs/agent-tracing/openrouter), if your calls go through OpenRouter
- [OpenTelemetry GenAI instrumentations for Python](https://github.com/open-telemetry/opentelemetry-python-genai)

---

# Trace any AI agent with the OpenTelemetry GenAI conventions

> AI Frameworks · https://maple.dev/docs/agent-tracing/opentelemetry

Write the OpenTelemetry GenAI spans Maple reads by hand, in any language, so a custom agent loop shows up in Agent Sessions.

import LanguageTabs from "../../../components/docs/LanguageTabs.astro"
import LanguageTab from "../../../components/docs/LanguageTab.astro"

Use this guide when no other guide covers your agent, for example a hand-written agent loop. You write the agent spans yourself with any OpenTelemetry SDK: an `invoke_agent` span per user turn, a `chat` span per model call and an `execute_tool` span per tool call.

Tested with the OpenTelemetry JS SDK 2.11 on Node.js 26 and the Python SDK 1.45 on Python 3.14, against the [GenAI conventions](https://github.com/open-telemetry/semantic-conventions-genai) as of September 2026.

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing-opentelemetry](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-opentelemetry) skill and follows it.

```text
Set up Maple agent tracing for my hand-rolled agent in this project, using the OpenTelemetry GenAI conventions.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing-opentelemetry -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## The spans and attributes Maple reads

One user message produces one trace:

```text
invoke_agent support                 gen_ai.conversation.id = chat_42
├── chat openai/gpt-4o-mini          model call: asks for get_weather
├── execute_tool get_weather         tool call
└── chat openai/gpt-4o-mini          model call: final answer
```

Set `gen_ai.operation.name` on every span. Agent Sessions ignores spans without it.

| Span (kind) | Attribute | Value |
| --- | --- | --- |
| `invoke_agent` (`INTERNAL`) | `gen_ai.operation.name` | `invoke_agent` |
| | `gen_ai.agent.name` | `support` |
| | `gen_ai.conversation.id` | your chat or thread id, see [sessions](#group-turns-into-one-session) |
| | `gen_ai.input.messages`, `gen_ai.output.messages` | optional, JSON string |
| `chat` (`CLIENT`) | `gen_ai.operation.name` | `chat` |
| | `gen_ai.provider.name` | the API you called: `openai`, `anthropic`, `gcp.gemini`, `openrouter`... |
| | `gen_ai.request.model`, `gen_ai.response.model` | model ids |
| | `gen_ai.response.id` | the provider's response id |
| | `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` | int |
| | `gen_ai.usage.cache_read.input_tokens`, `gen_ai.usage.cache_write.input_tokens`, `gen_ai.usage.reasoning.output_tokens` | int, optional |
| | `gen_ai.usage.cost` | double in USD, optional |
| | `gen_ai.system_instructions`, `gen_ai.input.messages`, `gen_ai.output.messages` | JSON string |
| | `gen_ai.response.finish_reasons` | string array, e.g. `["stop"]` |
| | `gen_ai.response.time_to_first_chunk` | double, in **seconds**, streamed calls |
| `execute_tool` (`INTERNAL`) | `gen_ai.operation.name` | `execute_tool` |
| | `gen_ai.tool.name` | `get_weather` |
| | `gen_ai.tool.call.id` | the id the model gave the tool call |
| | `gen_ai.tool.call.arguments` | JSON string of an object |
| | `gen_ai.tool.call.result` | the tool's result: a string as is, anything else as a JSON string |

Mark a failed span with status `ERROR` and an `error.type` attribute, even when you return a tool's error to the model as its result.

Put token usage on `chat` spans only. Input tokens include cached tokens and output tokens include reasoning: for Anthropic, add the cache reads and writes to `input_tokens`; for Gemini, add `thoughtsTokenCount` to `candidatesTokenCount`. On OpenAI's streaming API, set `stream_options: { include_usage: true }`, or streamed calls report no tokens. Maple doesn't price tokens, so a session without `gen_ai.usage.cost` shows as unpriced. OpenRouter returns the cost in `usage.cost`.

### The message format

`gen_ai.input.messages` and `gen_ai.output.messages` are JSON arrays of `{role, parts}` messages, serialized to a string:

```json
[
  { "role": "user", "parts": [{ "type": "text", "content": "What's the weather in Berlin?" }] },
  {
    "role": "assistant",
    "parts": [{ "type": "tool_call", "id": "call_1", "name": "get_weather", "arguments": { "city": "Berlin" } }]
  },
  { "role": "tool", "parts": [{ "type": "tool_call_response", "id": "call_1", "response": "{\"temperature_c\":21}" }] }
]
```

Output messages add a `finish_reason` to each message. `gen_ai.system_instructions` is an array of parts without a role: `[{"type":"text","content":"You are a concise assistant."}]`.

Set the message attributes as JSON strings. Maple doesn't read span events, logs or indexed keys like `gen_ai.prompt.0.content`. Leave `OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT` and `OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT` unset, because a truncated message array no longer parses.

To keep prompts and results out of Maple, skip the content attributes (`gen_ai.system_instructions`, `gen_ai.input.messages`, `gen_ai.output.messages`, `gen_ai.tool.call.arguments`, `gen_ai.tool.call.result`). The transcript is then empty and everything else still shows up.

## Export spans to Maple

Point the OTLP exporter at Maple with the standard variables:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
```

For an EU organization, use `https://ingest.eu.maple.dev`. The exporters append `/v1/traces` themselves.

<LanguageTabs label="Language" tabs={[{ id: "typescript", label: "TypeScript" }, { id: "python", label: "Python" }]}>
<LanguageTab id="typescript">

TypeScript (Node.js 20 or newer):

```bash
npm install @opentelemetry/api @opentelemetry/sdk-trace-node @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-proto @opentelemetry/resources openai
```

```ts
// tracing.ts: import this first in every entry point
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto"
import { resourceFromAttributes } from "@opentelemetry/resources"
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base"
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node"

export const provider = new NodeTracerProvider({
	resource: resourceFromAttributes({
		"service.name": "support-agent",
		"deployment.environment.name": "production",
	}),
	// Reads OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS
	spanProcessors: [new BatchSpanProcessor(new OTLPTraceExporter())],
})
// Registers the global provider and the async context manager, so spans nest across awaits
provider.register()
```

</LanguageTab>
<LanguageTab id="python">

Python (3.10 or newer):

```bash
pip install "opentelemetry-sdk>=1.45" "opentelemetry-exporter-otlp-proto-http>=1.45" openai
```

```py
# tracing.py: import this first in every entry point
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider(
    resource=Resource.create(
        {"service.name": "support-agent", "deployment.environment.name": "production"}
    )
)
# Reads OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
```

</LanguageTab>
</LanguageTabs>

If your app already has a `TracerProvider` (Sentry, Datadog, `opentelemetry-instrument`, `NodeSDK`), add the `BatchSpanProcessor` to it instead of creating a second one.

Name the tracer after your app, like `support-agent`. A tracer named after a framework or gateway, such as `openrouter` or `langsmith`, makes Maple treat your spans as that framework's.

## Instrument the agent loop

Complete loops that stream, call tools and record every attribute above are in the skill: [TypeScript](https://github.com/MapleTechLabs/maple/blob/main/skills/maple-agent-tracing-opentelemetry/references/typescript.md) and [Python](https://github.com/MapleTechLabs/maple/blob/main/skills/maple-agent-tracing-opentelemetry/references/python.md). Wrap your own loop's existing calls the same way.

This is the `execute_tool` span from the TypeScript version. The `chat` span follows the same pattern around each model call:

```ts
async function runTool(agent: Agent, call: ToolCall) {
	const name = call.function.name
	return tracer.startActiveSpan(
		`execute_tool ${name}`,
		{
			kind: SpanKind.INTERNAL,
			attributes: {
				"gen_ai.operation.name": "execute_tool",
				"gen_ai.tool.name": name,
				"gen_ai.tool.type": "function",
				"gen_ai.tool.call.id": call.id,
				"gen_ai.tool.call.arguments": call.function.arguments || "{}",
			},
		},
		async (span) => {
			try {
				const result = await agent.tools[name]!.run(JSON.parse(call.function.arguments || "{}"))
				const output = typeof result === "string" ? result : json(result ?? null)
				span.setAttribute("gen_ai.tool.call.result", output)
				return output
			} catch (error) {
				markFailed(span, error)
				return json({ error: error instanceof Error ? error.message : String(error) })
			} finally {
				span.end()
			}
		},
	)
}
```

Start the `chat` and `execute_tool` spans inside the `invoke_agent` span's callback, so they land in the same trace. For a sub-agent, run its loop inside the delegating tool's `execute_tool` span and give it a distinct `gen_ai.agent.name`.

## Group turns into one session

Set `gen_ai.conversation.id` on the `invoke_agent` span of every turn, using the id your app already has for the conversation. A chat backend passes it once per user message:

```ts
// One history per conversation. Store it in your database in a real backend.
const histories = new Map<string, ChatCompletionMessageParam[]>()

export async function handleMessage(chatId: string, text: string, onText?: (delta: string) => void) {
	const history = histories.get(chatId) ?? []
	histories.set(chatId, history)
	history.push({ role: "user", content: text })
	return runAgent(assistant, history, { conversationId: chatId, onText })
}
```

Without the id, or with one generated per request, each trace becomes its own one-turn session named `trace:<trace id>`. Don't give sub-agents their own id; they inherit the session from the trace.

If a framework writes its session id under a key Maple doesn't read, wrap each turn in your own span that carries `maple_ai.session.id`, and run the framework inside it:

```ts
await tracer.startActiveSpan(
	"invoke_agent support",
	{
		attributes: {
			"gen_ai.operation.name": "invoke_agent",
			"gen_ai.agent.name": "support",
			"maple_ai.session.id": chatId,
		},
	},
	async (span) => {
		try {
			return await frameworkAgent.run(message) // the framework's spans nest under this one
		} finally {
			span.end()
		}
	},
)
```

Put `maple_ai.session.id` only on your own wrapper span, never on the framework's spans.

## Flush before a short-lived process exits

`BatchSpanProcessor` exports every few seconds, so a script, CLI or Lambda can exit before the last batch is sent. At the end of a script, call `await provider.shutdown()` (TypeScript) or `provider.shutdown()` (Python). In a serverless handler, call `provider.forceFlush()` (`force_flush()` in Python) before returning.

## Check that it works

Run one conversation with two messages and a tool call, then open **Agent Sessions** in Maple. After a few seconds you should see one session with your conversation id, one turn per message with its transcript, and `chat` and `execute_tool` spans nested under each turn's `invoke_agent` span. Hand-written spans show the framework as **Unidentified**.

## Troubleshooting

- **Nothing shows up in Agent Sessions, but the trace is in Traces.** No span has `gen_ai.operation.name`. Add it to every span.
- **Every message is its own session.** `gen_ai.conversation.id` is missing or changes per request. Pass the conversation's id on each turn's `invoke_agent` span.
- **The transcript is empty, but tokens are there.** The messages are plain text, in span events, or cut by an attribute length limit. Set them as JSON strings and unset `OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT`.
- **Model and tool spans are separate traces.** The spans weren't started inside the `invoke_agent` span, the Node provider wasn't registered with `provider.register()`, or the work ran in a new Python thread (pass the context with `contextvars.copy_context().run(...)`).
- **Every model call appears twice.** A provider auto-instrumentation (OpenAI, Anthropic, OpenLLMetry, OpenInference) is also active. Keep your `chat` spans or the instrumentation, not both. See [provider SDKs](/docs/agent-tracing/provider-sdks).

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview)
- [All agent tracing guides](/docs/agent-tracing)
- [Provider SDKs](/docs/agent-tracing/provider-sdks), for auto-instrumented OpenAI, Anthropic and Gemini clients
- [GenAI spans](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md) and [GenAI agent spans](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md)

---

# Trace OpenRouter calls in Maple with Broadcast

> AI Gateways · https://maple.dev/docs/agent-tracing/openrouter

Send every OpenRouter model call to Maple with OpenRouter Broadcast, grouped into one Agent Session per conversation.

import LanguageTabs from "../../../components/docs/LanguageTabs.astro"
import LanguageTab from "../../../components/docs/LanguageTab.astro"

OpenRouter Broadcast sends a trace of every request on your OpenRouter account to Maple, with tokens, cost, prompt and completion. You set it up once in the OpenRouter dashboard, then add a `session_id` to your requests. Without it, every model call is its own session.

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing-openrouter](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-openrouter) skill and follows it.

```text
Set up Maple agent tracing for OpenRouter in this project.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing-openrouter -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt. The agent prints the values for the OpenRouter dashboard, which you enter yourself as described in the next section.

## Point Broadcast at Maple

1. In OpenRouter, open [Settings → Observability](https://openrouter.ai/settings/observability) and turn on **Enable Broadcast**. In an organization account, only an admin can change this.
2. Click the edit icon next to **OpenTelemetry Collector** and set **Endpoint** to the full traces URL. OpenRouter doesn't append `/v1/traces`. EU organizations use `https://ingest.eu.maple.dev/v1/traces`.

   ```text
   https://ingest.maple.dev/v1/traces
   ```

3. Set **Headers** to your Maple ingest key:

   ```json
   { "Authorization": "Bearer YOUR_INGEST_KEY" }
   ```

4. Click **Test Connection**. OpenRouter only saves the destination if the test passes.

Leave the sampling rate at 1.0, since a lower rate drops whole conversations. Leave the API key filter empty. If your app calls `eu.openrouter.ai`, add the Europe data region.

## Send a session id with every request

Send a `session_id` field in the request body (or an `x-session-id` header). Use the same id on every request of a conversation, such as your chat thread id, and a new one for each conversation.

<LanguageTabs label="Language" tabs={[{ id: "typescript", label: "TypeScript" }, { id: "python", label: "Python" }]}>
<LanguageTab id="typescript">

With the `openai` SDK in TypeScript, the field isn't in the types, so it needs a `@ts-expect-error`:

```ts
import OpenAI from "openai"

const client = new OpenAI({
	baseURL: "https://openrouter.ai/api/v1",
	apiKey: process.env.OPENROUTER_API_KEY,
})

const completion = await client.chat.completions.create({
	model: "openai/gpt-4o-mini",
	messages,
	// @ts-expect-error OpenRouter-only field
	session_id: conversationId,
})
```

With the Vercel AI SDK, pass it under `providerOptions.openrouter`:

```ts
import { createOpenRouter } from "@openrouter/ai-sdk-provider"
import { streamText } from "ai"

const openrouter = createOpenRouter({ apiKey: process.env.OPENROUTER_API_KEY })

const result = streamText({
	model: openrouter("openai/gpt-4o-mini"),
	messages,
	providerOptions: { openrouter: { session_id: conversationId } },
})
```

</LanguageTab>
<LanguageTab id="python">

In Python, use `extra_body`:

```py
import os

from openai import OpenAI

client = OpenAI(base_url="https://openrouter.ai/api/v1", api_key=os.environ["OPENROUTER_API_KEY"])

completion = client.chat.completions.create(
    model="openai/gpt-4o-mini",
    messages=messages,
    extra_body={"session_id": conversation_id},
)
```

</LanguageTab>
</LanguageTabs>

OpenRouter's own SDKs have a typed field: `sessionId` in `@openrouter/sdk` and `session_id=` in the `openrouter` Python package.

## Nest Broadcast under your own traces

Skip this if OpenRouter is your only source of traces. If your app also sends its own traces to Maple, every model call is recorded twice. Pass the active span's ids in the `trace` field so OpenRouter places its spans inside your trace.

<LanguageTabs label="Language" tabs={[{ id: "typescript", label: "TypeScript" }, { id: "python", label: "Python" }]}>
<LanguageTab id="typescript">

In TypeScript, wrap `fetch` and pass it to the client (`new OpenAI({ baseURL, apiKey, fetch: openRouterFetch })` or `createOpenRouter({ apiKey, fetch: openRouterFetch })`):

```ts
import { trace } from "@opentelemetry/api"

export const openRouterFetch: typeof fetch = (input, init) => {
	const span = trace.getActiveSpan()?.spanContext()
	if (span && typeof init?.body === "string") {
		const body = JSON.parse(init.body)
		body.trace = { ...body.trace, trace_id: span.traceId, parent_span_id: span.spanId }
		init = { ...init, body: JSON.stringify(body) }
	}
	return fetch(input, init)
}
```

</LanguageTab>
<LanguageTab id="python">

In Python, build the body next to `session_id`:

```py
from opentelemetry import trace


def openrouter_extra_body(conversation_id: str) -> dict:
    body = {"session_id": conversation_id}
    ctx = trace.get_current_span().get_span_context()
    if ctx.is_valid:
        body["trace"] = {
            "trace_id": format(ctx.trace_id, "032x"),
            "parent_span_id": format(ctx.span_id, "016x"),
        }
    return body


client.chat.completions.create(model=model, messages=messages, extra_body=openrouter_extra_body(conversation_id))
```

</LanguageTab>
</LanguageTabs>

Use the same value for `session_id` as your framework's conversation id.

Broadcast has no tool calls or agent names. For those, trace your app with its [framework guide](/docs/agent-tracing) and nest Broadcast under it as shown here.

## Check that it works

Run a conversation of three turns with the same `session_id`. After about a minute, open **Agent Sessions** and filter by service `openrouter`.

You should see one session named after your `session_id`, with vendor **OpenRouter**, an `LLM Generation` span per model call, and tokens and cost in USD. To keep content out of Maple, turn on **Privacy Mode** on the destination.

## Troubleshooting

- **Test Connection fails.** Use the full `https://ingest.maple.dev/v1/traces` URL and valid JSON headers.
- **Test Connection passes but nothing arrives.** Check the destination's API key filter and data regions against the key and endpoint your app uses, and that you're not using a placeholder key like `MAPLE_TEST`.
- **Every call is its own session, named `trace:<id>`.** The request has no `session_id`. Log the outgoing body, since some wrappers drop unknown fields.
- **Tokens or LLM calls are about double.** Your app and Broadcast both record each call. Nest Broadcast with the `trace` field as shown above.

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview)
- [OpenRouter Broadcast](https://openrouter.ai/docs/guides/features/broadcast)
- [Vercel AI SDK](/docs/agent-tracing/vercel-ai-sdk), if you call OpenRouter through `@openrouter/ai-sdk-provider`

---

# Trace Cloudflare AI Gateway calls in Maple

> AI Gateways · https://maple.dev/docs/agent-tracing/cloudflare-ai-gateway

Send every model call that goes through Cloudflare AI Gateway to Maple with the gateway's OpenTelemetry exporter, grouped into one Agent Session per conversation.

import LanguageTabs from "../../../components/docs/LanguageTabs.astro"
import LanguageTab from "../../../components/docs/LanguageTab.astro"

Cloudflare AI Gateway can export a trace span for every request it proxies, with the model, provider, tokens, cost, prompt and completion. You add Maple as an exporter on the gateway once, then send a `session_id` in each request's metadata. Without it, every model call is its own session.

## Point the gateway at Maple

1. In the Cloudflare dashboard, open **AI → AI Gateway**, pick your gateway and go to **Settings**.
2. Add an OpenTelemetry exporter and set **URL** to the full traces URL. The gateway doesn't append `/v1/traces`. EU organizations use `https://ingest.eu.maple.dev/v1/traces`.

   ```text
   https://ingest.maple.dev/v1/traces
   ```

3. Add a header named `Authorization` with the value `Bearer YOUR_INGEST_KEY`. Your ingest key is in **Settings → Ingestion**.
4. Leave **Content type** on JSON (protobuf works too) and save.

Each gateway has its own exporters, so repeat this for every gateway you want in Maple.

## Send a session id with every request

The gateway turns the keys in a request's metadata into span attributes. Send a `session_id` with the same value on every request of a conversation, such as your chat thread id, and a new one for each conversation. Keys starting with `gen_ai.` are dropped by the gateway, so `gen_ai.conversation.id` doesn't work here.

<LanguageTabs label="Client" tabs={[{ id: "http", label: "HTTP / OpenAI SDK" }, { id: "binding", label: "Workers AI binding" }]}>
<LanguageTab id="http">

Over HTTP, metadata goes in the `cf-aig-metadata` header as a JSON object. With the `openai` SDK and the gateway's OpenAI-compatible endpoint, pass it per request:

```ts
import OpenAI from "openai"

const client = new OpenAI({
	baseURL: `https://gateway.ai.cloudflare.com/v1/${ACCOUNT_ID}/${GATEWAY_ID}/compat`,
	apiKey: process.env.OPENAI_API_KEY,
})

const completion = await client.chat.completions.create(
	{ model: "openai/gpt-4o-mini", messages },
	{ headers: { "cf-aig-metadata": JSON.stringify({ session_id: conversationId }) } },
)
```

</LanguageTab>
<LanguageTab id="binding">

From a Worker, pass the gateway id and the metadata with the call:

```ts
const reply = await env.AI.run(
	"@cf/meta/llama-3.3-70b-instruct-fp8-fast",
	{ messages },
	{ gateway: { id: "my-gateway", metadata: { session_id: conversationId } } },
)
```

</LanguageTab>
</LanguageTabs>

## Nest the gateway span under your own traces

Skip this if the gateway is your only source of traces. If your app also sends traces to Maple, pass the active span's ids so the gateway span lands inside your trace instead of starting a new one. The gateway reads them from two request headers: `cf-aig-otel-trace-id` (32 hex characters) and `cf-aig-otel-parent-span-id` (16 hex characters).

<LanguageTabs label="Client" tabs={[{ id: "http", label: "HTTP / OpenAI SDK" }, { id: "binding", label: "Workers AI binding" }]}>
<LanguageTab id="http">

```ts
import { trace } from "@opentelemetry/api"

function gatewayHeaders(conversationId: string): Record<string, string> {
	const headers: Record<string, string> = {
		"cf-aig-metadata": JSON.stringify({ session_id: conversationId }),
	}
	const span = trace.getActiveSpan()?.spanContext()
	if (span) {
		headers["cf-aig-otel-trace-id"] = span.traceId
		headers["cf-aig-otel-parent-span-id"] = span.spanId
	}
	return headers
}

await client.chat.completions.create(
	{ model: "openai/gpt-4o-mini", messages },
	{ headers: gatewayHeaders(conversationId) },
)
```

</LanguageTab>
<LanguageTab id="binding">

The binding forwards the same headers through `extraHeaders`:

```ts
const span = trace.getActiveSpan()?.spanContext()

await env.AI.run(model, { messages }, {
	gateway: { id: "my-gateway", metadata: { session_id: conversationId } },
	extraHeaders: span
		? { "cf-aig-otel-trace-id": span.traceId, "cf-aig-otel-parent-span-id": span.spanId }
		: {},
})
```

</LanguageTab>
</LanguageTabs>

The gateway span has no tool or agent spans around it. For those, trace your app with its [framework guide](/docs/agent-tracing) and nest the gateway under it as shown here.

## Check that it works

Run a conversation of three turns with the same `session_id`. After about a minute, open **Agent Sessions** and filter by service `ai-gateway`.

You should see one session named after your `session_id`, with vendor **Cloudflare AI Gateway**, a `cf.aig.request` span per model call, the prompt and reply as a transcript, and tokens and cost in USD. Embedding calls show up as `embeddings` operations rather than chat turns.

The example below is a two-turn support chat through the gateway, with one tool call in the first turn.

<figure class="shot">
  <img src="/screenshots/docs/cloudflare-ai-gateway-01-trace.webp" alt="The Traces view of a Cloudflare AI Gateway session: two turns, the first with two cf.aig.request spans and the second with one, each labeled with its Llama 3.3 70B model, a token bar and its duration on a shared time axis." loading="lazy" />
  <figcaption>The Traces view. Each turn is one trace because the app passed <code>cf-aig-otel-trace-id</code>; every model call is a <code>cf.aig.request</code> span with its model and tokens.</figcaption>
</figure>

<figure class="shot">
  <img src="/screenshots/docs/cloudflare-ai-gateway-02-transcript.webp" alt="The Transcript view of the same session: the system prompt, the user asking where order 4821 is, a get_order_status tool call, the assistant's answer with the model named, then a second turn about changing the delivery address and its answer." loading="lazy" />
  <figcaption>The Transcript view. The gateway's raw request and response bodies read as a conversation, including the <code>get_order_status</code> tool call.</figcaption>
</figure>

## What the gateway doesn't export

- **Cached responses.** A request the gateway answers from its cache produces no span, so cache hits don't appear in Maple.
- **Failed requests.** A request the provider rejects, such as an unknown model, produces no span. Track failures in your app's own traces.
- **Content controls.** Prompts and completions are exported in full. If they must stay out of Maple, don't add the exporter to that gateway.

## Troubleshooting

- **Nothing arrives.** Check the exporter URL ends in `/v1/traces` and the header value starts with `Bearer `. The key must belong to the organization you're looking at in Maple.
- **Every call is its own session, named `trace:<id>`.** The request has no `session_id` in its metadata. Check the `cf-aig-metadata` header is valid JSON.
- **Tokens or LLM calls are about double.** Your app and the gateway both record each call. Nest the gateway span with the trace headers as shown above.

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview)
- [AI Gateway OpenTelemetry integration](https://developers.cloudflare.com/ai-gateway/observability/otel-integration/)
- [Cloudflare Agents](/docs/agent-tracing/cloudflare-agents), if your gateway calls come from the Agents SDK

---

# Trace LiteLLM agents and the LiteLLM Proxy with OpenTelemetry

> AI Gateways · https://maple.dev/docs/agent-tracing/litellm

Send LiteLLM's model-call spans to Maple from the Python SDK or the LiteLLM Proxy, and add the agent and tool spans that group each conversation into one Agent Session.

LiteLLM traces each model call. Your code adds the agent and tool spans and passes a session id on every call so Maple groups the turns into one session.

## Quick setup with a coding agent

Copy this prompt into a coding agent that can run shell commands, such as Claude Code, Codex or Cursor. It installs the [maple-agent-tracing-litellm](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-agent-tracing-litellm) skill and follows it.

```text
Set up Maple agent tracing for LiteLLM in this project.

Install the skill with `npx skills add MapleTechLabs/maple/skills --skill maple-agent-tracing-litellm -y`, then follow it.

My Maple ingest key is maple_pk_... and my organization is in the US region.
```

Your ingest key is in **Settings → Ingestion**. If your organization is in the EU region, change `US` to `EU` in the prompt.

## Trace in your app or at the proxy

If your code calls `litellm.acompletion()`, follow the next sections. If your app calls a LiteLLM Proxy you run, see [Trace at the LiteLLM Proxy](#trace-at-the-litellm-proxy). Trace model calls in one place only, or every call shows up twice.

## Install LiteLLM and the exporter

```bash
pip install "litellm==1.103.0" "opentelemetry-sdk==1.43.0" "opentelemetry-exporter-otlp-proto-http==1.43.0"
```

Keep OpenTelemetry at 1.43. Version 1.44 and later break LiteLLM 1.103's logger.

Point the exporter at Maple:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
```

For an EU organization, use `https://ingest.eu.maple.dev`.

## Register LiteLLM's v2 logger

Use the v2 logger (`OpenTelemetryV2`). The default v1 logger makes every call its own session. Pass it your `TracerProvider`:

```py
# tracing.py
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

import litellm
from litellm.integrations.otel.logger import OpenTelemetryV2
from litellm.integrations.otel.model.config import OpenTelemetryV2Config

provider = TracerProvider(
    resource=Resource.create(
        {"service.name": "support-agent", "deployment.environment.name": "production"}
    )
)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

litellm.callbacks = [
    OpenTelemetryV2(
        config=OpenTelemetryV2Config(capture_message_content="span_only"),
        tracer_provider=provider,
    )
]

tracer = trace.get_tracer("support-agent")
```

Import `tracing` at the top of your entry point. If your app already has a `TracerProvider`, pass that one. Don't also add `"otel"` to `litellm.callbacks`, which registers a second logger.

`"span_only"` records prompts and replies for the transcript. Use `"no_content"` to keep them out of Maple.

## Wrap the agent loop in agent and tool spans

Wrap each agent run in an `invoke_agent` span and each tool call in an `execute_tool` span. The v2 logger only traces `acompletion()`, not `completion()`:

```py
# agent.py
import asyncio
import inspect
import json
from contextlib import contextmanager
from dataclasses import dataclass, field

import litellm
from opentelemetry.trace import StatusCode

from tracing import tracer


@dataclass
class Agent:
    name: str
    model: str
    instructions: str
    tools: dict = field(default_factory=dict)  # tool name -> function (sync or async)
    schemas: list = field(default_factory=list)  # OpenAI-style tool definitions


@contextmanager
def agent_span(name: str):
    with tracer.start_as_current_span(f"invoke_agent {name}") as span:
        span.set_attribute("gen_ai.operation.name", "invoke_agent")
        span.set_attribute("gen_ai.agent.name", name)
        yield span


async def run_tool(agent: Agent, call) -> str:
    name = call.function.name
    with tracer.start_as_current_span(f"execute_tool {name}") as span:
        span.set_attribute("gen_ai.operation.name", "execute_tool")
        span.set_attribute("gen_ai.tool.name", name)
        span.set_attribute("gen_ai.tool.call.id", call.id)
        span.set_attribute("gen_ai.tool.call.arguments", call.function.arguments)
        try:
            result = agent.tools[name](**json.loads(call.function.arguments or "{}"))
            if inspect.isawaitable(result):
                result = await result
        except Exception as exc:
            span.set_status(StatusCode.ERROR, str(exc))
            span.set_attribute("error.type", type(exc).__name__)
            result = {"error": str(exc)}
        output = json.dumps(result)
        span.set_attribute("gen_ai.tool.call.result", output)
        return output


async def run_agent(agent: Agent, conversation_id: str, messages: list) -> str:
    with agent_span(agent.name):
        while True:
            response = await litellm.acompletion(
                model=agent.model,
                messages=[{"role": "system", "content": agent.instructions}, *messages],
                tools=agent.schemas or None,
                litellm_session_id=conversation_id,
            )
            message = response.choices[0].message
            messages.append(message.model_dump(exclude_none=True))
            if not message.tool_calls:
                return message.content or ""
            results = await asyncio.gather(*(run_tool(agent, call) for call in message.tool_calls))
            for call, output in zip(message.tool_calls, results):
                messages.append({"role": "tool", "tool_call_id": call.id, "content": output})
```

For sub-agents, call `run_agent` for the worker from inside a tool function, with its own `name` and the same conversation id.

## Group every turn into one session

`litellm_session_id=` sets the session. Pass the chat or thread id your app already has, stable for the whole conversation:

```py
from agent import Agent, run_agent

assistant = Agent("assistant", "openrouter/openai/gpt-4o-mini", "You are a concise assistant.")
history: dict[str, list] = {}


async def handle_message(chat_id: str, text: str) -> str:
    messages = history.setdefault(chat_id, [])
    messages.append({"role": "user", "content": text})
    return await run_agent(assistant, chat_id, messages)
```

Don't set `gen_ai.conversation.id` on your own `invoke_agent` span, or the session is labeled **Unidentified** instead of **LiteLLM**.

For streaming, pass `stream_options={"include_usage": True}` and consume the stream inside the agent span, or the streamed call has no token counts.

## Trace at the LiteLLM Proxy

To trace at a proxy you run, enable the logger in its `config.yaml`:

```yaml
model_list:
  - model_name: gpt-4o-mini
    litellm_params:
      model: openrouter/openai/gpt-4o-mini
      api_key: os.environ/OPENROUTER_API_KEY

litellm_settings:
  callbacks: ["otel"]
```

Set these in the proxy's environment:

```bash
LITELLM_OTEL_V2=true
OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_SERVICE_NAME=litellm-proxy
OTEL_ENVIRONMENT_NAME=production
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only
```

The `ghcr.io/berriai/litellm` image works as is. A pip-installed proxy needs these packages:

```bash
pip install "litellm[proxy]==1.103.0" "opentelemetry-sdk==1.43.0" \
  "opentelemetry-exporter-otlp-proto-http==1.43.0" "opentelemetry-instrumentation-fastapi==0.64b0"
```

Then start it with `litellm --config config.yaml`.

In your app, keep `agent_span` and `run_tool`, drop the LiteLLM logger from `tracing.py`, and send `traceparent` and `x-litellm-session-id` with every request:

```py
import os

from openai import AsyncOpenAI
from opentelemetry import propagate

client = AsyncOpenAI(base_url="http://localhost:4000", api_key=os.environ["LITELLM_API_KEY"])


async def call_model(conversation_id: str, messages: list, tools: list | None):
    headers = {"x-litellm-session-id": conversation_id}
    propagate.inject(headers)
    return await client.chat.completions.create(
        model="gpt-4o-mini", messages=messages, tools=tools, extra_headers=headers
    )
```

Don't also instrument the OpenAI client in the app, or calls and tokens double. On this path the session page's LLM call count shows 2x the real number. Tokens, cost and the transcript are correct.

## Flush before a short-lived process exits

A script, Lambda or notebook cell that ends right after its last call loses that call's span. Drain LiteLLM's queue and flush:

```py
import asyncio

from litellm.litellm_core_utils.logging_worker import GLOBAL_LOGGING_WORKER

from tracing import provider


async def flush_tracing() -> None:
    await asyncio.sleep(0)  # LiteLLM queues its log event on the next loop tick
    await GLOBAL_LOGGING_WORKER.flush()
    provider.force_flush()


async def main() -> None:
    try:
        print(await handle_message("chat-42", "What's the weather in Berlin?"))
    finally:
        await flush_tracing()


asyncio.run(main())
provider.shutdown()
```

A long-running server needs this only in its shutdown hook.

## Check that it works

Run a conversation with two messages and a tool call, then open **Agent Sessions** in Maple. You should see one session with the id you passed and framework **LiteLLM**, one turn per message, and a transcript with the prompts, replies and tool calls.

## Troubleshooting

- **`ModuleNotFoundError: No module named 'opentelemetry._events'`, or no LiteLLM spans and `Error initializing custom logger` in the log.** OpenTelemetry 1.44+ on LiteLLM 1.103. Pin OpenTelemetry to 1.43.0.
- **Your spans arrive but no `chat` spans.** The code calls sync `litellm.completion()`. Switch to `acompletion()`.
- **Every call is its own session, or spans are named `litellm_request`.** You're on the v1 logger, or no session id was sent. Use the `OpenTelemetryV2` setup and pass `litellm_session_id=` (SDK) or `x-litellm-session-id` (proxy) on every call.
- **Every model call appears twice.** Two loggers, or the proxy and an in-app OpenAI instrumentor both trace the call. Keep one.
- **The last call of a script is missing.** Await `flush_tracing()` before the event loop ends.

## Related

- [Agent Sessions overview](/docs/agent-sessions/overview)
- [LiteLLM: OpenTelemetry v2](https://docs.litellm.ai/docs/observability/opentelemetry_v2)

---

# Hosts

> Infrastructure · https://maple.dev/docs/infrastructure/hosts

Send host CPU, memory, disk, network and load metrics to Maple with the OpenTelemetry Collector hostmetrics receiver, and read them on the Hosts page.

The **Hosts** page (**Infrastructure → Hosts**) lists every machine that sends OpenTelemetry host metrics to Maple, with its CPU, memory, disk and load. Each host has a detail page with charts over time and its platform metadata.

Hosts appear from the OpenTelemetry `hostmetrics` receiver. There are two ways to run it:

- **Kubernetes.** The `maple-k8s-infra` Helm chart runs it on every node. Follow [Kubernetes infrastructure](/docs/infrastructure/kubernetes) and your nodes appear here with no extra setup.
- **Any other Linux, macOS or Windows host.** Run the OpenTelemetry Collector with the `hostmetrics` receiver, as below.

The [Docker agent](/docs/infrastructure/docker) collects per-container metrics, not host metrics. Its containers appear under **Infrastructure → Containers**.

## Prerequisites

- The [OpenTelemetry Collector Contrib](https://github.com/open-telemetry/opentelemetry-collector-releases/releases) distribution (`otelcol-contrib`) on the host. It includes the `hostmetrics` receiver and the `resourcedetection` processor.
- A private ingest key (`maple_sk_…`) from **Settings → Ingestion**.

## Configure the Collector

Save this as `config.yaml`:

```yaml
receivers:
    hostmetrics:
        collection_interval: 30s
        scrapers:
            cpu:
                metrics:
                    system.cpu.utilization:
                        enabled: true
            load: {}
            memory:
                metrics:
                    system.memory.utilization:
                        enabled: true
            filesystem:
                metrics:
                    system.filesystem.utilization:
                        enabled: true
            network: {}

processors:
    resourcedetection:
        detectors: [env, system]
    batch: {}

exporters:
    otlphttp/maple:
        endpoint: https://ingest.maple.dev
        compression: gzip
        headers:
            x-maple-ingest-key: ${env:MAPLE_INGEST_KEY}

service:
    pipelines:
        metrics:
            receivers: [hostmetrics]
            processors: [resourcedetection, batch]
            exporters: [otlphttp/maple]
```

Then start the Collector with your key in the environment:

```bash
MAPLE_INGEST_KEY=YOUR_INGEST_KEY otelcol-contrib --config config.yaml
```

Notes on the config:

- **The three `*.utilization` metrics must be enabled.** The `hostmetrics` receiver leaves them off by default, and the Hosts page reads them for CPU, memory and disk.
- **`resourcedetection` sets `host.name`.** Maple keys each host on `host.name`. The `system` detector also sets `os.type` and `host.arch`, which the detail page shows. Add a cloud detector (`ec2`, `gcp` or `azure`) to fill in `cloud.provider` and `cloud.region`.
- **EU organizations** use `https://ingest.eu.maple.dev` as the endpoint.

If you already run a Collector on the host for application telemetry, add the `hostmetrics` receiver and a `metrics` pipeline to it instead of running a second one.

## What the Hosts page shows

| Metric                          | Where it appears                                                |
| ------------------------------- | --------------------------------------------------------------- |
| `system.cpu.utilization`        | CPU in **Usage**, **Avg CPU** card, fleet grid, CPU chart       |
| `system.memory.utilization`     | MEM in **Usage**, **Avg memory** card, fleet grid, Memory chart |
| `system.filesystem.utilization` | DSK in **Usage**, fleet grid, Filesystem chart (per mountpoint) |
| `system.cpu.load_average.15m`   | **Load 15m** column and chart                                   |
| `system.network.io`             | Network chart (per device, in and out)                          |

The list shows each host's **Status**, **Usage** (CPU, memory and disk), **Load 15m** and **Last seen**. With four or more hosts, a fleet grid shows them side by side. Filter by name, or by status:

| Status | Meaning                                  |
| ------ | ---------------------------------------- |
| Active | Last report less than 1 minute ago.      |
| Idle   | Last report between 1 and 5 minutes ago. |
| Ended  | No report for 5 minutes or more.         |

Click a host to open its detail page: **CPU**, **Memory**, **Filesystem**, **Network** and **Load 15m** charts, and the host's `host.name`, `os.type`, `host.arch`, `cloud.provider`, `cloud.region`, and first and last seen times.

## Verify

Start the Collector and open **Infrastructure → Hosts**. The host appears within about a minute, with status **Active**. CPU and memory fill in on the first report. Charts on the detail page need a few reports to draw a line.

## Troubleshooting

- **The host does not appear.** Check the Collector's output for exporter errors. A `401` means the ingest key is wrong or was rotated. Confirm `resourcedetection` is in the pipeline: without `host.name`, Maple cannot list the host.
- **The host appears but CPU, memory or disk read 0.** The matching `*.utilization` metric is not enabled in the receiver config.
- **The host shows Ended.** The Collector stopped or cannot reach Maple. Status turns **Active** again on the next report.
- **Two rows for one machine.** `host.name` changed, for example after a hostname change or a container restart. Each distinct `host.name` is its own host.

## Next steps

- [Kubernetes infrastructure](/docs/infrastructure/kubernetes): nodes, pods and workloads.
- [Docker infrastructure](/docs/infrastructure/docker): per-container metrics and logs.
- [Metrics](/docs/explore/metrics): query any metric, including the `system.*` metrics above.

---

# Kubernetes infrastructure

> Infrastructure · https://maple.dev/docs/infrastructure/kubernetes

Deploy Maple's Kubernetes infrastructure collector with Helm to stream host, kubelet and cluster metrics, and wire the service map's Infrastructure tab to your workloads.

The `maple-k8s-infra` Helm chart collects host, kubelet and cluster metrics from your Kubernetes cluster over OpenTelemetry and sends them to Maple. Once it runs, the **Infrastructure** pages list your nodes, pods and workloads, and each service on the service map gains a pod-count badge and an Infrastructure tab.

Running plain Docker hosts instead of, or alongside, Kubernetes? See [Docker infrastructure](/docs/infrastructure/docker). A single-container agent covers per-container metrics and logs. For hosts outside Kubernetes, see [Hosts](/docs/infrastructure/hosts).

The chart uses a split-collector architecture:

- a **DaemonSet** for node-local OTLP, host metrics, kubelet/pod metrics, and optional pod logs
- a single-replica **Deployment** for cluster-wide metrics and optional Kubernetes events

All signals are exported over OTLP HTTP to Maple's ingest gateway, which authenticates the ingest key.

## Prerequisites

- `kubectl` and `helm` (v3.8+, for OCI registry support) pointed at the target cluster
- Cluster-admin (the chart installs RBAC and, by default, the OpenTelemetry Operator)
- A **private ingest key**. Copy it from **Settings → Ingestion**.

## Install

The install script creates the namespace and the ingest-key Secret and runs Helm for you. It prints your active `kubectl` context and asks for confirmation first (set `MAPLE_INSTALL_YES=1` to skip):

```bash
curl -fsSL https://raw.githubusercontent.com/MapleTechLabs/maple/main/deploy/k8s-infra/install.sh | \
  MAPLE_INGEST_KEY=YOUR_INGEST_KEY \
  MAPLE_CLUSTER_NAME=production \
  bash
```

Prefer Helm directly? Install the published OCI chart:

```bash
helm upgrade --install maple-k8s-infra \
  oci://ghcr.io/mapletechlabs/charts/maple-k8s-infra \
  --namespace maple --create-namespace \
  --set-string maple.ingestKey.value=YOUR_INGEST_KEY \
  --set-string global.clusterName=production
```

For production, keep the key out of your Helm values and reference an existing Secret instead:

```bash
kubectl create namespace maple
kubectl -n maple create secret generic maple-ingest-key \
  --from-literal=ingest-key=YOUR_INGEST_KEY

helm upgrade --install maple-k8s-infra \
  oci://ghcr.io/mapletechlabs/charts/maple-k8s-infra \
  --namespace maple \
  --set maple.ingestKey.existingSecret.name=maple-ingest-key \
  --set maple.ingestKey.existingSecret.key=ingest-key \
  --set-string global.clusterName=production
```

Confirm the rollout:

```bash
kubectl -n maple rollout status daemonset/maple-k8s-infra-agent
kubectl -n maple rollout status deployment/maple-k8s-infra-cluster
```

### EU organizations

The chart and the install script default to the US ingest endpoint (`https://ingest.maple.dev`). With the install script, set `MAPLE_INGEST_ENDPOINT`:

```bash
curl -fsSL https://raw.githubusercontent.com/MapleTechLabs/maple/main/deploy/k8s-infra/install.sh | \
  MAPLE_INGEST_KEY=YOUR_INGEST_KEY \
  MAPLE_CLUSTER_NAME=production \
  MAPLE_INGEST_ENDPOINT=https://ingest.eu.maple.dev \
  bash
```

With Helm, add:

```bash
  --set-string maple.ingest.endpoint=https://ingest.eu.maple.dev
```

The command on **Infrastructure → Hosts → Add host → Kubernetes** includes this flag when your organization is in the EU region.

## What gets collected

Enabled by default:

- **Host metrics:** CPU, memory, disk, network and load per node. These fill the [Hosts](/docs/infrastructure/hosts) page.
- **Kubelet and pod metrics:** per-pod CPU usage, and CPU and memory utilization against limits and requests.
- **Cluster metrics:** node conditions, pod phases, deployment availability and namespace phases.
- **OTLP receivers:** gRPC `:4317` and HTTP `:4318` on every node, so in-cluster apps can send traces, logs and metrics to the local agent.

Off by default (enable per your ingestion budget):

```yaml
presets:
    podLogs:
        enabled: true
    k8sEvents:
        enabled: true
    fargateMetrics: # EKS Fargate per-pod CPU/memory
        enabled: true
```

In-cluster apps can export to the agent's Service:

```
OTLP gRPC: maple-k8s-infra-agent.maple.svc:4317
OTLP HTTP: http://maple-k8s-infra-agent.maple.svc:4318
```

## Verify

Within about a minute of a healthy rollout:

1. Open **Infrastructure → Hosts**. Your nodes appear with CPU, memory and disk. The **Kubernetes** pages list pods, nodes and workloads.
2. If a view stays empty, check the agent logs: `kubectl -n maple logs -l app.kubernetes.io/component=agent --tail=200`.

## Wire the service map's Infrastructure tab

To get a pod-count badge and an Infrastructure tab on each **service** node (correlating your app traces to the workload running them), opt a namespace into env-var injection:

```bash
kubectl annotate namespace shop \
  instrumentation.opentelemetry.io/inject-sdk=maple/maple-default

kubectl rollout restart deployment -n shop
```

The chart bundles the OpenTelemetry Operator and an `Instrumentation/maple-default` resource. It injects downward-API environment variables: `OTEL_EXPORTER_OTLP_ENDPOINT`, and the pod IP, UID, name, namespace and node in `OTEL_RESOURCE_ATTRIBUTES`. Your app's OpenTelemetry SDK reads those on startup. The agent's `k8sattributes` processor then adds `k8s.deployment.name`, `k8s.statefulset.name` or `k8s.daemonset.name` to each span, which joins `service.name` to its workload.

**This only injects environment variables. It does not add an SDK to your app.** Your service must already emit OTLP spans, and its SDK must read the standard `OTEL_*` variables. Most auto-instrumentation agents and Maple's SDKs do. To opt many namespaces in at install time, list them under `autoInstrumentation.instrumentation.autoInstrumentNamespaces` instead of annotating each one.

## Per-cloud notes

| Distribution                    | Notes                                                                                                                                                                               |
| ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| EKS standard, GKE Standard, AKS | Works out of the box.                                                                                                                                                               |
| EKS Fargate                     | The DaemonSet can't run on Fargate nodes. Keep one EC2 node for the agent, and set `presets.fargateMetrics.enabled=true` so per-pod CPU/memory is scraped via the API-server proxy. |
| GKE Autopilot                   | Only annotate your own namespaces. Mutating webhooks are rejected on Google-managed namespaces.                                                                                     |
| k3s / kind / k0s                | Works with auto-generated webhook certs (no cert-manager needed).                                                                                                                   |
| Service mesh (Linkerd, Istio)   | Sidecars rewrite source IPs, but the `k8s.pod.uid` / `(name, namespace)` association keys ride inside the OTLP payload and rescue the workload join. No mesh-side config needed.    |

## Troubleshooting

- **No hosts or pods after a few minutes.** Confirm both workloads are `Ready` and look for `k8sattributes` or exporter errors in `kubectl -n maple logs -l app.kubernetes.io/component=agent`.
- **Empty Infrastructure tab on a service.** The namespace annotation wasn't applied or the pods weren't restarted, or your SDK ignores `OTEL_*` env vars. Verify with `kubectl exec -n <ns> <pod> -- env | grep OTEL`.
- **Operator webhook crash-looping.** This is usually a certificate issue. Switch to cert-manager with `opentelemetryOperator.admissionWebhooks.certManager.enabled=true` if your cluster requires it.

Already running the OpenTelemetry Operator yourself? Set `autoInstrumentation.operator.enabled=false`. The `Instrumentation` resource still renders, so you can apply it to your own setup.

## Uninstall

```bash
helm uninstall maple-k8s-infra --namespace maple
```

To stop env-var injection for a namespace without uninstalling, remove the annotation and restart:

```bash
kubectl annotate namespace shop instrumentation.opentelemetry.io/inject-sdk-
kubectl rollout restart deployment -n shop
```

---

# Docker infrastructure

> Infrastructure · https://maple.dev/docs/infrastructure/docker

Run the Maple Docker agent as a single container to stream per-container CPU, memory, network, block I/O and logs, and correlate them with your app's traces.

Maple's Docker agent is a single OpenTelemetry Collector container with read-only access to the Docker socket. Once it runs, **Infrastructure → Containers** lists every container on the host, spans and logs that carry container identity gain an **Infrastructure** tab, and the **Docker Containers** dashboard template fills in.

Running Kubernetes? Use [Kubernetes infrastructure](/docs/infrastructure/kubernetes) instead. The Helm chart covers pods, nodes and workloads across the cluster. For host-level CPU, memory and disk, see [Hosts](/docs/infrastructure/hosts).

The agent collects:

- **Per-container metrics** through the `docker_stats` receiver: CPU, memory, network, block I/O, restarts, uptime and PID counts, every 30 seconds.
- **Container logs** from the mounted `json-file` log directory. This is optional: drop the mount to skip logs.
- **App OTLP** on ports 4317 (gRPC) and 4318 (HTTP), so it also serves as the host's local collector.

All signals are exported over OTLP HTTP to Maple's ingest gateway.

## Prerequisites

- Docker Engine with the default `json-file` logging driver (for log collection).
- A **private ingest key**. Copy it from **Settings → Ingestion**.
- Ports 4317 and 4318 free on the host. If you already run a collector there, drop the `-p` flags below and keep pointing your apps at the existing one. The agent still collects container metrics and logs without them.

## Install

Run the agent on each Docker host:

```bash
docker run -d --name maple-agent \
  --restart unless-stopped --user 0:0 \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v /var/lib/docker/containers:/var/lib/docker/containers:ro \
  -v maple-agent-state:/var/lib/otelcol \
  -p 4317:4317 -p 4318:4318 \
  -e MAPLE_INGEST_KEY=YOUR_INGEST_KEY \
  ghcr.io/mapletechlabs/maple/otel-collector-maple:0.2.0 \
  --config /etc/otel/docker-config.yaml
```

The same command, with your key filled in, is on **Infrastructure → Hosts → Add host → Docker**.

Notes on the flags:

- `--user 0:0` is required. The Docker socket and `/var/lib/docker/containers` are not readable by the image's non-root user. The socket mount is read-only.
- The `/var/lib/docker/containers` mount only feeds **log** collection. Drop it if you do not want container logs.
- The `maple-agent-state` volume keeps log-read checkpoints across agent restarts.
- `-e MAPLE_ENVIRONMENT=staging` sets the deployment environment. The default is `production`.
- EU organizations add `-e MAPLE_ENDPOINT=https://ingest.eu.maple.dev`. The default is the US endpoint, `https://ingest.maple.dev`.

### Docker Compose

```yaml
services:
    maple-agent:
        image: ghcr.io/mapletechlabs/maple/otel-collector-maple:0.2.0
        command: ["--config", "/etc/otel/docker-config.yaml"]
        restart: unless-stopped
        user: "0:0"
        environment:
            MAPLE_INGEST_KEY: YOUR_INGEST_KEY
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock:ro
            - /var/lib/docker/containers:/var/lib/docker/containers:ro
            - maple-agent-state:/var/lib/otelcol
        ports:
            - "4317:4317"
            - "4318:4318"

volumes:
    maple-agent-state:
```

The agent copies the `com.docker.compose.project` and `com.docker.compose.service` labels onto every container's metrics, so the Containers page can filter by Compose project and service.

## Verify

Open **Infrastructure → Containers**. Containers on the host appear within about a minute, with CPU and memory filled in. If you kept the log mount, container logs appear under **Logs**.

## What gets collected

| Metric                                           | What it powers                              |
| ------------------------------------------------ | ------------------------------------------- |
| `container.cpu.utilization`                      | CPU column, saturation ranking, CPU chart   |
| `container.memory.percent`                       | Memory-vs-limit column and chart            |
| `container.memory.usage.total` / `.limit`        | Memory bytes chart and limit metadata       |
| `container.network.io.usage.rx_bytes`/`tx_bytes` | Network I/O chart                           |
| `container.blockio.io_service_bytes_recursive`   | Block I/O chart (by operation)              |
| `container.restarts`, `container.uptime`         | Restart count and uptime on the detail page |
| `container.cpu.limit`, `container.pids.count`    | Returned by the container API; not charted  |

Identity rides on resource attributes: `container.name`, `container.id`, `container.image.name`, `container.runtime`, and `host.name`. The agent reads `host.name` from the Docker daemon, so it reports the host, not the agent container. Container names are only unique per host, so Maple keys every container on `(container.name, host.name)`.

CPU utilization is Docker's percentage. It can exceed 100% on multi-core containers. Most plain-Docker containers run without CPU limits, so read the saturation ranking as a list of the heaviest containers first. It does not measure remaining capacity.

## Correlate app telemetry

Spans and logs open an **Infrastructure** tab when they carry container identity. Kubernetes injects that identity through the OpenTelemetry Operator. Plain Docker has no injection mechanism, so your app's SDK has to set it:

- **`@maple-dev/effect-sdk`** detects Docker identity automatically. It reads `/proc/self/mountinfo`, then `/proc/self/cgroup`, and falls back to the short container id in the hostname.
- **Any other OpenTelemetry SDK:** set it in your Compose file. Docker's default hostname is the short container id:

    ```yaml
    environment:
        OTEL_RESOURCE_ATTRIBUTES: "container.id=${HOSTNAME},container.name=myservice"
    ```

If your containers set a custom `hostname:`, the hostname fallback cannot work. Set `OTEL_RESOURCE_ATTRIBUTES` explicitly.

## Security notes

- The agent mounts the Docker socket read-only, but socket access is still effectively root on the host. Run the agent only on hosts you control, and pin the image tag instead of using `latest`.
- The install command embeds your **private ingest key**. Rotate it from **Settings → Ingestion** if it leaks.

## Troubleshooting

- **Nothing after two minutes.** Check the agent's own logs with `docker logs maple-agent`. A `401` from the exporter means the ingest key is wrong or was rotated.
- **`port is already allocated`.** Another collector already owns 4317 or 4318 on this host. Drop the `-p` flags (see Prerequisites) or remap them.
- **`permission denied` on the socket.** The agent is not running as root (`--user 0:0`), or the socket lives elsewhere. Rootless Docker uses `$XDG_RUNTIME_DIR/docker.sock`, and the command above does not support rootless setups.
- **Metrics but no logs.** The log-directory mount is missing, or your daemon uses a logging driver other than `json-file`.
- **Containers counted under Stale agent.** The agent has not reported them for over five minutes. Check whether the agent restarted or the host is overloaded.

## Uninstall

Stop and remove the agent container:

```bash
docker rm -f maple-agent
```

To also delete the saved log-read checkpoints:

```bash
docker volume rm maple-agent-state
```

With Docker Compose, remove the `maple-agent` service and its volume from your Compose file, then run `docker compose up -d --remove-orphans`.

Data already sent to Maple stays until it ages out under your plan's [retention](/docs/reference/retention).

---

# Alchemy

> Infrastructure as Code · https://maple.dev/docs/iac/alchemy

Send a Cloudflare Worker's traces, logs and metrics to Maple from your alchemy.run.ts, then declare alerts and dashboards next to the infrastructure they watch.

`@maple-dev/alchemy` connects [Alchemy](https://alchemy.run) stacks to Maple. Add one layer to a Cloudflare Worker and every request is traced, with its logs and metrics, without touching the handler. When you want more, the same package declares Maple alert rules, dashboards and keys in your `alchemy.run.ts`, so they deploy with the service they observe.

|                   |                                                                                                   |
| ----------------- | ------------------------------------------------------------------------------------------------- |
| Package           | [`@maple-dev/alchemy`](https://www.npmjs.com/package/@maple-dev/alchemy)                          |
| Peer dependencies | `alchemy >=2.0.0-beta.80 <3`, `effect ^4.0.0`                                                     |
| Source            | [packages/alchemy-maple](https://github.com/MapleTechLabs/maple/tree/main/packages/alchemy-maple) |

## Quick start

Get a Worker's telemetry into Maple in four steps.

### 1. Install

```bash
npm install @maple-dev/alchemy
```

`alchemy` and `effect` are peer dependencies, which your stack already has.

### 2. Register the provider

Merge `Maple.providers()` into your stack's providers. It lets the stack read your organization's ingest key, so you never copy it by hand.

```typescript
// alchemy.run.ts
import * as Alchemy from "alchemy"
import * as Cloudflare from "alchemy/Cloudflare"
import * as Maple from "@maple-dev/alchemy"
import { Effect, Layer } from "effect"
import Api from "./src/worker.ts"

export default Alchemy.Stack(
	"my-app",
	{ providers: Layer.mergeAll(Cloudflare.providers(), Maple.providers()) },
	Effect.gen(function* () {
		yield* Api
	}),
)
```

### 3. Add telemetry to the Worker

Declare the ingest keys once, then provide `Telemetry` on the Worker's init Effect:

```typescript
// src/maple.ts
import * as Maple from "@maple-dev/alchemy"

export const Ingest = Maple.IngestKeys("ingest")
```

```typescript
// src/worker.ts
import * as Cloudflare from "alchemy/Cloudflare"
import { Telemetry } from "@maple-dev/alchemy/telemetry"
import { Effect } from "effect"
import { HttpServerResponse } from "effect/http"
import { Ingest } from "./maple.ts"

export default Cloudflare.Worker(
	"api",
	{ main: import.meta.url },
	Effect.gen(function* () {
		const fetch = Effect.succeed(HttpServerResponse.text("Hello from Maple"))
		return { fetch }
	}).pipe(Effect.provide(Telemetry({ serviceName: "api", ingestKey: Ingest }))),
)
```

That is the whole integration. Each request becomes a server span, Effect logs and spans inside the handler are attached to it, and the batch is sent after the response so it adds no latency.

### 4. Deploy

Create an API key in **Settings → Keys**. An org admin's key with full access is simplest; see [Authentication](#authentication) for the minimum scopes. Then deploy:

```bash
MAPLE_API_KEY=maple_ak_… alchemy deploy
```

Send a request to the Worker and open [Traces](/docs/explore/traces) in Maple. The `api` service appears within a few seconds.

> **Already have the ingest key in the Worker's env?** If `MAPLE_INGEST_KEY` is set on the Worker another way, call `Telemetry({ serviceName: "api" })` without `ingestKey`. You then don't need `Maple.providers()` or an API key at all.

## Telemetry options

`Telemetry` binds these onto the Worker at deploy time:

| Option        | Bound as                    | Notes                                                                                 |
| ------------- | --------------------------- | ------------------------------------------------------------------------------------- |
| `ingestKey`   | `MAPLE_INGEST_KEY` (secret) | A `Maple.IngestKeys` resource (its private key is used) or a `Redacted` key           |
| `endpoint`    | `MAPLE_ENDPOINT`            | Defaults to `https://ingest.maple.dev`. Use `https://ingest.eu.maple.dev` for EU orgs |
| `environment` | `MAPLE_ENVIRONMENT`         | Becomes `deployment.environment.name`, e.g. your Alchemy stage                        |

Every other option (`serviceName`, `serviceNamespace`, `dropSpanNames`, `anticipatedErrorIdentifiers`, …) is the Effect SDK's and is forwarded unchanged. See [Effect SDK on Cloudflare Workers](/docs/sdks/effect-cloudflare) for the full list.

Import `Telemetry` from the `/telemetry` subpath, not the package root, so the Maple API client stays out of the Worker bundle.

## Add alerts and dashboards

The same stack can declare what you watch the Worker with. Add a notification channel, an alert rule that delivers to it, and a dashboard:

```typescript
// alchemy.run.ts
import * as Alchemy from "alchemy"
import * as Cloudflare from "alchemy/Cloudflare"
import * as Maple from "@maple-dev/alchemy"
import { Effect, Layer, Redacted } from "effect"
import Api from "./src/worker.ts"

export default Alchemy.Stack(
	"my-app",
	{ providers: Layer.mergeAll(Cloudflare.providers(), Maple.providers()) },
	Effect.gen(function* () {
		yield* Api

		const oncall = yield* Maple.AlertDestination("oncall", {
			type: "pagerduty",
			name: "On-call PagerDuty",
			integration_key: Redacted.make(process.env.PAGERDUTY_ROUTING_KEY!),
		})

		// Alchemy creates the destination first, because the rule uses its output.
		yield* Maple.AlertRule("api-errors", {
			name: "API error rate",
			severity: "critical",
			signal_type: "error_rate",
			comparator: "gt",
			threshold: 0.05, // error rates are 0-1 ratios
			window_minutes: 5,
			service_names: ["api"],
			destination_ids: [oncall.destinationId],
		})

		yield* Maple.Dashboard("api-health", {
			name: "API health",
			tags: ["api"],
			time_range: { type: "relative", value: "12h" },
		})
	}),
)
```

Change a threshold and redeploy, and the rule is updated in place. Remove it from the stack and the next deploy deletes it.

| Resource                                                                      | What it manages                                          |
| ----------------------------------------------------------------------------- | -------------------------------------------------------- |
| [`Maple.Dashboard`](/docs/iac/alchemy-resources#mapledashboard)               | Dashboards, their widgets, sections and variables        |
| [`Maple.AlertDestination`](/docs/iac/alchemy-resources#maplealertdestination) | PagerDuty, webhook, Discord, Telegram and email channels |
| [`Maple.AlertRule`](/docs/iac/alchemy-resources#maplealertrule)               | Alert rules on any signal, delivering to destinations    |
| [`Maple.ApiKey`](/docs/iac/alchemy-resources#mapleapikey)                     | Scoped API keys, with in-place rotation                  |
| [`Maple.IngestKeys`](/docs/iac/alchemy-resources#mapleingestkeys)             | Your organization's ingest keys, as outputs (read-only)  |

Every prop and output is in [Alchemy resources](/docs/iac/alchemy-resources).

## Authentication

`Maple.providers()` reads two environment variables at deploy time:

| Variable        | Required | Default                 | Meaning                                                                             |
| --------------- | -------- | ----------------------- | ----------------------------------------------------------------------------------- |
| `MAPLE_API_KEY` | Yes      |                         | A `maple_ak_…` API key with the scopes your resources need                          |
| `MAPLE_API_URL` | No       | `https://api.maple.dev` | API base URL. Use `https://api.eu.maple.dev` for [EU](/docs/reference/regions) orgs |

The key needs these permissions, depending on which resources the stack declares:

| Resource                                    | Scope              | Org admin key |
| ------------------------------------------- | ------------------ | ------------- |
| `Maple.IngestKeys` (used by `Telemetry`)    | `ingest_keys:read` | Yes           |
| `Maple.Dashboard`                           | `dashboards:write` | No            |
| `Maple.AlertRule`, `Maple.AlertDestination` | `alerts:write`     | Yes           |
| `Maple.ApiKey`                              | `api_keys:write`   | Yes           |

A key with no scopes has full access. See [API keys and scopes](/docs/reference/api#authentication).

### Custom configuration or transport

`Maple.providers()` reads the environment and uses the runtime's global `fetch`. To source the key elsewhere, point at another deployment, or use your own `HttpClient`, build the providers with `providersWithDependencies()` and supply both services yourself:

```typescript
import * as Maple from "@maple-dev/alchemy"
import { Layer, Redacted } from "effect"
import { HttpClient } from "effect/http"

const mapleProviders = Maple.providersWithDependencies().pipe(
	Layer.provide(
		Layer.succeed(Maple.MapleEnvironment, {
			baseUrl: "https://api.eu.maple.dev",
			apiKey: Redacted.make(mySecretStore.get("maple")),
		}),
	),
	Layer.provide(Layer.succeed(HttpClient.HttpClient, myHttpClient)),
)
```

`Maple.MapleApiFromHttpClient()` gives you the same API client outside the Alchemy provider collection, for scripts that call the Maple API directly.

---

# Alchemy resources

> Infrastructure as Code · https://maple.dev/docs/iac/alchemy-resources

Every Maple resource the Alchemy provider manages: props, outputs and lifecycle for dashboards, alert rules, alert destinations, API keys and ingest keys.

The resources `@maple-dev/alchemy` manages. New here? Start with the [Alchemy quick start](/docs/iac/alchemy).

| Resource                                           | Lifecycle                                                    | Main output     |
| -------------------------------------------------- | ------------------------------------------------------------ | --------------- |
| [`Maple.Dashboard`](#mapledashboard)               | Create, update in place, delete                              | `dashboardId`   |
| [`Maple.AlertDestination`](#maplealertdestination) | Create, update in place, delete. Changing `type` replaces it | `destinationId` |
| [`Maple.AlertRule`](#maplealertrule)               | Create, update in place, delete. Ownership-tagged per stack  | `ruleId`        |
| [`Maple.ApiKey`](#mapleapikey)                     | Create, roll, revoke. Any other prop change replaces the key | `secret`        |
| [`Maple.IngestKeys`](#mapleingestkeys)             | Read-only singleton                                          | `privateKey`    |

Props use the API's `snake_case` wire shape, exactly as the [interactive API reference](https://api.maple.dev/v2/docs) documents the matching create endpoint. Outputs use `camelCase`.

## Maple.Dashboard

```typescript
Effect.gen(function* () {
	const dash = yield* Maple.Dashboard("checkout", {
		name: "Checkout",
		description: "Golden signals for the checkout flow",
		tags: ["checkout"],
		time_range: { type: "relative", value: "12h" },
		widgets: [/* widget documents */],
		sections: [/* collapsible widget groups */],
		variables: [/* dashboard variables */],
	})
})
```

| Prop          | Type             | Notes                                                                                             |
| ------------- | ---------------- | ------------------------------------------------------------------------------------------------- |
| `name`        | `string`         | Required                                                                                          |
| `description` | `string \| null` |                                                                                                   |
| `tags`        | `string[]`       |                                                                                                   |
| `time_range`  | `object`         | e.g. `{ type: "relative", value: "12h" }`                                                         |
| `widgets`     | `object[]`       | Passed through verbatim                                                                           |
| `sections`    | `object[]`       | Widgets join one with `section_id` and `tab_id`; their `layout` is relative to the section's grid |
| `variables`   | `object[]`       | Passed through verbatim                                                                           |

Outputs: `dashboardId` (`dash_…`), `name`.

The easiest way to get a widget document is to build the dashboard in the UI and copy it from `GET /v2/dashboards/{id}`. See [Build dashboards](/docs/dashboards/build-dashboards).

## Maple.AlertDestination

A notification channel that alert rules deliver to. Props are discriminated on `type`:

| `type`      | Channel fields                                 |
| ----------- | ---------------------------------------------- |
| `pagerduty` | `integration_key`                              |
| `webhook`   | `url`, optional `signing_secret`               |
| `discord`   | `webhook_url`                                  |
| `telegram`  | `bot_token`, `chat_id`                         |
| `email`     | `member_user_ids` (Maple user IDs in your org) |

Every variant also takes `name` (required) and `enabled` (defaults to `true` on both create and update).

```typescript
Effect.gen(function* () {
	const hooks = yield* Maple.AlertDestination("incident-webhook", {
		type: "webhook",
		name: "Incident bot",
		url: "https://hooks.example.com/maple",
		signing_secret: Redacted.make(process.env.WEBHOOK_SECRET!),
	})
})
```

- Channel secrets (`integration_key`, `signing_secret`, `webhook_url`, `bot_token`) are write-only: the API never returns them. Pass them as `Redacted` values so they stay out of logs and plan output. Because they cannot be read back, the provider re-sends them on every update.
- Changing `type` replaces the destination with a new ID.
- Slack, other chat-connector and Hazel destinations depend on integrations linked in Maple, so manage them in [Notification destinations](/docs/alerting/notification-destinations), not here.
- Deleting a destination that rules still reference fails with a conflict. When the rules live in the same stack, Alchemy deletes them first.

Outputs: `destinationId` (`dest_…`), `name`, `type`, `enabled`.

## Maple.AlertRule

```typescript
Effect.gen(function* () {
	yield* Maple.AlertRule("api-p95", {
		name: "API p95 latency",
		severity: "warning",
		signal_type: "p95_latency",
		comparator: "gt",
		threshold: 800,
		window_minutes: 10,
		service_names: ["api"],
		consecutive_breaches_required: 2,
		destination_ids: [oncall.destinationId],
	})
})
```

| Prop                            | Type                      | Notes                                                               |
| ------------------------------- | ------------------------- | ------------------------------------------------------------------- |
| `name`                          | `string`                  | Required. Unique per organization                                   |
| `severity`                      | `"warning" \| "critical"` | Required                                                            |
| `signal_type`                   | `AlertSignalType`         | Required. See values below                                          |
| `comparator`                    | `AlertComparator`         | Required. See values below                                          |
| `threshold`                     | `number`                  | Required. Error rates are 0-1 ratios                                |
| `threshold_upper`               | `number \| null`          | Upper bound for `between` / `not_between`                           |
| `window_minutes`                | `number`                  | Required                                                            |
| `destination_ids`               | `string[]`                | Required. `dest_…` IDs, usually `destination.destinationId` outputs |
| `enabled`                       | `boolean`                 |                                                                     |
| `notes`                         | `string \| null`          |                                                                     |
| `service_names`                 | `string[]`                | Only these services                                                 |
| `exclude_service_names`         | `string[]`                | All services except these                                           |
| `group_by`                      | `string[] \| null`        |                                                                     |
| `tags`                          | `string[]`                | Up to 19. `alchemy:` tags are reserved (see below)                  |
| `minimum_sample_count`          | `number`                  |                                                                     |
| `consecutive_breaches_required` | `number`                  | Breaching windows before it fires                                   |
| `consecutive_healthy_required`  | `number`                  | Healthy windows before it resolves                                  |
| `renotify_interval_minutes`     | `number`                  |                                                                     |
| `alert_on_no_data`              | `boolean`                 | Count a window with no data as a breach                             |
| `apdex_threshold_ms`            | `number \| null`          | For `apdex` rules                                                   |
| `query_builder_draft`           | `object \| null`          | For `builder_query` rules, passed through verbatim                  |
| `raw_query_sql`                 | `string \| null`          | For `raw_query` rules                                               |
| `raw_query_reducer`             | `string \| null`          | `identity`, `sum`, `avg`, `min` or `max`                            |
| `notification_template`         | `object \| null`          |                                                                     |

- `signal_type`: `error_rate`, `p95_latency`, `p99_latency`, `apdex`, `throughput`, `builder_query`, `raw_query`
- `comparator`: `gt`, `gte`, `lt`, `lte`, `eq`, `neq`, `between`, `not_between`

Signal-specific fields are validated by the API against `signal_type`, so a mismatch fails the deploy with a `400`. See [Alert rules](/docs/alerting/alert-rules) for what each signal measures.

Outputs: `ruleId` (`alrt_…`), `name`, `enabled`.

### Ownership

Each rule carries one `alchemy:<hash>` tag that identifies the stack, stage and logical resource that owns it. It uses one of Maple's 20 tag slots, which is why `tags` is capped at 19. The tag is what makes state recovery safe:

- If Alchemy state is lost, the provider finds the rule by name and takes it back only when the ownership tag matches.
- A rule with the same name and no matching tag (created in the UI, or by another stack or stage) fails the deploy instead of being overwritten. To take it over deliberately, wrap the resource in Alchemy's `adopt(true)` or deploy with `--adopt`.
- A stack never deletes a rule another stack has adopted.

Keep the ownership tag intact, and use distinct rule names for different stages that share one Maple organization.

## Maple.ApiKey

```typescript
Effect.gen(function* () {
	const ci = yield* Maple.ApiKey("ci", {
		name: "ci-pipeline",
		scopes: ["dashboards:write", "alerts:read"],
		rotate: 1,
	})
	// ci.secret is a Redacted<string>; bind it into another resource's env.
})
```

| Prop                 | Type                  | Notes                                                                                 |
| -------------------- | --------------------- | ------------------------------------------------------------------------------------- |
| `name`               | `string`              | Required                                                                              |
| `description`        | `string`              |                                                                                       |
| `scopes`             | `string[]`            | Omit for full access                                                                  |
| `kind`               | `"standard" \| "mcp"` | Defaults to `standard`. `mcp` keys only work on the [MCP server](/docs/reference/mcp) |
| `expires_in_seconds` | `number`              |                                                                                       |
| `rotate`             | `number \| string`    | Bump it (`1` to `2`) to roll the key                                                  |

The API has no update endpoint for keys:

- Changing `name`, `description`, `scopes`, `kind` or `expires_in_seconds` **replaces** the key: a new ID and a new secret, and the old key is revoked.
- Bumping `rotate` **rolls** the key in place: same ID, name and scopes, new secret.
- Deleting the resource revokes the key.

Outputs: `keyId` (`key_…`), `name`, `keyPrefix`, `secret` (`Redacted`, the full `maple_ak_…` value).

The API returns a secret only once, at create or roll time. The provider stores it in Alchemy state, which is the only place it survives, so keep your state somewhere durable and encrypted.

## Maple.IngestKeys

```typescript
Effect.gen(function* () {
	const ingest = yield* Maple.IngestKeys("ingest")
	// ingest.publicKey  -> maple_pk_… for browser and mobile senders
	// ingest.privateKey -> maple_sk_… for server-side senders
})
```

A read-only view of your organization's [ingest keys](/docs/reference/authentication). It takes no props, and there is one per organization. Deleting the resource only stops tracking it; the keys stay valid. Rolling is not modeled, because a roll immediately invalidates the old key for every sender. Roll ingest keys from **Settings → Ingestion** instead.

Outputs: `publicKey`, `privateKey` (both `Redacted`), `publicRotatedAt`, `privateRotatedAt`.

## Drift and sync

- A deploy compares declared props against what the API returns and patches only when they differ. Fields you leave out of props stay unmanaged, so edits to them in the UI are kept.
- `alchemy sync` restores declared dashboard fields and alert rule configuration to what the stack declares.
- Alert destination secrets cannot be compared, so drift in them is only caught when you change the prop.

## Errors and retries

The client retries `429` and `5xx` responses with bounded exponential backoff and honors `retry_after_seconds` from the [error envelope](/docs/reference/api#errors). The API allows 600 requests per 60 seconds per key, which a large stack stays well under.

Failures surface as typed errors in Effect's error channel. An API failure is tagged with the server's own `@maple/http/…` error tag and carries the HTTP `status` and the [error body](/docs/reference/api#errors) (`type`, `code`, `message`); `Maple.MapleErrorTags` has the tags to `catchTag` on. Client-side failures are `MapleApiTransportError`, `MapleApiResponseDecodeError`, `MapleApiProtocolError` and `MapleApiRequestEncodingError`. Rule ownership conflicts are `MapleAlertRuleOwnershipError`, and invalid or reserved rule tags are `MapleAlertRuleTagsError`.

---

# Maple Local

> Local Mode · https://maple.dev/docs/local-mode

Run the whole of Maple as one binary on your machine: OTLP ingest, an embedded ClickHouse, a query API, the dashboard and a CLI. No account, no containers, nothing to deploy.

import LocalHero from "../../components/local/LocalHero.astro"
import LocalFlow from "../../components/local/LocalFlow.astro"
import LocalCommandGrid from "../../components/local/LocalCommandGrid.astro"
import InstallTabs from "../../components/local/InstallTabs.astro"

<LocalHero />

Maple Local is the full product running on your machine, free, for local development. Point any OpenTelemetry exporter at `localhost`, open the dashboard, and explore traces, logs, metrics, errors and a service map. It is not a client for the hosted service and needs no account. Your telemetry is stored and queried on your machine. The CLI does make a few network calls of its own by default, listed in [What connects to the internet](#what-connects-to-the-internet).

## What you get

| Surface              | What it is                                                                                                                                                                   |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **OTLP/HTTP ingest** | `POST /v1/{traces,logs,metrics}` on port `4318`, protobuf or JSON, gzip optional. Any OTel SDK works unchanged.                                                              |
| **Dashboard**        | The same Maple UI, either auto-updating from `local.maple.dev` or bundled in the binary for offline use.                                                                     |
| **CLI**              | `maple services`, `maple traces`, `maple errors` and the other query commands in the [CLI reference](/docs/reference/cli), printing JSON you can pipe into `jq` or an agent. |
| **Raw SQL**          | `maple query "<sql>"` against the embedded store when the typed commands don't cover it.                                                                                     |

Data lives in `~/.maple/data` and survives restarts. See [Your data](#your-data) for retention and restore points.

## 1. Install

Homebrew is the easiest path on macOS and Linux. The install script works anywhere a shell does.

<InstallTabs />

> Prefer to do it by hand? Download a release bundle (both `maple` and `libchdb`) from [GitHub Releases](https://github.com/MapleTechLabs/maple/releases), or read [`scripts/install.sh`](https://github.com/MapleTechLabs/maple/blob/main/scripts/install.sh) before piping it to a shell. Later, `maple update` upgrades a script install in place; Homebrew installs use `brew upgrade maple`.

## 2. Start the server

```bash
maple start            # ingest + embedded ClickHouse + query API on :4318
maple start --offline  # …and serve the dashboard from the binary
maple start -d         # …detached; logs to ~/.maple/maple.log, stop with `maple stop`
```

`maple start` is the one long-lived process. It owns the embedded database and hosts ingest, the query API and, with `--offline`, the dashboard, all on a single port bound to loopback. The startup banner prints the addresses and a link to open.

The flags you are likely to touch:

| Flag                 | Default         | Use it when                                                           |
| -------------------- | --------------- | --------------------------------------------------------------------- |
| `--port <int>`       | `4318`          | Something else already listens on 4318                                |
| `--data-dir <path>`  | `~/.maple/data` | You want a second, separate store                                     |
| `--offline`          | off             | You have no internet, or want the UI same-origin                      |
| `--background`, `-d` | off             | You want it out of the way; `maple stop` ends it                      |
| `--reset`            | off             | A new binary refuses an old store; wipes live data, keeps checkpoints |

Everything else, including bind and advertise hosts, checkpoint cadence and recovery policy, is in the [CLI reference](/docs/reference/cli#maple-start).

## 3. Send telemetry

Point your app at the server. No auth header is needed locally, and most exporters default to protobuf, so this is usually all it takes:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="http://127.0.0.1:4318"
export OTEL_SERVICE_NAME="my-service"
```

For custom spans, log correlation and framework auto-instrumentation, follow the guide for your language in [Instrument your application](/docs/instrumentation). Anything you can send to hosted Maple flows into the local server unchanged.

<LocalFlow />

## 4. Open the dashboard

The banner links to the dashboard. There are two ways to serve it:

|                         | Default (`local.maple.dev`)                                | `--offline`               |
| ----------------------- | ---------------------------------------------------------- | ------------------------- |
| Where the UI comes from | Hosted, always the latest build                            | Bundled inside the binary |
| Talks to your data over | Loopback, from a public origin                             | Same origin               |
| Needs internet          | Yes, to load the page                                      | No                        |
| Browser prompt          | Chrome asks once to "access devices on your local network" | None                      |

Both read the same local store. Pick `--offline` when you are on a plane, behind a strict proxy, or would rather not see the prompt.

### Reaching it from another machine

Bind to all interfaces and print a hostname the other browser will actually use:

```bash
maple start --host 0.0.0.0 --advertise-host maple.home.arpa --offline
```

Anyone who can reach the port can then use the UI, run raw SQL through `/local/query`, and send OTLP data. None of these routes check credentials. The maintenance routes (checkpoint, day retirement and the local event routes) require a token that `maple start` writes beside the data directory with mode `0600` (`<data-dir>.maintenance-token`, and `<data-dir>.event-consumer-token` for event consumers). They are only as safe as those files. `/local/retention/retire` deletes a day of live data. The full route list is in the [CLI reference](/docs/reference/cli#server-endpoints).

Use a non-loopback bind only on a network you trust, or behind TLS with browser-managed auth such as a session cookie. The bundled UI does not attach API keys to its requests.

## Query from the terminal

The same binary is the query CLI. Every command runs against the running server and prints JSON by default. Add `--format table` for an aligned table, or `--debug` to see the compiled SQL on stderr.

```bash
maple services                         # active services at a glance
maple traces --service api --since 1h  # recent spans for one service
maple errors --since 24h               # error groups by fingerprint
maple query "SELECT count() FROM traces"
```

<LocalCommandGrid />

Most query commands share the same filters: `--since` (`30m`, `1h`, `24h`, `7d`) or absolute `--start`/`--end`, `--service`/`-s`, `--env`/`-e`, and `--limit`/`-n`. The [CLI reference](/docs/reference/cli) has every command, argument and flag.

## Your data

- **Where it lives.** `~/.maple/data`, or whatever `--data-dir` you passed.
- **Retention.** Logs and traces are kept for 30 days, metrics for 90. `maple start --minimum-raw-telemetry-retention-days <n>` raises the floor for raw logs, traces and metrics (90 to 3,650 days). Hosted retention is on [Retention](/docs/reference/retention).
- **Restore points.** The running server takes a checkpoint on a timer, and `maple restore` rolls the store back to one. Cadence, restore and unclean-shutdown behavior are on [Checkpoints and archives](/docs/local-mode/checkpoints-and-archives).
- **Starting over.** `maple reset` clears live data and keeps checkpoints. `maple start --reset` does the same in one step after an incompatible upgrade.
- **Keeping history.** `maple archive` exports whole days as Parquet files you can open in DuckDB. See [Archives](/docs/local-mode/checkpoints-and-archives#archives).

## What connects to the internet

Your telemetry and query results stay on your machine. These calls go out by default:

| Call                                                                                                          | When                                                       | Turn it off                                          |
| ------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- | ---------------------------------------------------- |
| The CLI's own traces and metrics, sent to `https://ingest.maple.dev` with an ingest key built into the binary | Every command, and every request `maple start` handles     | `MAPLE_TELEMETRY=off`                                |
| Release check against the GitHub releases API                                                                 | At most once per 24 hours, only in an interactive terminal | `MAPLE_NO_UPDATE_CHECK=1` (Homebrew installs set it) |
| The dashboard page from `local.maple.dev`                                                                     | When you open the dashboard                                | `maple start --offline`                              |

The CLI's own telemetry describes the CLI, not your data. It reports the service name `maple-cli`, the command, its duration and errors. For each OTLP request `maple start` handles it records the signal, the body size and the number of items, never the payload. For each `/local/query` request it records only the timing and whether it succeeded. SQL text, filter values and error messages are never exported, and the CLI's own logs are not sent at all. `MAPLE_INGEST_KEY`, `MAPLE_ENDPOINT` and `MAPLE_ENVIRONMENT` redirect or relabel this export; see [Environment variables](/docs/reference/cli#environment-variables).

## One CLI, two backends

The same `maple` also talks to a hosted Maple workspace. Sign in from the terminal and it opens your browser:

```bash
maple auth login     # browser sign-in; `--with-token` reads a token from stdin instead
maple whoami         # which backend a command would hit right now
maple use local      # pin it; `maple use auto` restores detection
```

Per command, an explicit `--local` or `--remote` wins, then the pinned default, then auto-detect: a stored login means remote, otherwise the CLI probes the local server. Everything except `maple query` works against both. Raw SQL stays local because the hosted warehouse is shared between organizations. See [Using the CLI with hosted Maple](/docs/reference/cli#using-the-cli-with-hosted-maple).

## Under the hood

One compiled binary is both the CLI and the server. It talks to ClickHouse in-process through `libchdb`, so there is no subprocess and no second runtime to install. Short-lived CLI commands and the dashboard read the store over the same `POST /local/query` contract. The architecture, the UI origin model and the release bundle are described in the [local mode design doc](https://github.com/MapleTechLabs/maple/blob/main/docs/local-mode.md).

---

# Checkpoints and archives

> Local Mode · https://maple.dev/docs/local-mode/checkpoints-and-archives

How Maple Local protects your telemetry: automatic restore points, what happens after an unclean shutdown, reset and restore, and Parquet archives for long-term history.

Maple Local keeps everything in one embedded ClickHouse store under `~/.maple/data`. Three mechanisms keep that store safe, and one keeps it small:

| Mechanism              | What it protects against                   | Command                              |
| ---------------------- | ------------------------------------------ | ------------------------------------ |
| **Checkpoints**        | A crash, a bad upgrade, a mistake          | `maple checkpoint`, `maple restore`  |
| **Dirty-store policy** | Silent data loss after an unclean shutdown | `maple start --on-dirty-store`       |
| **Journaled reset**    | Deleting more than the store               | `maple reset`, `maple start --reset` |
| **Archives**           | The live store growing without bound       | `maple archive …`                    |

When a destructive step is uncertain, the CLI stops and prints the paths it kept. It does not guess.

## Checkpoints

A checkpoint is a snapshot of the store that has been test-restored before it is kept. The running server takes one every 30 minutes by default, so a crash loses at most 30 minutes of telemetry. Change the cadence with `maple start --checkpoint-interval` (`45s`, `2h`, or `off` to disable).

```bash
maple start
maple checkpoint      # take one now, on top of the automatic ones
```

`maple checkpoint` asks the running server to take the checkpoint. If you started the server with a custom `--host`, `--port` or `--data-dir`, pass the same values.

### What a checkpoint contains

Each finished checkpoint gets a UUID and lives under `<data-dir>/backups`:

```text
<data-dir>/backups/
  state.json                   # which checkpoints are current and previous
  snapshots/<checkpoint-id>/
    backup/                    # the ClickHouse backup
    manifest.json              # written only after the backup passed its test restore
  operations/                  # records of reset, restore and archive runs
  pins/                        # checkpoints an archive export is reading from
  quarantine/                  # stores moved aside by restore, never deleted
  retiring/
```

Before a new checkpoint becomes the current one, Maple restores it into a temporary instance and checks all six raw telemetry tables (logs, traces and the four metric tables). If that check fails, the previous checkpoint stays current. Maple keeps the two most recent good checkpoints. When a third one succeeds, the oldest is deleted, unless an archive export is still reading from it or it cannot be confirmed as complete. In either of those cases it is kept.

### Bring your own ClickHouse config

Checkpoints need backups enabled in the config of the **running** server, because that process writes the backup. `maple start` generates such a config at `~/.maple/chdb-config.xml` by default. If you pass your own `--chdb-config-file`, it must include:

```xml
<clickhouse>
  <backups>
    <allowed_disk>default</allowed_disk>
    <allowed_path>backups</allowed_path>
  </backups>
</clickhouse>
```

## Restoring

`maple restore` puts the store back to the current checkpoint, or to one you name:

```bash
maple stop
maple restore --yes
maple restore --checkpoint-id 01234567-89ab-4cde-8fab-0123456789ab --yes
```

The server must be stopped. The store you are replacing is moved beside the data directory as `<data-dir>.quarantine-<ids>`, not deleted, so you can go back to it. `maple schema gc` lists what was set aside and `maple schema gc --apply` removes it. If a restore is interrupted, running `maple restore` again finishes it. Checkpoint and restore never run at the same time: if a server still owns the store, the command reports it as busy and does nothing.

## After an unclean shutdown

If the server did not close the store cleanly, the next `maple start` applies the `--on-dirty-store` policy:

| Policy               | What happens                                                                                                            |
| -------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `fail` (default)     | Refuse to start and say so. Nothing is deleted; choose one of the others deliberately                                   |
| `restore-checkpoint` | Roll back to the current checkpoint and move the dirty store beside the data directory as `<data-dir>.quarantine-<ids>` |
| `wipe`               | Discard the live data and start empty. Checkpoints are kept and pinned                                                  |

A detached start (`-d`) passes the policy to the background process unchanged. A store whose schema the binary does not recognize also refuses to start until you reset it or run [`maple schema migrate`](/docs/reference/cli#maple-schema).

## Reset

`maple reset` (or `maple start --reset`) clears the live data and keeps all checkpoints, so a fresh store still has restore points behind it. They are pinned, so later refreshes never rotate them out; release them with `maple schema gc --apply --release-preserved`.

A reset removes only the directories the embedded engine owns: `control`, `data`, `metadata`, `status`, `store` and `tmp`. If it finds anything else in the data directory, it changes nothing and lists the unexpected paths. If a reset is interrupted, the next `maple start` finishes it before opening the store.

## Archives

The live store keeps logs and traces for 30 days and metrics for 90 (see [Your data](/docs/local-mode#your-data)). Archives keep older history as Parquet files you can query with DuckDB. An export reads from a checkpoint, never from the live store, and the dashboard does not read archives.

An archive covers one UTC day of one **signal**: `logs`, `traces`, `metrics_sum`, `metrics_gauge`, `metrics_histogram` or `metrics_exponential_histogram`. Aggregation tables are not archived; they can be rebuilt from the raw data.

```bash
maple archive create 2026-09-20 traces              # export one day of one signal
maple archive list                                  # what is archived
maple archive list --output paths --signal traces   # file paths, ready for DuckDB
maple archive verify                                # re-check every file's SHA-256
maple archive retire-live 2026-09-20 --apply        # delete the day from the live store once it is fully archived
```

`archive create` exports the day from the current checkpoint (or the one you pass with `--checkpoint-id`), then checks row counts and checksums before the archive appears in `archive list`. The checkpoint is not deleted while the export runs. Exporting the same day again, for example after late data arrives, writes a new copy that replaces the old one in `archive list`; if the new copy has fewer rows it is refused unless you pass `--allow-shrink`. Days past their retention are refused, since the checkpoint no longer holds them. The old files stay on disk until `maple archive gc --apply` removes them (without `--apply`, gc only reports what it would delete).

Every subcommand and flag is in the [CLI reference](/docs/reference/cli#maple-archive). The on-disk layout and the recovery rules for interrupted runs are in the [local telemetry archives design doc](https://github.com/MapleTechLabs/maple/blob/main/docs/local-telemetry-archives.md).

---

# Maple API

> Reference · https://maple.dev/docs/reference/api

The Maple REST API: base URLs, API keys and scopes, the resources it exposes, pagination, the error envelope, rate limits, and where the OpenAPI specification lives.

The Maple API is the HTTP interface to your Maple organization. It covers dashboards, alert rules and destinations, error issues, scrape targets, ingest and API keys, and read access to traces, logs, metrics, services and session replays. The dashboard uses the same endpoints.

|                           |                                                                                                                           |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| Base URL (US)             | `https://api.maple.dev/v2`                                                                                                |
| Base URL (EU)             | `https://api.eu.maple.dev/v2`                                                                                             |
| Interactive reference     | [api.maple.dev/v2/docs](https://api.maple.dev/v2/docs) (EU: [api.eu.maple.dev/v2/docs](https://api.eu.maple.dev/v2/docs)) |
| OpenAPI 3.1 specification | [maple.dev/openapi.json](/openapi.json) (also [api.maple.dev/openapi.json](https://api.maple.dev/openapi.json))           |
| MCP server for AI agents  | [Maple MCP server](/docs/reference/mcp)                                                                                   |
| Auth                      | `Authorization: Bearer maple_ak_…`                                                                                        |

Use the base URL of your organization's [region](/docs/reference/regions). A key created in one region does not work in the other.

## Authentication

Create an API key in the dashboard under **Settings → Keys**, or with `POST /v2/api_keys` using an existing key. Send it as a Bearer token on every request:

```bash
curl https://api.maple.dev/v2/services \
  -H "Authorization: Bearer maple_ak_…"
```

Keys can be **scoped** at creation. A scope is `<family>:read`, `<family>:write`, or `*`. The family is the first path segment under `/v2`, listed in the resource index below. `write` implies `read`, and a key with no scopes has full access. `GET` requests need `read`; `POST`, `PATCH`, `PUT` and `DELETE` need `write`, except the search, timeseries, breakdown and preview `POST`s, which need only `read`. A request outside a key's scopes fails with `403`, `type: "permission_error"` and `code: "insufficient_scope"`.

Keys belong to one organization. [Authentication](/docs/reference/authentication) covers every credential Maple accepts, including dashboard session tokens and the `x-maple-org-id` header.

## Resources

Each row is one scope family. The [interactive reference](https://api.maple.dev/v2/docs) documents every operation.

| Family               | Endpoints                                                                                |
| -------------------- | ---------------------------------------------------------------------------------------- |
| `api_keys`           | List, create, get, revoke and roll API keys                                              |
| `ingest_keys`        | Read the organization's public and private ingest keys, and roll either one              |
| `organization`       | The current organization                                                                 |
| `audit_log`          | The organization's audit log                                                             |
| `dashboards`         | Dashboards, templates, version history and restore, public share links, Perses import    |
| `alerts`             | Alert rules (with test and preview), rule checks, destinations, incidents and deliveries |
| `error_issues`       | Error issues and per-service issue counts                                                |
| `traces`             | Search traces, time series and breakdowns, one trace, one span                           |
| `logs`               | Search logs, time series and breakdowns, one log record                                  |
| `metrics`            | List metrics, time series and breakdowns                                                 |
| `services`           | Services, one service, and a service overview                                            |
| `service_map`        | Service dependency edges                                                                 |
| `environments`       | Deployment environments seen in your data                                                |
| `session_replays`    | Search replays, the replays for a trace, one replay's manifest, events and transcript    |
| `scrape_targets`     | Prometheus scrape targets, their checks, and a test probe                                |
| `attribute_mappings` | Attribute mappings                                                                       |
| `instrumentation`    | Instrumentation recommendations, the setup audit, and which signals each service sends   |
| `integrations`       | Chat connectors and the PlanetScale integration                                          |

## Conventions

- Resources are plural nouns under `/v2` (`/v2/api_keys`, `/v2/alerts/rules`). Actions that are not plain CRUD are `POST`s to a sub-resource (`POST /v2/api_keys/{id}/roll`). Complex reads are `POST …/search`.
- Every object carries an `object` field and a prefixed, opaque public ID (`key_…`, `dash_…`, `alrt_…`).
- The wire format is snake_case JSON with ISO-8601 UTC timestamps. Nullable fields are explicit `null`.
- Updates are JSON `PATCH` bodies.

## Pagination

List endpoints accept `limit` (1 to 100, default 20) and an opaque `cursor`. They return:

```json
{ "object": "list", "data": [], "has_more": true, "next_cursor": "…" }
```

To read the next page, send the same request with `cursor` set to `next_cursor`. Stop when `has_more` is `false`. A `limit` outside 1 to 100 is a `400`.

## Errors

Every failure, including an unknown route, returns a JSON envelope with the same shape:

```json
{
	"error": {
		"_tag": "@maple/http/v2/RouteNotFoundError",
		"type": "not_found_error",
		"code": "route_not_found",
		"title": "No such route",
		"message": "No route matches GET /v2/typo. The Maple API is documented at https://api.maple.dev/v2/docs; the OpenAPI specification is at https://api.maple.dev/openapi.json.",
		"retryable": false,
		"recovery": "fix_request"
	}
}
```

| Field              | Meaning                                                                                                                                                                                                                         |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `_tag`             | The exact failure. Branch on this. The OpenAPI spec lists the tags each operation can return                                                                                                                                    |
| `type`             | The status family: `invalid_request_error` (400), `authentication_error` (401), `payment_error` (402), `permission_error` (403), `not_found_error` (404), `conflict_error` (409), `rate_limit_error` (429) or `api_error` (5xx) |
| `code`             | A short category for display                                                                                                                                                                                                    |
| `title`, `message` | Human-readable text, safe to show to users                                                                                                                                                                                      |
| `retryable`        | `true` when the same request can succeed later. The response then also carries `Retry-After`                                                                                                                                    |
| `recovery`         | What the client should do: `none`, `fix_request`, `reauthenticate`, `request_access`, `reconnect`, `refresh`, `retry` or `contact_support`                                                                                      |

Stack traces and upstream error messages are never included.

## Rate limits

API-key requests share **600 requests per 60 seconds per key** across the whole `/v2` surface. Over the budget you get `429` with `type: "rate_limit_error"`, `code: "rate_limited"`, and `Retry-After: 60`. See [Limits](/docs/reference/limits) for the rest.

## OpenAPI specification

Every operation in the [OpenAPI document](/openapi.json) has a unique `operationId`, a `summary` and `description`, typed parameters, request and response schemas, and the error `_tag`s it can return. You can import it into Postman, Insomnia, Scalar or `openapi-generator`, or point an LLM function-calling tool at `https://maple.dev/openapi.json`. The [interactive reference](https://api.maple.dev/v2/docs) renders the same document.

To connect an AI agent without generating a client, use the [MCP server](/docs/reference/mcp).

## Versioning

`/v2` is the major version. Changes within it are additive, and error `_tag` values do not change. The v1 endpoints under `/api/…` remain available for existing integrations but get no new features.

---

# Maple MCP server

> Reference · https://maple.dev/docs/reference/mcp

Connect Claude, Cursor and other AI agents to your Maple telemetry over the Model Context Protocol: endpoints, authentication, and every tool the server exposes.

Maple runs a hosted **Model Context Protocol (MCP)** server. An AI agent connected to it can list services, search traces and logs, triage error issues, read and edit dashboards, and manage alert rules in your organization.

|                          |                                                                                                                                            |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
| Endpoint (US)            | `https://api.maple.dev/mcp`                                                                                                                |
| Endpoint (EU)            | `https://api.eu.maple.dev/mcp`                                                                                                             |
| Transport                | Streamable HTTP                                                                                                                            |
| Manifest (`server.json`) | [maple.dev/.well-known/mcp.json](/.well-known/mcp.json) · [api.maple.dev/.well-known/mcp.json](https://api.maple.dev/.well-known/mcp.json) |
| Registry name            | `dev.maple/maple`                                                                                                                          |
| Auth                     | Maple API key as Bearer token, or OAuth 2.1                                                                                                |
| Rate limit               | 120 requests per 10 seconds per key or user; over it, `429` with `Retry-After: 10`                                                         |

Use the endpoint of your organization's [region](/docs/reference/regions). The examples below use the US endpoint; for an EU organization replace `api.maple.dev` with `api.eu.maple.dev`.

## Connecting a client

Any MCP client that speaks Streamable HTTP can connect. **Settings → MCP** in the dashboard shows the endpoint for your region and ready-made configuration for common clients. There are two ways to authenticate. Both are described in more detail on [Authentication](/docs/reference/authentication).

**API key.** Create a key under **Settings → MCP** or **Settings → Keys** and pass it as a Bearer token:

```json
{
	"mcpServers": {
		"maple": {
			"type": "http",
			"url": "https://api.maple.dev/mcp",
			"headers": { "Authorization": "Bearer maple_ak_…" }
		}
	}
}
```

For Claude Code: `claude mcp add --transport http maple https://api.maple.dev/mcp --header "Authorization: Bearer maple_ak_…"`.

**OAuth.** Clients that support the MCP authorization flow need no key. The server publishes its protected-resource metadata at `https://api.maple.dev/.well-known/oauth-protected-resource/mcp` and its authorization server metadata at `https://api.maple.dev/.well-known/oauth-authorization-server`. Dynamic client registration and PKCE (`S256`) are supported, so the client needs only the endpoint URL. It opens a browser sign-in on first use.

Requests are scoped to the organization the key belongs to, or the one chosen at sign-in. A user in several organizations picks one during the OAuth consent step.

## Tools

The server exposes the tools below. Read-only tools are marked **read**; the rest change data in your organization. `tools/list` returns the current set with full parameter schemas.

### Services and discovery

| Tool                         | Access | What it does                                                                                         |
| ---------------------------- | ------ | ---------------------------------------------------------------------------------------------------- |
| `list_services`              | read   | Active services with throughput, error rate and P95 latency                                          |
| `diagnose_service`           | read   | One service's health, top errors, recent traces and logs, and the platform resources behind it       |
| `get_service_top_operations` | read   | A service's top operations (endpoints or spans), ranked by a chosen metric                           |
| `service_map`                | read   | Service-to-service dependencies with call counts, error rates and latency per edge                   |
| `compare_periods`            | read   | Error rate, throughput and P95 across two periods, overall and per service, with regressions flagged |
| `explore_attributes`         | read   | Attribute keys and their values for filtering; `source=services` lists environments and commit SHAs  |
| `list_metrics`               | read   | Custom metrics with type, unit, monotonicity and data volume                                         |
| `describe_warehouse_tables`  | read   | Tables and columns available to raw SQL, with sorting keys and notes on units and casing             |

### Traces, logs and queries

| Tool                | Access | What it does                                                                                 |
| ------------------- | ------ | -------------------------------------------------------------------------------------------- |
| `search_traces`     | read   | Find traces by service, duration, error status, HTTP method, span name or one span attribute |
| `find_slow_traces`  | read   | The slowest traces with p50, p95, min and max for context                                    |
| `inspect_trace`     | read   | Span tree and logs for one trace                                                             |
| `inspect_span`      | read   | Full attribute set for one span; decodes messages and tool calls on AI agent spans           |
| `search_logs`       | read   | Log entries, newest first, filtered by service, severity, body text, trace or span           |
| `mine_log_patterns` | read   | Cluster log messages into templates with counts and a severity and service breakdown         |
| `query_data`        | read   | Aggregate traces, logs, metrics or product events into a time series or top-N breakdown      |
| `run_sql`           | read   | Run one read-only ClickHouse `SELECT`; see the [SQL reference](/docs/reference/sql)          |

### Errors

| Tool                               | Access | What it does                                                                                        |
| ---------------------------------- | ------ | --------------------------------------------------------------------------------------------------- |
| `find_errors`                      | read   | Errors grouped by type with counts, affected services and a stable fingerprint                      |
| `error_detail`                     | read   | Sample traces and correlated logs for one error fingerprint                                         |
| `list_error_issues`                | read   | Error issues with workflow state, counts, assignee and lease holder                                 |
| `list_error_issue_events`          | read   | An issue's audit log: state changes, claims, comments, agent notes and fix proposals                |
| `list_error_incidents`             | read   | Time-bounded flare-ups under an error issue (first seen and regressions)                            |
| `claim_error_issue`                | write  | Take a lease on an issue so other agents do not duplicate the work                                  |
| `release_error_issue`              | write  | Give up your lease, optionally moving the issue to another state                                    |
| `transition_error_issue`           | write  | Move an issue to another workflow state                                                             |
| `set_issue_severity`               | write  | Set or clear an issue's triage severity                                                             |
| `comment_on_error_issue`           | write  | Add a comment or agent note to an issue's timeline                                                  |
| `propose_fix`                      | write  | Record a proposed fix and move the issue to `in_review`                                             |
| `link_pull_request`                | write  | Attach a GitHub pull request to an issue; Maple verifies the fix after it merges                    |
| `register_agent`                   | write  | Register an LLM agent so it can claim and transition issues. Needs a signed-in user, not an API key |
| `update_error_notification_policy` | write  | Choose whether error incidents notify alert destinations, and which ones. Org admins only           |

### Alerting

| Tool                      | Access | What it does                                                                             |
| ------------------------- | ------ | ---------------------------------------------------------------------------------------- |
| `list_alert_rules`        | read   | Alert rules with severity, signal type and condition                                     |
| `get_alert_rule`          | read   | One rule's full configuration: thresholds, filters, evaluation settings and destinations |
| `list_alert_destinations` | read   | Notification destinations with id, type and delivery health                              |
| `list_alert_incidents`    | read   | Triggered alert incidents, open and resolved, with the last observed value               |
| `get_incident_timeline`   | read   | Trigger, notification and resolution times for one rule's incidents, or all rules'       |
| `list_alert_checks`       | read   | A rule's recent evaluations with observed value, threshold and sample count              |
| `preview_alert_rule`      | read   | Replay a saved or draft rule over past data: each window's verdict and skip reason       |
| `create_alert_rule`       | write  | Create an alert rule from a template or from signal type, comparator and threshold       |
| `update_alert_rule`       | write  | Change the fields you pass on an alert rule                                              |
| `delete_alert_rule`       | write  | Delete an alert rule and its incident history. Requires `confirm=true`                   |

### Dashboards

| Tool                        | Access | What it does                                                                             |
| --------------------------- | ------ | ---------------------------------------------------------------------------------------- |
| `list_dashboards`           | read   | Dashboards with widget counts and timestamps                                             |
| `get_dashboard`             | read   | One dashboard's full configuration, every widget included                                |
| `describe_dashboard_schema` | read   | What a widget can be: panel types, data sources, units, aggregations and display options |
| `inspect_chart_data`        | read   | Run a widget's query and report row counts, series statistics and a health verdict       |
| `create_dashboard`          | write  | Create a dashboard from a template, simplified widget specs or full JSON                 |
| `update_dashboard`          | write  | Change a dashboard's name, description or time range                                     |
| `add_dashboard_widget`      | write  | Add one widget, from the query builder, raw SQL or a funnel                              |
| `update_dashboard_widget`   | write  | Replace one widget with a full widget object                                             |
| `remove_dashboard_widget`   | write  | Remove one widget                                                                        |
| `reorder_dashboard_widgets` | write  | Move or resize widgets                                                                   |
| `replace_dashboard_widgets` | write  | Replace every widget in one validated write                                              |

### Session replays and product events

| Tool                     | Access | What it does                                                                              |
| ------------------------ | ------ | ----------------------------------------------------------------------------------------- |
| `search_sessions`        | read   | Find browser session replays by user, client, errors, duration or what happened in them   |
| `get_session_transcript` | read   | A session replay as a text transcript of navigation, clicks, console, network and errors  |
| `get_session_traces`     | read   | The backend traces a browser session produced                                             |
| `list_product_events`    | read   | Product event names with how often each fired and how many sessions and people it reached |
| `query_funnel`           | read   | Conversion funnel over product events, with step counts, conversion and drop-off          |

### Agent sessions

| Tool                           | Access | What it does                                                                         |
| ------------------------------ | ------ | ------------------------------------------------------------------------------------ |
| `list_agent_sessions`          | read   | AI agent sessions with agent, models, LLM and tool calls, failures, tokens and cost  |
| `get_agent_session`            | read   | One agent session: verdict, checks, findings, timing, turns, tokens and cost         |
| `get_agent_session_transcript` | read   | One agent session step by step: prompts, replies, tool calls and results, sub-agents |
| `get_agent_tools_overview`     | read   | Calls, failures and latency per agent tool, compared with the previous window        |
| `get_agent_tool_error`         | read   | One agent tool failure group with affected sessions and sample calls                 |

### Setup and source code

| Tool                                  | Access | What it does                                                                                           |
| ------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------ |
| `audit_setup`                         | read   | Check the whole organization setup: alert routing, error notifications and per-service ingest coverage |
| `get_instrumentation_recommendations` | read   | Span and resource attribute problems found in your live data                                           |
| `list_integrations`                   | read   | Connected integrations, what each one feeds, and which resources are failing or stale                  |
| `list_source_repositories`            | read   | Repositories connected through the [GitHub integration](/docs/integrations/github)                     |
| `search_source_code`                  | read   | Search one connected repository through GitHub code search                                             |
| `read_source_file`                    | read   | Read a line range from a file in a connected repository                                                |
| `ci_runs`                             | read   | GitHub Actions runs from CI traces: failures per workflow, durations, and why a failed job failed      |
| `compare_commits`                     | read   | The commits between two deployed versions of a service, from the synced repository                     |

The source-code tools only return results for repositories the organization's GitHub App installation can access.

### Integrations

| Tool                   | Access | What it does                                                                     |
| ---------------------- | ------ | -------------------------------------------------------------------------------- |
| `sync_railway`         | write  | Poll Railway now and report each environment's last sync                         |
| `create_scrape_target` | write  | Start scraping a Prometheus endpoint that needs no credentials                   |
| `update_scrape_target` | write  | Pause, resume or change a Prometheus scrape target                               |
| `delete_scrape_target` | write  | Delete a Prometheus scrape target and its check history; its stored metrics stay |

These are admin-only. Connecting a provider, and adding a scrape credential, stay in Settings → Integrations.

### Feedback

| Tool                  | Access | What it does                                                                                            |
| --------------------- | ------ | ------------------------------------------------------------------------------------------------------- |
| `send_maple_feedback` | write  | Tell the Maple team about a bug, a missing capability or a misleading doc: its kind, the agent, and why |

The same feedback can be sent over the REST API with `POST /v2/agent_feedback`.

## Prompts and resources

The server also provides three prompts (`incident_triage`, `latency_analysis`, `debug_errors`) and one resource, `maple://instructions`, which explains Maple's data model to the model.

## Related

- [Maple API](/docs/reference/api): the REST API, with its [OpenAPI spec](/openapi.json)
- [Authentication](/docs/reference/authentication): API keys and OAuth
- [Limits](/docs/reference/limits): query ranges and rate limits
- [llms.txt](/llms.txt): the machine-readable index of this site

---

# CLI reference

> Reference · https://maple.dev/docs/reference/cli

Every maple command, argument and flag, plus the local server's endpoints, environment variables and a troubleshooting guide.

The `maple` binary is one CLI with two backends: the local server it starts itself (`maple start`) and a hosted Maple workspace (`maple auth login`). Every query command runs against whichever backend is [resolved](#auth-and-configuration) for that invocation. Output is JSON by default, clean enough to pipe into `jq` or an agent.

New here? Start with the [Maple Local](/docs/local-mode) walkthrough, or [use the CLI with hosted Maple](#using-the-cli-with-hosted-maple). This page is the complete surface.

## Command index

| Command                                                                                                                                                           | What it does                                                 |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| [`maple start`](#maple-start) · [`stop`](#maple-stop) · [`reset`](#maple-reset)                                                                                   | Run the local server, stop it, or clear its live data        |
| [`maple checkpoint`](#maple-checkpoint) · [`restore`](#maple-restore)                                                                                             | Take a restore point of the local store, or roll back to one |
| [`maple archive …`](#maple-archive)                                                                                                                               | Export sealed days to Parquet and manage those archives      |
| [`maple schema …`](#maple-schema)                                                                                                                                 | Inspect and migrate the local store's schema                 |
| [`maple update`](#maple-update)                                                                                                                                   | Upgrade a script-installed binary in place                   |
| [`maple services`](#maple-services) · [`diagnose`](#maple-diagnose-service-name) · [`service-map`](#maple-service-map) · [`top-ops`](#maple-top-ops-service-name) | Services, their health, dependencies and hottest operations  |
| [`maple traces`](#maple-traces) · [`trace`](#maple-trace-trace-id) · [`slow-traces`](#maple-slow-traces)                                                          | Search spans, inspect one trace, find the slowest            |
| [`maple errors`](#maple-errors) · [`error`](#maple-error-fingerprint-hash)                                                                                        | Error groups by fingerprint, and one group in detail         |
| [`maple logs`](#maple-logs) · [`log-patterns`](#maple-log-patterns)                                                                                               | Search logs, or cluster them into templates                  |
| [`maple attributes keys`](#maple-attributes-keys) · [`values`](#maple-attributes-values-key)                                                                      | Discover attribute keys and their values                     |
| [`maple metrics`](#maple-metrics) · [`query`](#maple-query-sql)                                                                                                   | List metrics; run raw SQL (local only)                       |
| [`maple timeseries`](#maple-timeseries) · [`breakdown`](#maple-breakdown) · [`compare`](#maple-compare)                                                           | Bucketed metrics, top-N breakdowns, two windows side by side |
| [`maple auth …`](#maple-auth-login) · [`whoami`](#maple-whoami) · [`use`](#maple-use-localremoteauto)                                                             | Sign in to a workspace, see the resolved backend, pin one    |

## Global flags

Accepted by every command, in any position (`maple --local traces` and `maple traces --local` both work):

| Flag                     | Description                                                                       |
| ------------------------ | --------------------------------------------------------------------------------- |
| `--local`                | Force local mode (requires a running `maple start`)                               |
| `--remote`               | Force remote mode (requires `maple auth login`)                                   |
| `--debug`                | Print the compiled SQL and per-query timing to stderr; stdout stays clean JSON    |
| `--format <json\|table>` | Output format, default `json`. `table` renders a flat row set as an aligned table |

Most **query** commands also share a set of filter flags. Which ones apply is listed per command below; the shapes are always the same:

| Flag               | Alias | Default | Description                                                  |
| ------------------ | ----- | ------- | ------------------------------------------------------------ |
| `--since <range>`  |       | `6h`    | Relative time range: `30m`, `1h`, `6h`, `24h`, `7d`          |
| `--start <time>`   |       |         | Absolute start, `YYYY-MM-DD HH:mm:ss` UTC (use with `--end`) |
| `--end <time>`     |       |         | Absolute end, `YYYY-MM-DD HH:mm:ss` UTC                      |
| `--service <name>` | `-s`  |         | Filter by service name                                       |
| `--env <name>`     | `-e`  |         | Filter by deployment environment, e.g. `production`          |
| `--limit <n>`      | `-n`  | `20`    | Maximum number of results                                    |
| `--offset <n>`     |       | `0`     | Pagination offset                                            |

## Server commands

Local mode only. `maple start` is the long-lived process that owns the embedded ClickHouse connection; every other command talks to it over HTTP.

### `maple start`

Start the local ingest and query server.

| Flag                                         | Default          | Description                                                                                                                                                   |
| -------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--host <address>`                           | `127.0.0.1`      | Bind address. Anything but loopback exposes the UI, OTLP ingest and raw SQL to the network without authentication (see [Server endpoints](#server-endpoints)) |
| `--advertise-host <host>`                    | the bind address | Hostname printed for clients and used by the bundled UI                                                                                                       |
| `--port <int>`                               | `4318`           | Port for OTLP ingest, the query API and the bundled UI                                                                                                        |
| `--data-dir <path>`                          | `~/.maple/data`  | Embedded ClickHouse data directory                                                                                                                            |
| `--offline`                                  | `false`          | Serve the UI bundled in the binary instead of linking to `local.maple.dev`                                                                                    |
| `--background`, `-d`                         | `false`          | Run detached, logging to `~/.maple/maple.log`; stop with `maple stop`                                                                                         |
| `--reset`                                    | `false`          | Wipe live data before starting, keeping checkpoints. For an incompatible upgrade                                                                              |
| `--checkpoint-interval <dur>`                | `30m`            | How often to refresh the restore point while running (`45s`, `2h`, or `off`)                                                                                  |
| `--on-dirty-store <policy>`                  | `fail`           | What to do when the store was not cleanly closed: `fail`, `wipe` or `restore-checkpoint`                                                                      |
| `--chdb-config-file <path>`                  | generated        | Your own ClickHouse config for the embedded engine. Must keep backups enabled for checkpoints to work                                                         |
| `--minimum-raw-telemetry-retention-days <n>` |                  | Persist a retention floor for the raw tables (at least 90 days). Survives reset and restore                                                                   |

```bash
maple start                    # foreground, UI from local.maple.dev
maple start --offline          # foreground, bundled UI, no internet needed
maple start -d --port 4400     # detached on a custom port
maple start --host 0.0.0.0 --advertise-host maple.home.arpa --offline
```

The default recovery policy is `fail`, so an unclean shutdown never silently deletes telemetry. What each policy does, and how reset and restore protect the store, is on [Checkpoints and archives](/docs/local-mode/checkpoints-and-archives).

### `maple stop`

Stop a running server. Reads the PID file beside the data directory.

| Flag                | Default         | Description                          |
| ------------------- | --------------- | ------------------------------------ |
| `--data-dir <path>` | `~/.maple/data` | Data directory of the server to stop |

### `maple reset`

Delete live data so the next `maple start` bootstraps fresh. Checkpoints under `<data-dir>/backups` are kept. Refuses to run while a server owns the store.

| Flag                | Default         | Description                    |
| ------------------- | --------------- | ------------------------------ |
| `--data-dir <path>` | `~/.maple/data` | Store whose live data to clear |
| `--yes`, `-y`       | `false`         | Skip the confirmation prompt   |

### `maple checkpoint`

Create and validate a restorable checkpoint of the local store. Works out of the box against a running `maple start`.

| Flag                | Default         | Description                 |
| ------------------- | --------------- | --------------------------- |
| `--host <address>`  | `127.0.0.1`     | Host of the running server  |
| `--port <int>`      | `4318`          | Port of the running server  |
| `--data-dir <path>` | `~/.maple/data` | The server's data directory |

If the server was started with a custom host, port or data directory, pass the same values here.

### `maple restore`

Restore the local store from the current checkpoint. Refuses to run while a server owns the store. The existing store is moved beside the data directory as `<data-dir>.quarantine-<ids>`, never deleted; `maple schema gc` lists it and `--apply` removes it.

| Flag                     | Default                | Description                             |
| ------------------------ | ---------------------- | --------------------------------------- |
| `--data-dir <path>`      | `~/.maple/data`        | Store to restore                        |
| `--checkpoint-id <uuid>` | the current checkpoint | Restore one specific checkpoint instead |
| `--yes`, `-y`            | `false`                | Skip the confirmation prompt            |

```bash
maple restore --yes
maple restore --checkpoint-id 01234567-89ab-4cde-8fab-0123456789ab --yes
```

### `maple archive`

Local mode only. Export whole UTC days of the six raw telemetry tables from a checkpoint into Parquet files, and manage those exports. How archives work is explained on [Checkpoints and archives](/docs/local-mode/checkpoints-and-archives#archives).

| Subcommand                                                       | What it does                                                                                                                                                                               |
| ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `archive create <YYYY-MM-DD> <signal>`                           | Export one day of one signal. `--checkpoint-id` picks the checkpoint. Refuses days past retention, and refuses to replace an archived day with fewer rows unless you pass `--allow-shrink` |
| `archive list [--output summary\|paths\|json] [--signal <name>]` | List archived days. `paths` prints Parquet file paths for DuckDB and needs `--signal`                                                                                                      |
| `archive verify [--signal <name>]`                               | Re-check the SHA-256 of every archived file                                                                                                                                                |
| `archive expire <YYYY-MM-DD> --apply`                            | Delete one archived day across all six signals                                                                                                                                             |
| `archive retire-live <YYYY-MM-DD> --apply`                       | Delete a day from the live store once all six signals are archived and verified. The day must be at least `--sealing-lag-hours` (default `24`) past UTC midnight                           |
| `archive gc [--keep <n>] [--apply]`                              | Delete replaced copies of re-exported days, keeping the newest `n` per signal and day (default `1`). Without `--apply` it only prints the plan                                             |
| `archive reconcile`                                              | Finish an interrupted `create` or `gc` without exporting again                                                                                                                             |
| `archive rebuild <signal>`                                       | Rebuild a signal's `catalog.jsonl` from its manifests                                                                                                                                      |

Signals: `logs`, `traces`, `metrics_sum`, `metrics_gauge`, `metrics_histogram`, `metrics_exponential_histogram`.

Common flags: `--data-dir` (default `~/.maple/data`), `--archive-dir` (default `~/.maple/archive`) and `--scratch-root` (default `~/.maple/scratch`). `gc`, `expire` and `retire-live` change nothing unless you pass `--apply`. `reconcile` acts by default; pass `--dry-run` to print the plan without changing anything.

Export settings are fixed: one writer thread, 10,000-row row groups, and files of at most 500,000 rows or 256 MiB. Older releases had `archive calibrate`; `archive create` still accepts its `--config` flag but ignores it.

### `maple schema`

Inspect and migrate the local store's schema. Needed only when a release note says a store needs migrating; a normal upgrade opens the store as is.

| Subcommand                                  | What it does                                                                                                                                                                                   |
| ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `schema status`                             | Show the store's schema identity and migration journal state                                                                                                                                   |
| `schema plan`                               | Show the deterministic migration plan for this store                                                                                                                                           |
| `schema migrate [--dry-run] [--yes]`        | Migrate a populated store into a staged current-schema store. Needs a stopped server; keeps the original as a rollback point                                                                   |
| `schema abandon [--yes]`                    | Quarantine an unfinished staged target, preserving the active source (also when the source was left dirty)                                                                                     |
| `schema gc [--apply] [--release-preserved]` | List what restores, resets and migrations set aside (quarantined stores, migration sources, stale locks). `--apply` deletes it; `--release-preserved` also unpins the checkpoints a reset kept |

All four take `--data-dir <path>` (default `~/.maple/data`).

### `maple update`

Update a script-installed binary to the latest release: download, verify the checksum, install in place.

| Flag             | Description                                      |
| ---------------- | ------------------------------------------------ |
| `--check`        | Only report whether a newer version is available |
| `--tag <vX.Y.Z>` | Install a specific release instead of the latest |

Homebrew installs refuse `maple update` so the package manager stays in charge; run `brew upgrade maple` instead.

## Services

### `maple services`

List active services with throughput, error rate and P95 latency. Flags: `--since` / `--start` / `--end`, `--env`.

### `maple diagnose <service-name>`

Deep-dive one service: health, top errors, recent traces and logs.

- **`<service-name>`**: the service to diagnose
- Flags: `--since` / `--start` / `--end`, `--env`

### `maple service-map`

Service dependency edges with call counts, errors and latency. Flags: `--since` / `--start` / `--end`, `--service`, `--env`.

- `edges`: service-to-service calls, drawn when an instrumented callee continues the caller's trace
- `databases`: calls from client spans with `db.system.name` (or the legacy `db.system`), keyed by system and `db.namespace`
- `external`: HTTP, messaging and RPC targets that are not a traced service; listed only with `--service`, since that query is scoped to one caller

Databases and external calls are local-only. In remote mode the command prints `edges`, and `--service` keeps that service's outbound edges only; locally it keeps both directions.

### `maple top-ops <service-name>`

Top operations (span names) for a service, ranked by a metric.

- **`<service-name>`**: the service to inspect
- `--metric <count|avg_duration|p50_duration|p95_duration|p99_duration|error_rate|apdex>`: ranking metric, default `count`
- Flags: `--since` / `--start` / `--end`, `--limit`

## Traces

### `maple traces`

Search traces and spans.

| Flag                      | Description                                       |
| ------------------------- | ------------------------------------------------- |
| `--span-name <substr>`    | Filter by span name (substring, case-insensitive) |
| `--errors`                | Only traces with errors                           |
| `--min-duration-ms <int>` | Minimum duration in milliseconds                  |
| `--max-duration-ms <int>` | Maximum duration in milliseconds                  |
| `--http-method <method>`  | Filter by HTTP method (`GET`, `POST`, …)          |

Plus `--since` / `--start` / `--end`, `--service`, `--limit`, `--offset`.

```bash
maple traces --service api --min-duration-ms 500 --errors --since 1h
```

### `maple trace <trace-id>`

Inspect one trace: the full span tree plus correlated logs.

- **`<trace-id>`**: the trace to inspect

### `maple slow-traces`

The slowest traces with duration stats. Flags: `--since` / `--start` / `--end`, `--service`, `--env`, `--limit`.

## Errors

### `maple errors`

Error groups by fingerprint, with count, affected services and last seen. Flags: `--since` / `--start` / `--end`, `--service`, `--env`, `--limit`.

### `maple error <fingerprint-hash>`

One error group in detail: sample traces and a timeseries.

- **`<fingerprint-hash>`**: the fingerprint from `maple errors`
- Flags: `--since` / `--start` / `--end`, `--service`, `--limit`

## Logs

### `maple logs`

Search logs.

| Flag                 | Alias | Description                                          |
| -------------------- | ----- | ---------------------------------------------------- |
| `--severity <level>` |       | `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR` or `FATAL` |
| `--search <text>`    | `-q`  | Substring match on the body                          |
| `--trace-id <id>`    |       | Only logs from one trace                             |

Plus `--since` / `--start` / `--end`, `--service`, `--limit`, `--offset`.

### `maple log-patterns`

Cluster logs into templates to surface the noisiest patterns. Flags: `--since` / `--start` / `--end`, `--service`, `--severity`, `--search`/`-q`, `--limit`.

## Attributes

### `maple attributes keys`

Discover the attribute keys present in your data.

| Flag                                   | Default  | Description                               |
| -------------------------------------- | -------- | ----------------------------------------- |
| `--source <traces\|metrics\|services>` | `traces` | Where to look                             |
| `--scope <span\|resource>`             | `span`   | Span or resource attributes (traces only) |

Plus `--service`, `--since` / `--start` / `--end`, `--limit`.

### `maple attributes values <key>`

List the values seen for one attribute key.

- **`<key>`**: the attribute key
- Flags: same as `attributes keys`

## Metrics and raw SQL

### `maple metrics`

List available metrics. Flags: `--since` / `--start` / `--end`, `--service`, `--search`/`-q`, `--limit`.

### `maple query "<sql>"`

Run raw ClickHouse SQL against the local store, for anything the typed commands don't cover.

- **`<sql>`**: the query to run

```bash
maple query "SELECT ServiceName, count() FROM traces GROUP BY ServiceName ORDER BY 2 DESC"
```

> **Local only.** Raw SQL against the hosted warehouse would let one client read other organizations' data, so `maple query` returns a clear error in remote mode. Every other command works in both modes.

## Analytics

### `maple timeseries`

Time-bucketed trace metrics: count, latency quantiles, error rate and apdex per bucket.

| Flag                                                              | Default | Description                     |
| ----------------------------------------------------------------- | ------- | ------------------------------- |
| `--group-by <none\|service\|span_name\|status_code\|http_method>` | `none`  | Split the series by a dimension |
| `--span-name <substr>`                                            |         | Filter by span name             |
| `--errors`                                                        | `false` | Only errored spans              |
| `--bucket <seconds>`                                              | `60`    | Bucket size                     |

Plus `--since` / `--start` / `--end`, `--service`, `--env`.

### `maple breakdown`

Top-N breakdown of traces by a dimension.

| Flag                                                        | Default     | Description           |
| ----------------------------------------------------------- | ----------- | --------------------- |
| `--group-by <service\|span_name\|status_code\|http_method>` | `span_name` | Dimension to group by |
| `--span-name <substr>`                                      |             | Filter by span name   |
| `--errors`                                                  | `false`     | Only errored spans    |

Plus `--since` / `--start` / `--end`, `--service`, `--env`, `--limit`.

### `maple compare`

Compare service health between two windows, for regression detection. Give **either** `--around` **or** all four explicit bounds.

| Flag                                            | Description                                                                   |
| ----------------------------------------------- | ----------------------------------------------------------------------------- |
| `--around <ts>`                                 | Compare the 30 minutes before and after this UTC time (`YYYY-MM-DD HH:mm:ss`) |
| `--current-start <ts>` / `--current-end <ts>`   | The "current" window                                                          |
| `--previous-start <ts>` / `--previous-end <ts>` | The baseline window                                                           |
| `--env <name>`                                  | Filter by deployment environment                                              |

## Using the CLI with hosted Maple

The query commands also work against a hosted Maple organization. Sign in once:

```bash
maple auth login                                   # US organizations
maple auth login --api-url https://api.eu.maple.dev # EU organizations
maple use remote                                   # optional: stop auto-detecting
maple services --since 1h
```

`maple auth login` opens your browser, and you approve the CLI for one organization. The CLI then holds an [API key](/docs/reference/authentication#api-keys) with full access that expires after 90 days. It appears under **Settings → Keys** with the description "Created by maple auth login". Run `maple auth login` again when it expires, and `maple auth logout` to revoke it.

Every command except `maple query` works in remote mode. The server commands (`start`, `stop`, `reset`, `checkpoint`, `restore`, `archive`, `schema`) always act on the local store. Which region your organization is in is on [Regions](/docs/reference/regions).

## Auth and configuration

### `maple auth login`

Sign in to a hosted Maple workspace. Opens your browser to approve the CLI; the credential is stored in the macOS keychain where available, otherwise in `~/.maple/config.json` (mode `0600`). `maple login` is a shorthand for the same command.

| Flag              | Description                                                                                          |
| ----------------- | ---------------------------------------------------------------------------------------------------- |
| `--api-url <url>` | Maple API base URL, default `https://api.maple.dev`                                                  |
| `--with-token`    | Read an existing API token from stdin instead of opening a browser, so it stays out of shell history |

```bash
maple auth login
echo "$MAPLE_TOKEN" | maple auth login --with-token
```

### `maple auth status`

Show and validate the active login: API URL, user, workspace and where the credential is stored.

### `maple auth logout`

Revoke the credential with the API and remove it locally. `maple logout` is the shorthand. A token supplied through `MAPLE_API_TOKEN` cannot be removed this way; unset the variable instead.

### `maple whoami`

Show the resolved mode (local or remote) and the target it would talk to, plus any pinned default.

### `maple use <local|remote|auto>`

Pin the default backend so commands stop auto-detecting, or `auto` to clear the pin.

**Mode resolution**, per command, in priority order:

1. An explicit `--local` or `--remote` flag.
2. The default pinned with `maple use`.
3. Auto-detect: a stored credential implies remote; otherwise a quick `GET /health` probe of the local server implies local. If neither is available the CLI prints what to do next.

## Server endpoints

`maple start` binds `127.0.0.1` by default. `--host` or `MAPLE_LOCAL_BIND_HOST` can select another address. Every route below is then reachable from the network.

| Method    | Path                                                                                                          | Auth                 | Purpose                                                                                 |
| --------- | ------------------------------------------------------------------------------------------------------------- | -------------------- | --------------------------------------------------------------------------------------- |
| `GET`     | `/health`                                                                                                     | None                 | Liveness probe, returns `OK`. Used by mode auto-detect                                  |
| `POST`    | `/v1/traces`                                                                                                  | None                 | OTLP traces ingest, responds `{ "accepted": <rowCount> }`                               |
| `POST`    | `/v1/logs`                                                                                                    | None                 | OTLP logs ingest                                                                        |
| `POST`    | `/v1/metrics`                                                                                                 | None                 | OTLP metrics ingest                                                                     |
| `POST`    | `/local/query`                                                                                                | None                 | Run SQL: `{ "sql": "..." }` in, a bare JSON array of rows out                           |
| `GET`     | any other path                                                                                                | None                 | The bundled UI, with `--offline` only                                                   |
| `OPTIONS` | `*`                                                                                                           | None                 | CORS and private-network preflight, for the configured hosted UI origin only            |
| `POST`    | `/local/checkpoint/backup`                                                                                    | Maintenance token    | Take a checkpoint. `maple checkpoint` calls this                                        |
| `POST`    | `/local/retention/retire`                                                                                     | Maintenance token    | Delete one UTC day from the live tables. `maple archive retire-live --apply` calls this |
| `GET`     | `/local/eventing/health`, `/projections`, `/consumers`, `/outbox`                                             | Maintenance token    | Read local event projection and consumer state                                          |
| `POST`    | `/local/eventing/projections`, `/consumers`, `/consumers/disable`, `/consumers/accept-gap`, `/outbox/abandon` | Maintenance token    | Administer local event projections and consumers                                        |
| `POST`    | `/local/eventing/claims`, `/local/eventing/acks`                                                              | Event consumer token | Claim and acknowledge local events                                                      |

**None** means no credential is checked. On a non-loopback bind, anyone who can reach the port can read all local telemetry through `/local/query` and write to it through OTLP.

The maintenance token is sent in `x-maple-maintenance-token` and the event consumer token in `x-maple-event-consumer-token`. `maple start` creates them on first run as `<data-dir>.maintenance-token` and `<data-dir>.event-consumer-token` (mode `0600`). Anyone who can read those files can call the token routes, including `/local/retention/retire`, which deletes data. The event routes are described in the [local event consumers design doc](https://github.com/MapleTechLabs/maple/blob/main/docs/local-event-consumers.md).

OTLP bodies may be protobuf (the default) or JSON, optionally gzip-encoded. The `/local/query` handler owns the output format: it strips any trailing `FORMAT <ident>`, appends `FORMAT JSONEachRow`, and wraps the rows into a JSON array, so clients POST their compiled SQL verbatim.

## Environment variables

**Runtime** (CLI and server):

| Variable                     | Default                   | Purpose                                                                                                                                                        |
| ---------------------------- | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `MAPLE_LOCAL_BIND_HOST`      | `127.0.0.1`               | Server bind host and the CLI's default local target; wildcards map to loopback                                                                                 |
| `MAPLE_LOCAL_ADVERTISE_HOST` | the bind host             | Host printed for clients and used by the bundled UI                                                                                                            |
| `MAPLE_LOCAL_URL`            | bind host + `4318`        | Explicit base URL for CLI queries and mode detection                                                                                                           |
| `MAPLE_LOCAL_UI_URL`         | `https://local.maple.dev` | The hosted UI origin `maple start` links to and allows through CORS                                                                                            |
| `MAPLE_LIBCHDB`              | _(auto)_                  | Explicit path to `libchdb`. Otherwise resolved beside the binary, then `~/.maple/bin/libchdb.{so,dylib}`                                                       |
| `MAPLE_API_URL`              | `https://api.maple.dev`   | Remote API base URL                                                                                                                                            |
| `MAPLE_API_TOKEN`            |                           | Remote bearer token; overrides the stored credential                                                                                                           |
| `MAPLE_ORG_ID`               |                           | Remote organization override                                                                                                                                   |
| `MAPLE_DEBUG`                |                           | `1` enables `--debug`                                                                                                                                          |
| `MAPLE_FORMAT`               | `json`                    | `json` or `table`, same as `--format`                                                                                                                          |
| `MAPLE_NO_UPDATE_CHECK`      |                           | Any non-empty value disables the startup update check (the Homebrew wrapper sets it). The check runs at most once per 24 hours, only when stderr is a terminal |

**CLI telemetry.** The CLI sends its own traces, logs and metrics (service `maple-cli`) to Maple by default. See [What connects to the internet](/docs/local-mode#what-connects-to-the-internet) for what is recorded.

| Variable            | Default                                | Purpose                                                                                                                                                                                                                                                             |
| ------------------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `MAPLE_TELEMETRY`   | on                                     | `off` disables the CLI's own telemetry entirely. Any other value leaves it on                                                                                                                                                                                       |
| `MAPLE_INGEST_KEY`  | a key built into the binary            | Ingest key used for the CLI's own telemetry                                                                                                                                                                                                                         |
| `MAPLE_ENDPOINT`    | `https://ingest.maple.dev`             | Where the CLI's own telemetry is sent. Takes precedence over `OTEL_EXPORTER_OTLP_ENDPOINT`, which is also read. If you export `OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318` for your app in the same shell, the CLI's telemetry goes to your local server too |
| `MAPLE_ENVIRONMENT` | `ci` when `CI` is set, otherwise `cli` | `deployment.environment` reported on the CLI's own telemetry                                                                                                                                                                                                        |

**Install script** (`scripts/install.sh`):

| Variable              | Default        | Purpose                                                  |
| --------------------- | -------------- | -------------------------------------------------------- |
| `MAPLE_VERSION`       | `latest`       | Release tag to install                                   |
| `MAPLE_INSTALL_DIR`   | `~/.maple/bin` | Where the two-file bundle lands                          |
| `MAPLE_BIN_DIR`       | _(auto)_       | Where `maple` is symlinked onto `PATH`                   |
| `MAPLE_SKIP_CHECKSUM` | `0`            | `1` skips SHA-256 verification (air-gapped mirrors only) |

`~/.maple/config.json` stores `apiUrl`, `orgId`, `defaultMode` and, when the keychain is unavailable, the token. Environment variables take precedence over stored values.

## Troubleshooting

**`libchdb` not found.** The binary loads `libchdb` from beside its own path, then falls back to `~/.maple/bin`. Homebrew keeps `maple` and `libchdb` together; the install script keeps them in `~/.maple/bin`. If you moved files by hand, keep `libchdb.so` or `.dylib` beside `maple`, or set `MAPLE_LIBCHDB` to its full path. Running from source has no sibling library, so set `MAPLE_LIBCHDB` or drop one into `~/.maple/bin`.

**Homebrew installed, but `maple` still runs the old binary.** A script-installer symlink is earlier on `PATH`. Confirm with `command -v maple`, then remove the old symlink or run `curl -fsSL https://maple.dev/cli/uninstall | sh` before reinstalling with Homebrew.

**`maple is already running (PID …)`.** Another server owns this data directory. Stop it with `maple stop`, or start a second instance on its own port and store: `maple start --port 4400 --data-dir ~/.maple/data-2`.

**Incompatible store after an upgrade.** A new binary that refuses an older store (`the local store … is incompatible`) needs the live data cleared: `maple reset --yes`, or `maple start --reset` in one step. Both keep checkpoints. If the release notes call for a migration instead, use [`maple schema`](#maple-schema).

**Store was not cleanly closed.** The default `--on-dirty-store fail` stops rather than guess. Restart with `--on-dirty-store restore-checkpoint` to roll back to the last good checkpoint, or `wipe` to discard live data. Neither touches checkpoints.

**Browser asks to "access devices on your local network", or CORS errors.** The default dashboard at `local.maple.dev` is a public origin reaching your loopback server, which trips Chrome's private network gate. Run `maple start --offline` to serve the dashboard same-origin. For a wildcard LAN bind, also set `--advertise-host` to the hostname the browser will use; other hosts and origins are rejected.

**Authentication proxy blocks the bundled UI.** The UI works behind TLS with browser-managed authentication such as a session cookie or HTTP auth. It does not inject a bearer token or copy an entry-page query parameter into its `/local/query` and OTLP requests.

**No data appearing.** Confirm the exporter points at the advertised host and port and the server is up (`maple whoami`, or `curl <host>:4318/health`). Widen the time range; the default is `--since 6h`. A successful ingest responds `{ "accepted": <n> }`.

**`No Maple backend found`.** Neither backend could be resolved. Start local mode (`maple start`), sign in to a workspace (`maple auth login`), or force one with `--local` / `--remote`.

---

# Authentication

> Reference · https://maple.dev/docs/reference/authentication

The credentials Maple accepts: public and private ingest keys for sending telemetry, API keys for the REST API and MCP server, OAuth for MCP clients, and which endpoint takes which.

Maple uses two kinds of credential. **Ingest keys** send telemetry. **API keys** read and change your organization through the REST API and the MCP server. An ingest key cannot read anything, and an API key cannot send telemetry.

| Credential         | Prefix      | Where you get it                         | What it can do                                 | Safe to ship in client code? |
| ------------------ | ----------- | ---------------------------------------- | ---------------------------------------------- | ---------------------------- |
| Public ingest key  | `maple_pk_` | **Settings → Ingestion**                 | Send telemetry                                 | Yes                          |
| Private ingest key | `maple_sk_` | **Settings → Ingestion**                 | Send telemetry                                 | No                           |
| API key            | `maple_ak_` | **Settings → Keys**                  | REST API and MCP server, limited by its scopes | No                           |
| MCP key            | `maple_ak_` | **Settings → MCP**                       | MCP server only                                | No                           |
| OAuth token        |             | Issued to an MCP client when you sign in | MCP server only                                | No                           |

Every credential belongs to one organization in one [region](/docs/reference/regions). A credential from the US region is rejected by the EU region, and the other way round.

## Which endpoint accepts what

| Endpoint                                | Accepts                                                                               |
| --------------------------------------- | ------------------------------------------------------------------------------------- |
| Ingest, `https://ingest.maple.dev/v1/*` | Public or private ingest key, as `Authorization: Bearer …` or `x-maple-ingest-key: …` |
| REST API, `https://api.maple.dev/v2/*`  | API key, or a dashboard session token, as `Authorization: Bearer …`                   |
| MCP server, `https://api.maple.dev/mcp` | API key, MCP key or OAuth token, as `Authorization: Bearer …`                         |

Use the EU hosts (`ingest.eu.maple.dev`, `api.eu.maple.dev`) for an EU organization.

## Ingest keys

Each organization has exactly one public key and one private key. Both are shown under **Settings → Ingestion**, and both have the same permission: they can send traces, logs, metrics, product events and session replays to your organization. Neither can read data.

- Use the **public key** (`maple_pk_…`, labeled **Client**) in browsers, mobile apps and anything else you ship to end users. It will be visible to anyone who inspects your app, which is expected. The worst someone can do with it is send data to your organization.
- Use the **private key** (`maple_sk_…`, labeled **Server**) on servers, in the OpenTelemetry Collector and in CI. Keep it out of client bundles, so you can rotate the public key without touching your backend.

The public key also works on servers. The [maple-onboard](https://github.com/MapleTechLabs/maple/tree/main/skills/maple-onboard) skill uses it everywhere so one key covers a repository; switch servers to the private key when you want to rotate the two separately.

Send the key on every request, either as `Authorization: Bearer maple_pk_…` or as `x-maple-ingest-key: maple_pk_…`. The `Bearer` prefix is case-insensitive. The [Ingest API](/docs/reference/ingest#authentication) page has the details.

To rotate a key, click **Regenerate** next to it in **Settings → Ingestion**, or call `POST /v2/ingest_keys/public/roll` or `POST /v2/ingest_keys/private/roll`. The old key stops working immediately, so deploy the new one first where you can.

The literal key `MAPLE_TEST` is accepted and returns `200`, but the data is discarded. Use it in CI or example code.

## API keys

Create API keys under **Settings → Keys**, or with `POST /v2/api_keys`. When you create one you choose:

| Field                        | Options                                               |
| ---------------------------- | ----------------------------------------------------- |
| **Name** and **Description** | Free text, to tell keys apart                         |
| **Expiration**               | Never, 7 days, 30 days, 90 days or 1 year             |
| **Access**                   | **Full access**, or **Restricted** to a set of scopes |

The key is shown once, when it is created. Store it then; Maple keeps only a hash. You can revoke a key or roll it (issue a new secret and revoke the old one) from the same page or with `POST /v2/api_keys/{id}/roll`.

A restricted key carries scopes of the form `<family>:read` or `<family>:write`, where the family is the first path segment under `/v2` (`dashboards`, `alerts`, `traces`, …). `write` implies `read`. The full list of families is in the [API reference](/docs/reference/api#resources). A request outside a key's scopes fails with `403` and `code: "insufficient_scope"`.

`maple auth login` also creates an API key: a full-access key that expires after 90 days, described as "Created by maple auth login". See [Using the CLI with hosted Maple](/docs/reference/cli#using-the-cli-with-hosted-maple).

API requests are limited to 600 per 60 seconds per key. See [Limits](/docs/reference/limits).

## MCP keys and OAuth

An MCP client can authenticate in three ways:

- **An API key** from **Settings → Keys**, sent as `Authorization: Bearer maple_ak_…`.
- **An MCP key** from **Settings → MCP**. It has full access to the MCP tools and is rejected by the REST API.
- **OAuth**, for clients that support the MCP authorization flow. The client discovers Maple's authorization server from the endpoint URL, registers itself, and opens a browser sign-in. You pick the organization on the consent screen. Access tokens last 1 hour and are refreshed automatically. The client has to sign in again after 30 days without a refresh, and 90 days after the first sign-in at the latest. OAuth tokens only work on the MCP server.

The [MCP server](/docs/reference/mcp#connecting-a-client) page has client configuration examples.

## Dashboard session tokens and `x-maple-org-id`

The dashboard calls the REST API with the signed-in user's session token. You can do the same from a script, but API keys are the supported way to automate.

A user who belongs to several organizations can choose which one a `/v2` request acts on with the `x-maple-org-id` header. Maple checks that the user is a member of that organization and applies the user's role there. If the check fails, the request is rejected with `403`; it is never served under a different organization. With an API key, the header may only name the key's own organization. Any other value is rejected.

---

# Regions

> Reference · https://maple.dev/docs/reference/regions

Maple runs in a US region and an EU region. The hosts for the app, ingest, API and MCP server in each, how to find your organization's region, and how to point SDKs and tools at it.

Maple runs as two separate instances: one in the United States and one in the European Union (Frankfurt). Each has its own dashboard, ingest endpoint, API, database and storage. An organization's telemetry is stored and processed only in its region. Nothing is copied between them.

## Hosts

|               | United States                   | European Union (Frankfurt)         |
| ------------- | ------------------------------- | ---------------------------------- |
| Dashboard     | `https://app.maple.dev`         | `https://app.eu.maple.dev`         |
| Ingest (OTLP) | `https://ingest.maple.dev`      | `https://ingest.eu.maple.dev`      |
| REST API      | `https://api.maple.dev/v2`      | `https://api.eu.maple.dev/v2`      |
| MCP server    | `https://api.maple.dev/mcp`     | `https://api.eu.maple.dev/mcp`     |
| API docs      | `https://api.maple.dev/v2/docs` | `https://api.eu.maple.dev/v2/docs` |

Examples on this site use the US hosts. For an EU organization, replace `maple.dev` with `eu.maple.dev` in the host and change nothing else.

Keys do not cross regions. An ingest key or API key created in one region is rejected by the other, so a mismatch shows up as `401` rather than as data landing in the wrong place.

## Find your organization's region

- **Settings → General** shows a **Data region** row: "United States" or "European Union (Frankfurt)".
- The organization switcher in the sidebar shows `· US` or `· EU` after the organization name.
- The dashboard host you use is the region: `app.maple.dev` is US, `app.eu.maple.dev` is EU.

## Choosing a region

A new organization chooses its region in the first onboarding step. The region cannot be changed later. Organizations created before regions existed are in the US region.

What the EU region covers is described on [EU hosting](/eu).

## Point your tools at a region

| Tool                                                                 | US (default)                                           | EU                                                        |
| -------------------------------------------------------------------- | ------------------------------------------------------ | --------------------------------------------------------- |
| Any OpenTelemetry SDK or Collector                                   | `OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev` | `OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.eu.maple.dev` |
| [`@maple-dev/effect-sdk`](/docs/sdks/effect) (server and Cloudflare) | nothing to set                                         | `MAPLE_REGION=eu`, or `region: "eu"` in the layer config  |
| [`@maple-dev/effect-sdk`](/docs/sdks/effect-client) (browser)        | nothing to set                                         | `region: "eu"`                                            |
| [`@maple-dev/browser`](/docs/session-replay/browser-sdk)             | nothing to set                                         | `region: "eu"`                                            |
| [`maple` CLI](/docs/reference/cli#using-the-cli-with-hosted-maple)   | `maple auth login`                                     | `maple auth login --api-url https://api.eu.maple.dev`     |
| [MCP clients](/docs/reference/mcp)                                   | `https://api.maple.dev/mcp`                            | `https://api.eu.maple.dev/mcp`                            |

In the Maple SDKs an explicit endpoint always wins over `region`. For `@maple-dev/effect-sdk` on a server that means `MAPLE_ENDPOINT`, then `OTEL_EXPORTER_OTLP_ENDPOINT`, then `MAPLE_REGION`. An unrecognized region value logs a warning and falls back to US.

## Related

- [Authentication](/docs/reference/authentication): which key goes where
- [Ingest API](/docs/reference/ingest): paths, limits and status codes
- [Retention](/docs/reference/retention): how long each region keeps your data

---

# Ingest API

> Reference · https://maple.dev/docs/reference/ingest

The OTLP ingest endpoint: paths, authentication, content types, compression, request limits, status codes, and how to retry.

Every signal reaches Maple through one OTLP/HTTP gateway. Any OpenTelemetry SDK or Collector that can export OTLP over HTTP works without a Maple-specific exporter.

|              |                                                                                     |
| ------------ | ----------------------------------------------------------------------------------- |
| Base URL     | `https://ingest.maple.dev`, or your [region's](/docs/reference/regions) ingest host |
| Protocol     | OTLP over HTTP, `POST`                                                              |
| Auth         | `Authorization: Bearer maple_pk_…` (or `maple_sk_…`)                                |
| Encodings    | Protobuf (recommended) or JSON, optionally gzip                                     |
| Max body     | 20 MiB per request, measured on the compressed body                                 |
| Request time | 30 seconds per request                                                              |

## Endpoints

| Path                        | Payload                                                                            |
| --------------------------- | ---------------------------------------------------------------------------------- |
| `POST /v1/traces`           | OTLP `ExportTraceServiceRequest`                                                   |
| `POST /v1/logs`             | OTLP `ExportLogsServiceRequest`                                                    |
| `POST /v1/metrics`          | OTLP `ExportMetricsServiceRequest`                                                 |
| `POST /v1/events`           | [Product events](/docs/product-events/api)                                         |
| `POST /v1/sessionReplays/*` | Session replay chunks, sent by the [browser SDK](/docs/session-replay/browser-sdk) |
| `POST /v1/sessionEvents`    | Session timeline events, sent by the browser SDK                                   |

Use the ingest host of your organization's [region](/docs/reference/regions). A key from one region is rejected by the other. A standard exporter appends the signal path itself:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer maple_pk_…"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
```

## Authentication

Send your ingest key on every request, either as a Bearer token or in the `x-maple-ingest-key` header. The `Bearer` prefix is case-insensitive.

```http
Authorization: Bearer maple_pk_…
x-maple-ingest-key: maple_pk_…
```

Ingest keys can only send data, and each belongs to one organization. Use the **public** key (`maple_pk_…`) in browsers and mobile apps, where it ships to end users, and the **private** key (`maple_sk_…`) on servers. Find both under **Settings → Ingestion** in the dashboard. [Authentication](/docs/reference/authentication#ingest-keys) explains the difference.

The literal key `MAPLE_TEST` is accepted and returns `200`, but the data is discarded. Use it in CI or example code where you want the exporter to run without sending anything.

## Content types and compression

| `Content-Type`                                   | Read as       |
| ------------------------------------------------ | ------------- |
| `application/x-protobuf`, `application/protobuf` | OTLP protobuf |
| `application/octet-stream`                       | OTLP protobuf |
| anything containing `json` (`application/json`)  | OTLP JSON     |
| header missing                                   | OTLP protobuf |
| anything else                                    | `415`         |

For compression, send `Content-Encoding: gzip`. Omit the header (or send `identity`) for an uncompressed body. **gzip is the only compression supported.** `zstd`, `deflate` and `br` are rejected with `415`, so set your exporter's compression to `gzip` or `none`.

## Browsers

The gateway answers CORS preflights from any origin, so a browser can export directly. Allowed request headers are `Authorization`, `Content-Type`, `Content-Encoding`, `x-maple-ingest-key` and the `x-maple-*` headers the browser SDK sends. No response headers are exposed to scripts, so browser code cannot read `Retry-After`; back off on a fixed schedule instead.

## Status codes

Errors carry a JSON body with the same envelope as the [Maple API](/docs/reference/api#errors):

```json
{
	"error": {
		"_tag": "@maple/ingest/OrgQueueThrottled",
		"type": "rate_limit_error",
		"code": "ingest_queue_throttled",
		"title": "Ingest queue full for this org",
		"message": "This org's ingest queue is at capacity. No data was written; resend this batch after the suggested delay.",
		"retryable": true,
		"recovery": "retry",
		"retry_after_seconds": 1
	}
}
```

| Field                 | Meaning                                                                                                                      |
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `_tag`                | The exact failure, stable across releases. Branch on this                                                                    |
| `type`                | The status family: `invalid_request_error`, `authentication_error`, `payment_error`, `rate_limit_error` or `api_error` (5xx) |
| `code`                | The short code in the table below                                                                                            |
| `title`, `message`    | Human-readable text                                                                                                          |
| `retryable`           | Whether resending the same batch can succeed                                                                                 |
| `recovery`            | `fix_request`, `reauthenticate`, `retry` or `contact_support`                                                                |
| `retry_after_seconds` | Present when a retry makes sense. The same value is in the `Retry-After` header                                              |

Checks run roughly in the order below, so a request fails on the first one it trips.

| Status | `code`                          | `_tag`                                 | Cause                                                                                                     | Retry?                           |
| ------ | ------------------------------- | -------------------------------------- | --------------------------------------------------------------------------------------------------------- | -------------------------------- |
| `200`  |                                 |                                        | Accepted and durably queued                                                                               |                                  |
| `401`  | `ingest_unauthorized`           | `@maple/ingest/Unauthorized`           | Missing, malformed or unknown ingest key, or a key sent to the other region's ingest                      | No. Fix the key                  |
| `429`  | `ingest_rate_limited`           | `@maple/ingest/RateLimited`            | More than 1,000 requests in flight for your organization                                                  | Yes, after 1 s                   |
| `413`  | `ingest_payload_too_large`      | `@maple/ingest/PayloadTooLarge`        | Body over 20 MiB                                                                                          | No. Send smaller batches         |
| `415`  | `ingest_unsupported_media_type` | `@maple/ingest/UnsupportedMediaType`   | Unknown `Content-Type` or `Content-Encoding`                                                              | No. Fix the exporter config      |
| `400`  | `ingest_bad_request`            | `@maple/ingest/BadRequest`             | Invalid gzip, a body that is not valid OTLP, or a malformed product event, session event or replay header | No                               |
| `400`  | `ingest_replay_body_not_gzip`   | `@maple/ingest/ReplayBodyNotGzip`      | A session replay chunk that is not a gzip stream                                                          | No                               |
| `402`  | `ingest_plan_limit_reached`     | `@maple/ingest/PlanLimitReached`       | No active subscription, or the plan's limit is reached                                                    | No. Check **Settings → Billing & usage** |
| `429`  | `ingest_queue_throttled`        | `@maple/ingest/OrgQueueThrottled`      | Your organization's ingest queue is full. Nothing was written                                             | Yes, after 1 s                   |
| `429`  | `ingest_export_lane_full`       | `@maple/ingest/ExportLaneBackpressure` | The write path for your organization is backed up. Nothing was written                                    | Yes, after 2 s                   |
| `503`  | `ingest_unavailable`            | `@maple/ingest/ServiceUnavailable`     | Key lookup or another dependency is temporarily unavailable                                               | Yes, after 5 s                   |
| `503`  | `ingest_queue_unavailable`      | `@maple/ingest/QueueUnavailable`       | The batch could not be written to the durable queue. Nothing was written                                  | Yes, after 5 s                   |
| `503`  | `ingest_collector_unavailable`  | `@maple/ingest/CollectorUnavailable`   | An upstream collector failed or its response could not be read                                            | Yes, after 5 s                   |
| `503`  | `ingest_request_timeout`        | `@maple/ingest/RequestTimeout`         | The request took longer than 30 seconds                                                                   | Yes, after 5 s                   |
| `503`  | `ingest_encode_failed`          | `@maple/ingest/PayloadEncodeFailed`    | The batch could not be encoded for storage. Resending the same batch fails the same way                   | No. Contact support              |
| `500`  | `ingest_internal_error`         | `@maple/ingest/InternalError`          | Unexpected gateway error                                                                                  | No. Contact support              |

OpenTelemetry SDKs and the Collector already retry `429` and `503` with backoff, and drop on other `4xx` codes. That is the right behavior for every code above except `ingest_encode_failed`, which an exporter retries even though it cannot succeed.

## Batching

A request is limited by its compressed size (20 MiB), not its span or record count. The default batch sizes of the OpenTelemetry SDKs (512 spans or log records per export) and of the Collector's `batch` processor (8,192 items) stay far below that. If you raise them and a batch reaches the limit, the gateway answers `413` and the exporter drops the whole batch.

## Related

- [OpenTelemetry conventions](/docs/concepts/otel-conventions): the attributes Maple reads from what you send
- [Instrumentation guides](/docs/instrumentation): per-language exporter setup
- [Limits](/docs/reference/limits): query ranges, SQL and API rate limits
- [Regions](/docs/reference/regions): US and EU hosts

---

# SQL reference

> Reference · https://maple.dev/docs/reference/sql

Query your telemetry with ClickHouse SQL: the tables and columns you can read, the required macros, result shapes for each chart type, and limits.

SQL widgets on dashboards, raw-SQL alert rules, and the MCP `run_sql` tool all run ClickHouse SQL against the same tables Maple's own pages read. This page lists what you can query and the rules a query has to follow.

## A first query

```sql
SELECT toStartOfInterval(Timestamp, INTERVAL $__interval_s SECOND) AS bucket,
       count() AS logs
FROM logs
WHERE $__orgFilter AND $__timeFilter(Timestamp)
GROUP BY bucket
ORDER BY bucket
```

Every query **must** contain `$__orgFilter`. It expands to your organization's filter, and a query without it is rejected before it runs. Access is also enforced by the credentials the query runs with, so the macro is a correctness check rather than your only protection.

## Macros

| Macro                | Expands to                                                    |
| -------------------- | ------------------------------------------------------------- |
| `$__orgFilter`       | Your organization filter. Required.                           |
| `$__timeFilter(col)` | `col >= <start> AND col <= <end>` for the selected time range |
| `$__timeGroup(col)`  | `toStartOfInterval(col, INTERVAL <bucket> SECOND)`            |
| `$__startTime`       | The range start, as a `DateTime`                              |
| `$__endTime`         | The range end, as a `DateTime`                                |
| `$__interval_s`      | The bucket width in seconds, an integer of at least 1         |

The argument to `$__timeFilter` and `$__timeGroup` must be a plain column name. Any other `$__name` is an error. When a widget has no explicit granularity, buckets target about 30 points with a 5-minute minimum.

On dashboards, variables are written `$name` or `${name}` and are substituted as quoted string literals. A variable set to **All** becomes a comma-separated list, so use it with `IN`:

```sql
WHERE $__orgFilter AND $__timeFilter(Timestamp) AND ServiceName IN ($service)
```

## Tables

The main tables are below. Run `describe_warehouse_tables` from the [MCP server](/docs/reference/mcp) for the full list with every column and its type.

| Table                           | One row per                                 | Time column |
| ------------------------------- | ------------------------------------------- | ----------- |
| `traces`                        | Span                                        | `Timestamp` |
| `service_overview_spans`        | Entry-point span (server, consumer or root) | `Timestamp` |
| `logs`                          | Log record                                  | `Timestamp` |
| `error_events`                  | Exception occurrence                        | `Timestamp` |
| `metrics_sum`                   | Counter data point                          | `TimeUnix`  |
| `metrics_gauge`                 | Gauge data point                            | `TimeUnix`  |
| `metrics_histogram`             | Histogram data point                        | `TimeUnix`  |
| `metrics_exponential_histogram` | Exponential histogram data point            | `TimeUnix`  |
| `session_replays`               | Browser session                             | `StartTime` |
| `session_events`                | Session timeline event                      | `Timestamp` |
| `product_events`                | Product event                               | `Timestamp` |
| `service_usage`                 | Service and hour                            | `Hour`      |
| `attribute_keys_hourly`         | Attribute key and hour                      | `Hour`      |

### `traces`

| Column                                              | Type                                  | Notes                                                             |
| --------------------------------------------------- | ------------------------------------- | ----------------------------------------------------------------- |
| `Timestamp`                                         | `DateTime64(9)`                       | Span start                                                        |
| `TraceId`, `SpanId`, `ParentSpanId`                 | `String`                              |                                                                   |
| `ServiceName`, `SpanName`                           | `LowCardinality(String)`              |                                                                   |
| `SpanKind`                                          | `LowCardinality(String)`              | `Internal`, `Server`, `Client`, `Producer`, `Consumer`            |
| `Duration`                                          | `UInt64`                              | **Nanoseconds.** Divide by `1e6` for milliseconds.                |
| `StatusCode`                                        | `LowCardinality(String)`              | `Ok`, `Error`, `Unset` (Title Case)                               |
| `StatusMessage`                                     | `String`                              |                                                                   |
| `SpanAttributes`, `ResourceAttributes`              | `Map(LowCardinality(String), String)` | Read with `SpanAttributes['http.route']`                          |
| `EventsTimestamp`, `EventsName`, `EventsAttributes` | Arrays                                | Span events, index-aligned                                        |
| `SampleRate`                                        | `Float64`                             | 1.0 when unsampled. Multiply counts by it to estimate throughput. |
| `IsEntryPoint`                                      | `UInt8`                               | 1 for server, consumer and root spans                             |

For per-service request rate, error rate and latency, query `service_overview_spans` instead. It has the same core columns, holds only entry-point spans, and is much smaller, but it has no attribute maps.

### `logs`

| Column                                                   | Type                                  | Notes                                                                 |
| -------------------------------------------------------- | ------------------------------------- | --------------------------------------------------------------------- |
| `Timestamp`                                              | `DateTime64(9)`                       |                                                                       |
| `TimestampTime`                                          | `DateTime`                            | Second precision                                                      |
| `ServiceName`                                            | `LowCardinality(String)`              |                                                                       |
| `SeverityText`                                           | `LowCardinality(String)`              | Casing varies by SDK                                                  |
| `SeverityNumber`                                         | `UInt8`                               | 1-4 trace, 5-8 debug, 9-12 info, 13-16 warn, 17-20 error, 21-24 fatal |
| `Body`                                                   | `String`                              |                                                                       |
| `TraceId`, `SpanId`                                      | `String`                              |                                                                       |
| `LogAttributes`, `ResourceAttributes`, `ScopeAttributes` | `Map(LowCardinality(String), String)` |                                                                       |

Filter severity on `SeverityNumber` rather than `SeverityText`: some SDKs send `Error` and others `ERROR`. `SeverityNumber BETWEEN 17 AND 20` matches every error.

### Metrics tables

All four metric tables share `ServiceName`, `MetricName`, `MetricDescription`, `MetricUnit`, `Attributes` (a map of data-point attributes), `ResourceAttributes`, `StartTimeUnix` and `TimeUnix`. Then:

- `metrics_sum` and `metrics_gauge` carry `Value Float64`. `metrics_sum` adds `IsMonotonic` and `AggregationTemporality` (1 is delta, 2 is cumulative). Delta rows already hold the per-interval increase, so `sum(Value)` per bucket is exact; cumulative rows need a rate.
- `metrics_histogram` carries `Count`, `Sum`, `Min`, `Max`, `BucketCounts` and `ExplicitBounds`.
- `metrics_exponential_histogram` carries `Count`, `Sum`, `Scale`, `ZeroCount` and the positive and negative bucket arrays.

### Things that trip people up

- **Missing map keys read as `''`, not `NULL`.** Test with `mapContains(SpanAttributes, 'key')` or `SpanAttributes['key'] != ''`.
- **Status and span kind are Title Case.** `StatusCode = 'Error'` matches; `'ERROR'` does not.
- **Hourly rollups use `Hour`, not `Timestamp`.** Snap your range with `toStartOfHour`, or a window shorter than an hour returns nothing.
- **Rollup columns are aggregate states.** Read `SimpleAggregateFunction` columns with `sum(...)` and `AggregateFunction` columns with the matching `…Merge(...)`.
- **Filter on the sort key first.** `traces` is sorted by service and span name, so adding `ServiceName = '…'` makes a query dramatically cheaper.

## Result shapes for charts

| Display                     | Return                                                              |
| --------------------------- | ------------------------------------------------------------------- |
| Line, area, bar             | A time column (name it `bucket`) plus one numeric column per series |
| Stat                        | One row with a numeric column named `value`                         |
| Pie, funnel, horizontal bar | A `name` column and a `value` column                                |
| Heatmap                     | `x`, `y` and `value` columns                                        |
| Histogram                   | One `value` column, one row per observation                         |
| Table                       | Any columns                                                         |

For time series, every non-time column becomes a series named after the column, and non-numeric columns are dropped. Rows are not split by a string column, so to draw several lines, return one column per line, for example with `countIf`:

```sql
SELECT $__timeGroup(Timestamp) AS bucket,
       countIf(StatusCode = 'Error') AS errors,
       count() AS requests
FROM service_overview_spans
WHERE $__orgFilter AND $__timeFilter(Timestamp)
GROUP BY bucket
ORDER BY bucket
```

## What is allowed

A query is a single `SELECT` or `WITH` statement. A trailing `;` is fine, and a trailing `FORMAT` clause is dropped. Rejected:

- More than one statement.
- Any write or DDL keyword: `INSERT`, `UPDATE`, `DELETE`, `DROP`, `ALTER`, `TRUNCATE`, `RENAME`, `ATTACH`, `DETACH`, `CREATE`, `GRANT`, `REVOKE`, `OPTIMIZE`, `SYSTEM`, `KILL`, and `INTO OUTFILE`.
- A `SETTINGS` clause.
- Table functions that reach outside Maple, such as `url`, `file`, `s3`, `remote`, `mysql`, `postgresql` and `dictionary`.

A rejected query returns `400` with one of these codes: `MissingOrgFilter`, `InvalidMacro`, `UnresolvedMacro`, `DisallowedStatement`, `DisallowedFunction`, `MultipleStatements`, `ResourceLimit`.

## Limits

| Limit               | Value                                                                  |
| ------------------- | ---------------------------------------------------------------------- |
| Query text          | 32,768 characters                                                      |
| Rows returned       | 1,000. More is an error, not a silent cut: aggregate or add a `LIMIT`. |
| Result size         | 5 MB of JSON, 64,000 characters per cell                               |
| Execution time      | 10 seconds for dashboards and `run_sql`, 5 seconds for alerts          |
| Raw-SQL alert rules | Must use `$__timeFilter(…)`                                            |

Your query is wrapped in an outer `LIMIT`, so `WITH TOTALS`, `LIMIT BY` and `WITH FILL` in the outermost query are discarded.

## Related

- [Maple MCP server](/docs/reference/mcp): `run_sql` and `describe_warehouse_tables`
- [OpenTelemetry conventions](/docs/concepts/otel-conventions): which attributes land in which column
- [Limits](/docs/reference/limits)

---

# Alert webhooks

> Reference · https://maple.dev/docs/reference/webhooks

The webhook destination contract: request headers, the JSON payload for each event, verifying the HMAC signature, retries, and idempotency.

A webhook [notification destination](/docs/alerting/notification-destinations#webhook) sends every alert event to a URL you own. This page is the contract your endpoint can rely on.

## The request

Maple sends a `POST` with a JSON body and these headers:

| Header                 | Value                                                                    |
| ---------------------- | ------------------------------------------------------------------------ |
| `content-type`         | `application/json`                                                       |
| `x-maple-event-type`   | `trigger`, `resolve`, `renotify` or `test`                               |
| `x-maple-delivery-key` | A stable ID for this delivery. Retries reuse it; use it to deduplicate.  |
| `x-maple-signature`    | Hex HMAC-SHA256 of the raw body. Only sent when a signing secret is set. |

Respond with any `2xx` within **15 seconds**. The response body is ignored.

## Events

| Event      | Sent when                                                                                  |
| ---------- | ------------------------------------------------------------------------------------------ |
| `trigger`  | An incident opens, after the rule's required number of consecutive breaching checks        |
| `renotify` | An incident is still breaching after the rule's re-notify interval (30 minutes by default) |
| `resolve`  | An incident closes, after the rule's required number of consecutive healthy checks         |
| `test`     | You press **Send test** on the destination, or test a rule with notifications on           |

## Payload

```json
{
	"eventType": "trigger",
	"incidentId": "b6f1…",
	"incidentStatus": "open",
	"dedupeKey": "org_…:alrt_…:checkout",
	"rule": {
		"id": "alrt_…",
		"name": "Checkout error rate",
		"signalType": "error_rate",
		"severity": "critical",
		"groupKey": "checkout",
		"comparator": "gt",
		"threshold": 5,
		"thresholdUpper": null,
		"windowMinutes": 5
	},
	"observed": { "value": 8.4, "sampleCount": 1250 },
	"template": null,
	"chart": { "url": "https://…" },
	"linkUrl": "https://app.maple.dev/alerts/…",
	"chatUrl": "https://app.maple.dev/chat?…",
	"sentAt": "2026-09-25T09:14:00.000Z",
	"event": { "specversion": "1.0", "type": "dev.maple.alert.lifecycle.trigger.v1", "…": "…" }
}
```

| Field                  | Description                                                                                                |
| ---------------------- | ---------------------------------------------------------------------------------------------------------- |
| `eventType`            | Same as the `x-maple-event-type` header                                                                    |
| `incidentId`           | The incident this event belongs to. `null` for tests.                                                      |
| `incidentStatus`       | `open` or `resolved`                                                                                       |
| `dedupeKey`            | Stable for one incident across `trigger`, `renotify` and `resolve`. Use it to thread events together.      |
| `rule.signalType`      | `error_rate`, `p95_latency`, `p99_latency`, `apdex`, `throughput`, `builder_query` or `raw_query`          |
| `rule.severity`        | `warning` or `critical`                                                                                    |
| `rule.groupKey`        | The group that breached, such as a service name. `__total__` for an ungrouped rule.                        |
| `rule.comparator`      | `gt`, `gte`, `lt`, `lte`, `eq`, `neq`, `between` or `not_between`                                          |
| `rule.threshold`       | The threshold, or the lower bound for `between` and `not_between`                                          |
| `rule.thresholdUpper`  | The upper bound for `between` and `not_between`, otherwise `null`                                          |
| `rule.windowMinutes`   | The evaluation window                                                                                      |
| `observed.value`       | The value the check measured                                                                               |
| `observed.sampleCount` | How many data points the value was computed from                                                           |
| `template`             | The rule's rendered notification template, or `null`                                                       |
| `chart`                | `url` of a chart image and/or a `sparkline`, or `null`                                                     |
| `linkUrl`              | The incident in the Maple dashboard                                                                        |
| `chatUrl`              | Opens Maple's AI chat with the incident as context                                                         |
| `sentAt`               | ISO-8601 time the event was first generated. Retries keep the original value.                              |
| `event`                | The same event as a [CloudEvents 1.0](https://cloudevents.io) envelope, for routers that speak CloudEvents |

The body is identical for every destination's webhook, regardless of the rule's notification template. Treat unknown fields as additive: new fields can appear without notice.

### Error issue notifications

A destination listed in your organization's error notification policy (set with the `update_error_notification_policy` MCP tool or the API) also receives error issue events: a new issue, a regression, a resolve, and optionally workflow changes and claims. They carry the same headers and a smaller body: `eventType`, `incidentId`, `incidentStatus`, `dedupeKey`, `rule`, `observed`, `linkUrl`, `chatUrl` and `sentAt`, without `event`, `template` or `chart`. Here `rule.id` is the issue ID, `rule.name` reads `<ExceptionType> in <service>`, and `observed.value` is the occurrence count. `dedupeKey` starts with `error:`.

Issues escalated by triage arrive with `eventType: "escalation"` in the body and an extra `escalation` object describing the issue. The `x-maple-event-type` header on these reads `trigger`, so branch on the body's `eventType`.

## Verifying the signature

Set a **signing secret** on the destination and Maple signs each body. The signature is the lowercase hex HMAC-SHA256 of the **exact bytes** of the request body, keyed with the secret, with no prefix. Compute it over the raw body before parsing the JSON, and compare in constant time.

```ts
import { createHmac, timingSafeEqual } from "node:crypto"

export function isFromMaple(rawBody: Buffer, signature: string | undefined, secret: string): boolean {
	if (!signature) return false
	const expected = createHmac("sha256", secret).update(rawBody).digest()
	const received = Buffer.from(signature, "hex")
	return received.length === expected.length && timingSafeEqual(received, expected)
}
```

```python
import hashlib, hmac

def is_from_maple(raw_body: bytes, signature: str | None, secret: str) -> bool:
    if not signature:
        return False
    expected = hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature)
```

The signature covers the body only, and the body carries no delivery timestamp you can trust for freshness, so a captured request could be replayed. Deduplicate on `x-maple-delivery-key` to make replays harmless.

## Retries and failures

| Your response                                  | What Maple does                                 |
| ---------------------------------------------- | ----------------------------------------------- |
| `2xx`                                          | Delivered                                       |
| `408`, `429`, `5xx`, timeout, connection error | Retries with backoff                            |
| `401`, `403`                                   | Fails without retrying (authentication problem) |
| `404`, `410`                                   | Fails without retrying (endpoint gone)          |
| Any other `4xx`                                | Fails without retrying (rejected)               |

Alert events are attempted up to **5 times**, about 1, 2, 4 and 8 minutes apart, with every retry carrying the same body and `x-maple-delivery-key`. **Send test** is attempted once.

After **3 consecutive non-retryable failures**, the destination is disabled and the reason is shown on it under **Alerts → Destinations**. Fix the endpoint and re-enable it; a successful delivery resets the count.

## Endpoint checklist

- Read the raw body, verify the signature, then parse.
- Return `2xx` quickly and do slow work asynchronously; anything past 15 seconds is a failed attempt.
- Deduplicate on `x-maple-delivery-key`; retries are normal.
- Group by `dedupeKey` to follow one incident from `trigger` to `resolve`.
- Accept `test` events so **Send test** succeeds.

---

# Retention

> Reference · https://maple.dev/docs/reference/retention

How long hosted Maple keeps each kind of data: traces, logs, metrics, session replays, product events, error issues, alert history and the audit log.

Hosted Maple deletes data automatically once it passes its retention period. The periods are the same in both [regions](/docs/reference/regions). Age is measured from the data's own timestamp (for example a span's start time), not from when Maple received it.

## Telemetry

| Data                                                                                              | Kept for |
| ------------------------------------------------------------------------------------------------- | -------- |
| Traces (spans)                                                                                    | 30 days  |
| Logs                                                                                              | 30 days  |
| Metrics (sum, gauge, histogram, exponential histogram)                                            | 90 days  |
| Session replays (recordings and their timeline events)                                            | 30 days  |
| Product events                                                                                    | 365 days |
| Error occurrences (the individual events behind error issues)                                     | 90 days  |
| Hourly rollups of traces and logs (service overview, service map edges, trace and log aggregates) | 365 days |

Individual spans, log lines and data points cannot be read once they pass these periods. Charts built on hourly rollups, such as service overviews and the service map, can reach further back, up to the rollup's 365 days. Some queries have shorter maximum time ranges. See [Limits](/docs/reference/limits#queries).

## Error issues

[Error issues](/docs/errors/overview) are not deleted by age while they are open, however old the underlying errors are. These rules clean them up:

| Issue state                                                                              | What happens                                                         |
| ---------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| Resolved (`done`) for 14 days                                                            | The issue is archived                                                |
| Grouped under an older fingerprinting scheme, after Maple changes how errors are grouped | The issue is archived, and new occurrences open a new issue          |
| Archived for 90 days                                                                     | The issue is deleted, with its incidents, comments and state history |
| A fingerprint that never became an issue                                                 | Forgotten after 24 hours without a new occurrence                    |

## Alerts and audit history

| Data                                                         | Kept for             |
| ------------------------------------------------------------ | -------------------- |
| Alert rule checks (each evaluation, with its observed value) | 365 days             |
| Organization audit log (**Settings → Audit log**)            | 6 years (2,190 days) |

## Maple Local

Maple Local keeps logs and traces for 30 days and metrics for 90, and lets you raise the floor for raw data. See [Your data](/docs/local-mode#your-data). To keep history longer, export it with [archives](/docs/local-mode/checkpoints-and-archives#archives).

---

# Limits

> Reference · https://maple.dev/docs/reference/limits

Every limit in one place: ingest request size and concurrency, query time ranges, raw SQL, alert rules, event fields, and API and MCP rate limits.

The limits Maple enforces, grouped by where you would hit them. Each links to the page that explains the behavior in full.

## Ingest

See [Ingest API](/docs/reference/ingest) for status codes and retry guidance.

| Limit               | Value                                    | Over the limit               |
| ------------------- | ---------------------------------------- | ---------------------------- |
| Request body        | 20 MiB, measured compressed              | `413`                        |
| Request duration    | 30 seconds                               | `503`, retry after 5 s       |
| Concurrent requests | 1,000 in flight per organization         | `429`, retry after 1 s       |
| Compression         | gzip only                                | `415`                        |
| Session replay      | 1 GiB of uncompressed data per recording | Recording stops at the limit |

## Product events and session events

See [Product events API](/docs/product-events/api). An event that breaks a **dropped** rule is discarded silently; the rest of the batch is still stored. Other fields are truncated to their limit.

| Field                                | Limit                                      | Over the limit |
| ------------------------------------ | ------------------------------------------ | -------------- |
| Event name                           | 128 bytes; no `$` prefix except `$screen`  | Dropped        |
| Event timestamp                      | 30 days in the past to 1 day in the future | Dropped        |
| User, visitor, session and group IDs | 256 bytes                                  | Truncated      |
| Page path                            | 1 KiB                                      | Truncated      |
| URL                                  | 2 KiB                                      | Truncated      |
| Session event message                | 1 KiB                                      | Truncated      |
| Session event attributes             | 32 per event; keys 64 bytes, values 1 KiB  | Truncated      |

## Queries

The dashboard, the API and the MCP tools share these limits. Queries time out after **30 seconds**.

| Query                                   | Longest time range |
| --------------------------------------- | ------------------ |
| Charts and time series                  | 31 days            |
| Trace and log lists                     | 7 days             |
| Breakdowns (top-N by service, route, …) | 30 days            |
| Breakdowns with no filter               | 24 hours           |
| Attribute and metric discovery          | 30 days            |
| Log pattern mining                      | 24 hours           |

A time series returns at most **1,500 points**. Bucket width is chosen from the range, from 1 minute up to 1 day.

## Raw SQL

See [SQL reference](/docs/reference/sql).

| Limit          | Value                                      |
| -------------- | ------------------------------------------ |
| Query text     | 32,768 characters                          |
| Rows returned  | 1,000 (more is an error)                   |
| Result size    | 5 MB of JSON; 64,000 characters per cell   |
| Execution time | 10 seconds; 5 seconds inside an alert rule |

## Alert rules

| Limit                     | Value                                           |
| ------------------------- | ----------------------------------------------- |
| Evaluation window         | 1 minute to 24 hours                            |
| Check interval            | Every minute                                    |
| Groups per raw-SQL alert  | 100                                             |
| Tags per rule             | 20, up to 32 characters each                    |
| Notification template     | 4,000 characters for the title and for the body |
| Webhook response time     | 15 seconds                                      |
| Webhook delivery attempts | 5                                               |

## API and MCP rate limits

| Surface                                    | Limit                               | Over the limit           |
| ------------------------------------------ | ----------------------------------- | ------------------------ |
| [Maple API](/docs/reference/api) (`/v2`)   | 600 requests per 60 seconds per key | `429`, `Retry-After: 60` |
| [MCP server](/docs/reference/mcp) requests | 120 per 10 seconds per key or user  | `429`, `Retry-After: 10` |
| API list pages                             | `limit` 1 to 100, default 20        | `400`                    |

Rate limits are counted per edge location, so treat them as approximate rather than an exact global budget.