OTLP, the OpenTelemetry Protocol, is how telemetry moves between an OpenTelemetry SDK, a Collector and a backend. If you have seen 4317, 4318 or OTEL_EXPORTER_OTLP_ENDPOINT, you have configured OTLP.
Most OTLP problems are configuration problems: the wrong port for the transport, a path appended twice, a header in the wrong format. This guide explains the parts and then works through the errors in the order you are likely to hit them.
Key takeaways
- gRPC listens on 4317 and HTTP on 4318 by default. Sending one transport to the other's port fails with confusing errors.
http/protobufis the specified default and the safest choice when in doubt.OTEL_EXPORTER_OTLP_ENDPOINTis a base URL. The per-signal variables are used as written, path included.- Retry on 429, 502, 503 and 504. Do not retry 400, 401, 403 or 413 unchanged.
- A 200 response can still carry
partialSuccesswith rejected items. Read it.
What OTLP is
OTLP is a specification with two parts. The first is a set of protobuf messages: ExportTraceServiceRequest, ExportMetricsServiceRequest, ExportLogsServiceRequest and, still in development, one for profiles. Each request groups data by resource (the service and host that produced it), then by instrumentation scope (the library that recorded it).
The second part is how those messages travel: over gRPC, or as HTTP POST requests. The traces, metrics and logs parts of the protocol are stable, so an exporter written today talks to a Collector from two years ago.
A request is a batch. An SDK collects spans for a few seconds (5 seconds by default for traces) or until it has 512 of them, and sends them in one call. That is why a span shows up in the backend a few seconds after the request finishes, and why a process that exits without flushing loses its last batch.
gRPC vs HTTP
Both transports carry the same messages. They differ in where they work well.
| OTLP/gRPC | OTLP/HTTP | |
|---|---|---|
| Default port | 4317 | 4318 |
| Protocol value | grpc | http/protobuf or http/json |
| Where data goes | The Export method of TraceService, MetricsService or LogsService | POST /v1/traces, /v1/metrics, /v1/logs |
| Encoding | Protobuf only | Protobuf or JSON |
| Browsers | No | Yes |
| Proxies and load balancers | Needs HTTP/2 end to end; long-lived connections can pin to one backend | Works through anything that forwards HTTP |
| Errors | gRPC status codes such as UNAVAILABLE | HTTP status codes |
The specification says SDKs should default to http/protobuf, though some SDKs kept grpc as their default for compatibility with earlier releases. Check your SDK's docs, or set OTEL_EXPORTER_OTLP_PROTOCOL explicitly so nobody has to guess.
Choose HTTP when traffic crosses a proxy, an API gateway or the public internet, or when it comes from a browser. Choose gRPC between services and Collectors you run yourself, where its single multiplexed connection saves a little overhead. The difference in throughput rarely matters next to batch size and compression.
One gRPC caveat: a layer-4 load balancer balances connections, not requests. A gRPC exporter opens one connection and keeps it, so all its data lands on whichever Collector it first reached. Use a load balancer that understands gRPC, or OTLP/HTTP.
Protobuf vs JSON
Over HTTP you choose the encoding with Content-Type: application/x-protobuf for binary protobuf, application/json for JSON. The server answers in the same encoding it received.
Protobuf is smaller and cheaper to encode, and it is what you want in production. JSON is for places where protobuf is awkward, such as a hand-written curl test, a shell script or a small browser exporter. OTLP/JSON is not the JSON you would guess from the protobuf schema. Four rules trip people up:
{
"traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
"spanId": "00f067aa0ba902b7",
"kind": 2,
"startTimeUnixNano": "1763712000000000000",
"droppedAttributesCount": 0
} - IDs are hex strings.
traceIdis 32 hex characters andspanIdis 16. The standard protobuf-to-JSON mapping would use base64, and a base64 ID is rejected with a 400. - Enums are integers.
"kind": 2for a server span, never"SPAN_KIND_SERVER". - Keys are lowerCamelCase.
droppedAttributesCount, notdropped_attributes_count. - 64-bit integers are strings. Timestamps such as
startTimeUnixNanoare decimal strings. Receivers accept numbers too, but a JavaScript number loses precision past 253.
Compression is separate from encoding. Set OTEL_EXPORTER_OTLP_COMPRESSION=gzip in the SDK; the Collector's OTLP exporters already use gzip by default. Telemetry repeats the same keys and values constantly and compresses well, which matters for both egress cost and the 413 errors below.
Endpoints and environment variables
Every OpenTelemetry SDK reads the same environment variables, so this setup works in any language:
export OTEL_SERVICE_NAME="checkout"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
Each variable has a per-signal form, such as OTEL_EXPORTER_OTLP_TRACES_ENDPOINT or OTEL_EXPORTER_OTLP_LOGS_HEADERS, which overrides the generic one for that signal. The two endpoint forms behave differently, and that difference causes most OTLP 404s:
# Generic variable: a base URL. The SDK appends /v1/traces, /v1/metrics, /v1/logs.
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318
# traces -> http://collector:4318/v1/traces
# Per-signal variable: used exactly as written. Include the path yourself.
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://collector:4318/v1/traces
With no endpoint set, SDKs send to http://localhost:4318 for HTTP or http://localhost:4317 for gRPC. That default is right when a Collector runs on the same host, and wrong almost everywhere else.
The export timeout defaults to 10 seconds per batch (OTEL_EXPORTER_OTLP_TIMEOUT, in milliseconds). Retries happen inside that budget, so a slow endpoint shows up as timeouts, then dropped batches.
Auth headers and TLS
OTLP has no authentication of its own. Backends read a header: usually Authorization: Bearer <key>, sometimes a vendor header. Over HTTP it is a request header; over gRPC it is metadata, where keys are lowercase.
OTEL_EXPORTER_OTLP_HEADERS takes a comma-separated list of key=value pairs in the W3C Baggage format. Two details matter. A comma inside a value ends it, so a key containing a comma must be percent-encoded. And quoting is part of the value, so Authorization="Bearer abc" sends the quotes. Most SDKs accept the space in Bearer abc as written; if yours rejects it, encode it as Bearer%20abc.
In a Collector, headers go on the exporter, and the key comes from the environment rather than the file:
config.yaml
exporters:
otlp_http:
endpoint: https://ingest.maple.dev
headers:
Authorization: Bearer ${env:MAPLE_INGEST_KEY}
otlp_grpc:
endpoint: https://ingest.maple.dev
headers:
authorization: Bearer ${env:MAPLE_INGEST_KEY}
The scheme in the endpoint sets transport security. https:// means TLS and http:// means plain text, for both transports. A gRPC endpoint with no scheme falls back to the insecure setting (OTEL_EXPORTER_OTLP_INSECURE), which defaults to TLS on. For a private certificate authority, point OTEL_EXPORTER_OTLP_CERTIFICATE at its PEM file; for mutual TLS, also set OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE and OTEL_EXPORTER_OTLP_CLIENT_KEY.
Common errors and fixes
OTLP defines which failures a client retries. Over HTTP, 429, 502, 503 and 504 are retryable, with exponential backoff or after the Retry-After header when the server sends one. Every other 4xx and 5xx is final: sending the same batch again gets the same answer. Over gRPC the retryable signal is UNAVAILABLE.
| Symptom | Usual cause | Fix |
|---|---|---|
| Connection refused | Nothing listens at that host and port | Check the host, the port and the Collector's bind address |
400 Bad Request | Undecodable body, such as JSON with base64 IDs | Fix the payload; never retry it |
401 / 403 | Missing, wrong or revoked key, or a key from another region | Check the header name, the Bearer prefix and the key |
404 Not Found | Wrong path, usually a missing or doubled /v1/traces | Use the base URL in the generic variable |
413 | The request body is over the server's limit | Smaller batches, gzip on |
415 | Unsupported Content-Type or compression | Protobuf or JSON, gzip or none |
429 / 503 | Rate limit or backpressure | Let the exporter retry; lower volume if it persists |
| Protocol or parse errors | gRPC sent to the HTTP port, or the reverse | Match the protocol to the port |
Connection refused
The TCP connection never opened. In order of likelihood: the Collector is not running; the exporter uses the default localhost while the Collector runs in another container, so localhost is the app's own container; or the Collector listens only on localhost inside its container, so nothing outside can reach it. The last one is the default since Collector v0.104.0. Bind the receiver to 0.0.0.0:4318 and point the app at the Collector's service name, such as http://otel-collector:4318.
401 and 403
A 401 means the backend did not accept the credential. Check that the header reached the exporter: a variable set in your shell is not set inside a container or a systemd unit. Check the format: Authorization=Bearer key in the environment variable, and the exact header name your backend documents. Some backends issue keys per region and reject a key sent to the other region's endpoint. A 403 usually means the key is valid but not allowed to send that signal.
404
Almost always the path. Setting OTEL_EXPORTER_OTLP_ENDPOINT=https://host/v1/traces sends traces to /v1/traces/v1/traces. Setting OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://host sends them to /. In a Node.js test, the HTTP exporter given a bare http://localhost:4318 as its URL got Not Found from the Collector, because a URL passed in code is also used as written.
413 Content Too Large
The specification tells servers to answer 413 when a request is over their size limit, and suggests supporting at least 64 MiB. Hosted backends often set lower limits, and proxies in front of a Collector (an nginx default is 1 MB) are lower still. Make batches smaller at the source:
# Smaller span batches from the SDK (default 512 spans)
export OTEL_BSP_MAX_EXPORT_BATCH_SIZE=256 In a Collector, cap the batch the exporter builds. max_size splits anything larger:
exporters:
otlp_http:
endpoint: https://ingest.maple.dev
sending_queue:
enabled: true
batch:
enabled: true
sizer: items
min_size: 2000 # the default minimum is 8192, above this maximum
max_size: 4000 Turn on gzip if it is off. Check which size the limit applies to: some backends measure the compressed body, others the decompressed one.
Protocol mismatch
Sending gRPC to the HTTP port, or HTTP to the gRPC port, does not produce a clear message. In a test with the Node.js exporters against a Collector, the gRPC exporter pointed at 4318 failed with 14 UNAVAILABLE: No connection established. Last error: Protocol error, and the HTTP exporter pointed at 4317 failed with Parse Error: Expected HTTP/, RTSP/ or ICE/. Other languages word it differently, often as an HTTP/2 frame or preface error. When you see a protocol-level error, check that OTEL_EXPORTER_OTLP_PROTOCOL and the port agree before anything else.
Debug an exporter
When data does not arrive, take the application out of the picture and test the endpoint directly. An empty OTLP/JSON request is valid and checks the URL, the TLS setup and the key in one go:
curl -i https://ingest.maple.dev/v1/traces \
-H "Authorization: Bearer YOUR_INGEST_KEY" \
-H "Content-Type: application/json" \
-d '{"resourceSpans":[]}' 200 means the endpoint and key work, so the problem is in the app's configuration. 401 points at the key and 404 at the path. If curl cannot connect at all, the network is the problem: DNS, a firewall, or a proxy that needs its own settings.
Then check the SDK side:
- Print the effective config. Log
OTEL_EXPORTER_OTLP_*at startup, minus the header values. A typo such asOTEL_EXPORTER_OTLP_ENDPIONTis silently ignored. - Turn on SDK logs.
OTEL_LOG_LEVEL=debugmakes the SDKs that support it log each failed export. Support varies by language. - Flush on exit. Short-lived processes, such as CLI tools, cron jobs and serverless functions, must call
shutdown()orforceFlush()on the provider before exiting, or the last batch never leaves. - Read partial success. A
200whose body haspartialSuccess.rejectedSpansgreater than zero means some items were dropped, with the reason inerrorMessage. Clients do not retry it, so log it. - Put a Collector in between. A local Collector with the
debugexporter shows exactly what the SDK sends. The Collector guide has a config that does this.
Data that arrives but is incomplete is usually not an OTLP problem. Missing child spans point at context propagation, and traces that appear for some requests but not others point at sampling.
Maple
Send OTLP to Maple
Set the endpoint to https://ingest.maple.dev and the header to your ingest key. Both transports work on port 443, and the ingest reference lists every status code with its fix.