OpenTelemetry tracing for Go services

Go has no in-process auto-instrumentation. You register the OpenTelemetry SDK once at startup, then wrap each library you use: your HTTP mux, your gRPC server, your database handle. The context you already pass around carries the trace.

GET /api/orders/{id} 5 spans · 48ms
DurationStatus
/api/orders/{id}orders-api
48.0msOk
SELECT orders
7.0msOk
inventory.v1.Inventory/Reserve
29.0msOk
inventory.v1.Inventory/Reserveinventory-svc
25.0msOk
UPDATE stock
12.0msOk

Example: a Go request, with otelgrpc and otelsql added

Quick start

Set up Go

1

Install the SDK, exporters and instrumentation packages

bash
go get go.opentelemetry.io/otel \
  go.opentelemetry.io/otel/sdk \
  go.opentelemetry.io/otel/sdk/log \
  go.opentelemetry.io/otel/log \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp \
  go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp \
  go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp \
  go.opentelemetry.io/contrib/bridges/otelslog
2

Register the providers at startup

go
// telemetry.go: the exporters read OTEL_EXPORTER_OTLP_* from the environment
package main

import (
	"context"
	"errors"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/log/global"
	"go.opentelemetry.io/otel/propagation"
	sdklog "go.opentelemetry.io/otel/sdk/log"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

func initTelemetry(ctx context.Context) (func(context.Context) error, error) {
	res, err := resource.New(ctx, resource.WithFromEnv())
	if err != nil {
		return nil, err
	}

	traceExporter, err := otlptracehttp.New(ctx)
	if err != nil {
		return nil, err
	}
	tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(traceExporter), sdktrace.WithResource(res))
	otel.SetTracerProvider(tp)
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
		propagation.TraceContext{}, propagation.Baggage{},
	))

	logExporter, err := otlploghttp.New(ctx)
	if err != nil {
		return nil, errors.Join(err, tp.Shutdown(ctx))
	}
	lp := sdklog.NewLoggerProvider(
		sdklog.WithProcessor(sdklog.NewBatchProcessor(logExporter)),
		sdklog.WithResource(res),
	)
	global.SetLoggerProvider(lp)

	return func(ctx context.Context) error {
		return errors.Join(tp.Shutdown(ctx), lp.Shutdown(ctx))
	}, nil
}
3

Wrap your handler, and flush when the process is stopped

go
// main.go
package main

import (
	"context"
	"log"
	"log/slog"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"

	"go.opentelemetry.io/contrib/bridges/otelslog"
	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
	defer stop()

	// A setup error only disables telemetry; it never stops the app.
	shutdown, err := initTelemetry(ctx)
	if err != nil {
		log.Printf("telemetry disabled: %v", err)
	} else {
		// slog records go to Maple; log with InfoContext so they carry the trace
		slog.SetDefault(otelslog.NewLogger("my-go-app"))
	}

	mux := http.NewServeMux()
	mux.HandleFunc("GET /api/orders/{id}", handleOrders)

	// Every request becomes a server span
	srv := &http.Server{Addr: ":8080", Handler: otelhttp.NewHandler(mux, "server")}
	go func() {
		if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
			log.Fatal(err)
		}
	}()

	<-ctx.Done()
	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()
	srv.Shutdown(ctx)
	if shutdown != nil {
		shutdown(ctx) // sends the last batch
	}
}
4

Point the exporters at Maple

bash
export OTEL_SERVICE_NAME="my-go-app"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production"

go run .

This is the short path. The full Go guide covers the rest.

Organization in the EU region? Send to https://ingest.eu.maple.dev instead of https://ingest.maple.dev. An ingest key only works in its own region.

What you get

What a Go service sends to Maple

HTTP server spans

otelhttp turns each request into one root server span. Current versions name it after the method and the ServeMux pattern, such as GET /api/orders/{id}.

Outgoing HTTP calls

Wrap the client transport with otelhttp.NewTransport. Each call gets a client span and sends traceparent. When the callee is instrumented too, that call is an edge on the service map.

gRPC

otelgrpc stats handlers trace both sides of a call. Client and server spans join into one trace across services.

database/sql queries

Open the database with github.com/XSAM/otelsql, or use otelpgx for pgx. Each query records its statement and database system, and the database appears as its own node on the service map.

slog records with trace ids

The otelslog bridge exports log/slog records. A record logged with the request context carries that request's trace and span id.

Errors on spans

span.RecordError and span.SetStatus(codes.Error, ...) mark the operation that failed, and the trace shows where in the request it happened.

What is different in Go

One package per library

Go cannot patch libraries at runtime. Nothing is traced until you wrap it with its contrib package, so list the libraries a request touches and wrap each one.

ctx is the trace

Spans link through context.Context. Call QueryContext, NewRequestWithContext and logger.InfoContext, or the child span starts a new trace.

Flush before exit

The span batcher sends every 5 seconds. A deploy stops the process with SIGTERM, and os.Exit and log.Fatal skip deferred calls. The main above waits for the signal and flushes before it exits.

インテグレーション

その他のインテグレーション

今日、最初のトレースを。

SDKを追加し、OTLPエクスポーターをMapleに向ければトレースが届きます。

maple.dev:OpenTelemetryの上のオブザーバビリティ