Go API observability: logs, metrics and traces linked by a trace_id

A thousand requests hit a small invoice API written in Go. An API is a program that answers other programs over the web. Each call it receives is a request, for example "give me invoice 1". The program that sends the request is the client.

About one request in twenty fails on purpose, and another one in twenty is slowed down on purpose by 700 ms. It is a test setup. It asks the everyday question of production, meaning the real service, facing real clients. Which requests failed, and where did the slow ones lose their time?

Three tools answer, and each one gives a piece of the answer:

• logs, lines the program writes at each event, say what happened;
• metrics, counters updated on every request, say how many requests, how many errors and how fast;
• traces say where the time goes. A trace is the full journey of a request, cut into timed steps.

Three separate pieces do not get you far. A log line must lead to the trace of the same request. A spike on a graph must lead to the slow traces of that moment. The thread that ties all three together is a number, the trace_id: the unique identifier of each trace. The diagram below shows that thread.

To produce and store these three signals, the repository runs five programs side by side:

• the invoice API, in Go, which writes its logs and keeps its counters;
• Prometheus, a server that reads the API's counters every five seconds and stores them;
• the Collector, a relay that receives the steps of the traces, sorts the traces and forwards the ones it keeps;
• Jaeger, a server that stores traces and shows them as a timeline;
• Grafana, the tool that draws dashboards from Prometheus and opens traces from Jaeger.

The whole thing starts with one command, docker compose up, and is called the stack. Traces are produced with OpenTelemetry, an open standard for creating traces, metrics and logs, with one library per language. The Collector is part of that standard.

Here is how I go about it in a small public repository: the source code, published on GitHub. The repository builds on three books, and each one has mistakes, gathered at the end of the article:

• Mastering Go, 4th edition, by Mihalis Tsoukalos (Packt, 2024);
• Modern REST API Development in Go, by Jesús Espino (Packt, 2025);
• Cloud-Native Observability with OpenTelemetry, by Alex Boten (Packt, 2022).

$ git clone https://github.com/ohugonnot/go-api-observability
$ cd go-api-observability && ./scripts/e2e.sh

slog in JSON: every log line carries the trace_id

Logs first. Since version 1.21, Go ships the log/slog package. That package writes structured logs: each line is a JSON object, a text format made of named fields such as "status":500. A tool can then filter on a field instead of searching text.

A trace is split into steps called spans. The whole journey is the trace, each step of the journey is a span. Spans form a tree: the request's span is the parent, the invoice read is its child. Each span has its own number, the span_id, and all the spans of a trace share the same trace_id.

In Go, a request carries a context (context.Context): a bag of values that follows it from function to function. OpenTelemetry stores the current span in that bag.

The repository wraps slog's JSON formatter in a small type, traceHandler. Before writing a line, its Handle method looks for a span in the context. If there is one, it adds two fields, trace_id and span_id, then hands the line to the usual JSON formatter:

func (h traceHandler) Handle(ctx context.Context, r slog.Record) error {
	if sc := trace.SpanContextFromContext(ctx); sc.IsValid() {
		r.AddAttrs(
			slog.String("trace_id", sc.TraceID().String()),
			slog.String("span_id", sc.SpanID().String()),
		)
	}
	return h.Handler.Handle(ctx, r)
}

Full file: logging.go, lines 19 to 27

The sc.IsValid() check matters: a line written outside any request has no span, so no empty field is written. The last line, h.Handler.Handle(ctx, r), hands control back to the original JSON formatter.

The practical trap is elsewhere: the fields only show up if the code passes the context. Write log.InfoContext(ctx, …), not log.Info(…), or traceHandler finds no span.

Prometheus metrics: one label per route, never per URL

Now metrics. The API publishes its measurements as text at one address, /metrics, and Prometheus reads them every five seconds. The repository keeps two measurements per request:

• a counter of requests, with their status code, which gives the request rate and the error rate;
• a histogram of durations, which sorts each request into a bucket, from 0.5 ms to 5 s.

Those three questions, rate, errors and duration, make up the RED method, for Rate, Errors, Duration.

Each measurement carries labels: tags such as the method, the route and the status code. Each combination of labels creates a separate curve, which Prometheus calls a series. For example:

• method="GET", invoice route, code 200: one series;
• same method, same route, code 404: a second series;
• method="POST", creation route, code 201: a third.

The trap is a label that takes too many values. With the full URL as a label, every invoice gets its own series: /invoices/1, /invoices/2, and so on. A million invoices make a million series for a single counter. That number of distinct values is the cardinality. Prometheus keeps every series in memory: unbounded, its memory grows with the cardinality.

Modern REST API Development in Go describes this trap very well in lesson 64 (the books are split into numbered lessons). Jesús Espino writes that "the abuse of labels can lead to inefficiencies": a badly chosen label is expensive. His example is a label per request ID, which would quickly lead to millions of series.

Two lessons later, his measuring code, meant for every route, still uses r.URL.Path as a label (lesson 66). r.URL.Path is the exact path of the URL, number included. Yet his API has routes such as GET /lists/{id}, where {id} is the slot for the list number. Every list therefore creates its own series: exactly the trap the book describes.

The fix is the route pattern: the URL with the slot for the number, not the number itself. Since Go 1.23, the standard router, the code that picks which function answers which URL, writes that pattern into the r.Pattern field. For an invoice, the pattern is GET /invoices/{id}, whatever the number.

The function below turns it into the route label. If no pattern matched the request, it returns unmatched. Otherwise, strings.Cut splits the pattern at the first space and keeps the path, without the method:

func routeLabel(r *http.Request) string {
	if r.Pattern == "" {
		return "unmatched" // 404/405: bounded, whatever the URL
	}
	if _, path, ok := strings.Cut(r.Pattern, " "); ok {
		return path // "GET /invoices/{id}" -> "/invoices/{id}"
	}
	return r.Pattern
}

Full file: metrics.go, lines 137 to 145

The code comments say it: unmatched covers 404 and 405 responses, whatever the URL. A client can send any URL it likes; without this branch, every invented URL would create its own series.

The next test sends four requests: two for invoice 1, one for invoice 42, which does not exist, and one for an unknown URL. It then writes the expected series, in the text format of /metrics, and compares them with the real counters:

	app.do("GET", "/invoices/1", "")
	app.do("GET", "/invoices/1", "")
	app.do("GET", "/invoices/42", "") // 404
	app.do("GET", "/does/not/exist", "")

	want := `
# HELP http_requests_total HTTP requests by method, route pattern and status code.
# TYPE http_requests_total counter
http_requests_total{code="200",method="GET",route="/invoices/{id}"} 2
http_requests_total{code="404",method="GET",route="/invoices/{id}"} 1
http_requests_total{code="404",method="GET",route="unmatched"} 1
`
	if err := testutil.GatherAndCompare(app.reg, strings.NewReader(want), "http_requests_total"); err != nil {
		t.Error(err)
	}

Full file: metrics_test.go, lines 21 to 35

The # HELP and # TYPE lines describe the counter. The next three lines are the three series, with their value at the end. None of them contains an invoice number. Putting r.URL.Path back in place of the pattern makes this test fail: it is one of the 54 deliberate changes listed in the box above.

Runtime metrics: the measurements Go already provides

The request counters say what the API does. What remains is what the Go program itself does. Go's execution engine, the runtime, manages memory and the garbage collector, which frees unused memory. It publishes its own measurements in the runtime/metrics package.

The Prometheus library for Go can publish them with no copying by hand. The function below creates the registry, the list of measurements that /metrics will publish, and adds two collectors to it:

func newRegistry() *prometheus.Registry {
	reg := prometheus.NewRegistry()
	reg.MustRegister(
		collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}),
		collectors.NewGoCollector(collectors.WithGoCollectorRuntimeMetrics(
			collectors.MetricsGC, collectors.MetricsMemory, collectors.MetricsScheduler,
		)),
	)
	return reg
}

Full file: metrics.go, lines 47 to 56

The first collector measures the process as the system sees it: CPU, memory, open files. The second reads runtime/metrics. The WithGoCollectorRuntimeMetrics option picks three families: the garbage collector, memory and the scheduler, the part of the runtime that spreads work across CPU cores. These series land on the same /metrics as the request counters.

OpenTelemetry: a trace follows the request from end to end

On to traces. The repository uses OpenTelemetry for them, and for them only. The trace_id that slog writes above already comes from there: it is read from the span OpenTelemetry stored in the context.

OpenTelemetry also has its own logging system, which the repository does not use. In October 2026, the official documentation lists Go logs as "Release candidate": an almost final version, not yet declared stable. Traces and metrics are "Stable". So the repository keeps slog for logs.

OpenTelemetry splits its library into two halves. The first, which it calls the API (nothing to do with our invoice API), is used to create spans in the code. Creating those spans is called instrumenting the code. The second half, the SDK, decides which spans to keep and where to send them. Think of the first as a wall socket, and of the SDK as the appliance you plug into it.

Cloud-Native Observability with OpenTelemetry makes this split a principle, in lesson 14. "A user who instruments their code by using the API and does not configure the SDK will not see any telemetry produced by design." Telemetry means all the observation data: logs, metrics and traces. Instrumented code with no SDK plugged in runs normally, but nothing comes out, and that is intended. A library can therefore create its spans without forcing a tool on the people who use it.

The repository follows that split. Among the application files, only tracing.go imports the SDK, that is, loads it as a dependency. The storage layer only uses the API. Its Get function starts by opening a child span:

func (s *Store) Get(ctx context.Context, id string) (Invoice, error) {
	_, span := s.tracer.Start(ctx, "store.Get", trace.WithAttributes(attribute.String("invoice.id", id)))
	defer span.End()

Full file: store.go, lines 54 to 56

s.tracer.Start(ctx, …) receives the request's context, and that is what makes the store.Get span a child. Without that context, the span would start a separate trace. The defer span.End() line closes the span when the function returns, which sets its duration. The first result of Start, a new context, is ignored (_): no finer step follows.

Passing the trace number from one service to another

The span of the request itself comes from a wrapper provided by OpenTelemetry, otelhttp, placed around every route. A real application often has several services: separate programs that call each other. A trace that goes from one service to another must keep its number.

The calling service writes that number in an HTTP header, a line of text sent before the body of the request. The header is called traceparent. The W3C, the body that sets web standards, defines it in its Trace Context recommendation. An example from that recommendation: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01.

The header reads as four parts separated by dashes. They are written in hexadecimal, with the digits 0 to 9 and the letters a to f:

• 00, the format version;
• 32 characters, the trace_id;
• 16 characters, the caller's span_id, which will become the parent;
• 01, a flag, that is, a switch: 01 means "the caller records this trace", 00 the opposite.

Cloud-Native Observability with OpenTelemetry stresses this hand-off, in lesson 25. "Failing to propagate the trace ID across the system means that requests will be broken into multiple traces". A service that forgets the header cuts the trace into separate pieces.

Tail sampling: keep the useful traces, drop the rest

Keeping every trace is expensive: each request produces several spans to store. So you keep only part of them, and that selection is called sampling. The decision can happen at two moments, at the head or at the tail of the trace:

• head sampling decides at the very start, at the first span, at random: cheap, but blind;
• tail sampling decides at the end, on the whole trace: it keeps every error, but holds everything in memory until then.

Cloud-Native Observability with OpenTelemetry sums up the flaw of the first, in lesson 174. "An inherent problem with head sampling is that the decision is made before all the information is available." You choose before knowing whether the request will fail. The same book estimates that in production, under 10% of the data is often representative of the whole (lesson 179). That landmark dates from 2022, and it is not a rule.

In the repository, tail sampling happens in the Collector. The Collector receives every span and waits five seconds for each trace to be complete. It then applies three rules, and a trace is kept as soon as one rule says yes:

    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow
        type: latency
        latency:
          threshold_ms: 500
      - name: ten-percent-of-the-rest
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

Full file: otel-collector/config.yaml, lines 22 to 34

The three rules read like this:

• errors keeps the traces with a span marked as failed, which otelhttp does for every 500 response;
• slow keeps traces of 500 ms or more;
• ten-percent-of-the-rest keeps 10% of all the others, at random.

The expected arithmetic: 5% errors and 5% slow requests, so 10% of traces kept no matter what. Add 10% of the remaining 90%, so 9%. In total, about 19% kept and 81% dropped.

The end-to-end test measures the result. On the last run, Jaeger held 58 error traces out of 58 and 64 slow traces out of 64. So not a single error trace or slow trace was lost. The Collector had dropped 78% of the traces, close enough to the expected 81% for a random draw.

This test did not always measure the right thing. Its first version counted traces in Jaeger before the Collector had finished deciding. It sometimes found 43 error traces out of 53, and passed anyway, because it only required 80%. The current version waits until all 1,000 traces are decided, then requires every error and every slow trace.

One trap to know: head sampling below 100% is best not combined with tail sampling. Head sampling drops traces before the Collector knows they failed. In the repository, the API therefore keeps 100% of traces, and only the Collector sorts.

From the spike on the graph to the slow trace

That leaves the diamond from the opening diagram. Prometheus can attach the trace_id of the measured request to a duration measurement. That annotated measurement is called an exemplar, and Grafana shows it as a diamond on the duration curve. Grafana can open the trace with that number in Jaeger.

The code only sets an exemplar on requests the Collector is sure to keep. So the slow threshold is written a second time, on the Go side. The comment points to the Collector's slow rule:

// slowThreshold must match the "slow" latency policy (threshold_ms: 500) in
// otel-collector/config.yaml: such traces are always kept by the Collector.
const slowThreshold = 500 * time.Millisecond

Full file: metrics.go, lines 66 to 68

slowThreshold is 500 ms, like the threshold_ms: 500 of the slow rule above: it is the same threshold, written twice. The code below uses it when recording the duration:

	obs := m.duration.WithLabelValues(method, route)
	sc := trace.SpanContextFromContext(r.Context())
	// Exemplar only for requests the Collector always keeps (5xx or slow):
	// a fast 200 would mostly link to a trace dropped by tail sampling.
	if eo, ok := obs.(prometheus.ExemplarObserver); ok && sc.IsSampled() &&
		(status >= 500 || elapsed >= slowThreshold) {
		eo.ObserveWithExemplar(elapsed.Seconds(), prometheus.Labels{"trace_id": sc.TraceID().String()})
	} else {
		obs.Observe(elapsed.Seconds())
	}

Full file: metrics.go, lines 114 to 123

The if asks two questions. The first, obs.(prometheus.ExemplarObserver), checks that the histogram accepts exemplars. The second requires a recorded trace, then a 500 or higher error, or a duration of at least slowThreshold. If all of it holds, ObserveWithExemplar records the duration with the trace_id. Otherwise, Observe records it alone.

Why not an exemplar on every request? The code comments answer: a fast, successful request has only one chance in ten of being kept. Its exemplar would usually lead to a dropped trace.

What breaks: a Jaeger upgrade

The repository pins Jaeger at version 2.20.0: it asks for that exact version, not the latest. The reason is worth telling. The 2.21.0 release notes announce: "remove v1 http endpoints the ui no longer calls". Jaeger removed old HTTP addresses that its own interface no longer called.

Grafana 13.2 reads Jaeger through those old addresses. With Jaeger 2.22.0, /api/services returns 404, and Grafana reports Jaeger as unreachable (observed on October 28, 2026). An update bot, Dependabot, proposed this move to 2.22.0 on the repository. The tests GitHub runs on each proposal failed, at the "jaeger not ready" step of the end-to-end test.

The Go tests alone would have seen nothing: each piece works on its own. Only the end-to-end test, which makes the pieces talk to each other, sees the failure.

What the three books do not do: protect what you observe

Observing an API adds risks: one more address, copied data, one more header to trust. The three books cover few of them. The repository adds four protections.

Serve /metrics on an internal port

A server listens on a port, a number that acts as an entrance door on the machine: the API answers on port 8080. /metrics reveals the Go version, memory and the list of routes. None of that is the clients' business.

Modern REST API Development in Go shows, in lesson 64, that /metrics can be added wherever the API answers. "Wherever you expose your API, you can expose your metrics using the promhttp.Handler." In its code, /metrics is served on the same port as the API.

Mastering Go leaves the ports of its demo stack open (application, Prometheus, Grafana). The book settles for a warning, in lesson 134. Reaching them from another machine "is not very secure and should be blocked by a firewall". A firewall is a filter that decides what may enter the machine.

The repository opens two servers, on two ports. The first serves the API to everyone. The second only serves /metrics:

	reg := newRegistry()
	public := newServer(cfg.Addr, newHandler(log, tp, reg, cfg.DemoFaults))
	// /metrics lives on a separate, internal listener: it is not part of the
	// public API, it reveals internals, and scraping must not depend on the
	// public port's routing, auth or rate limits.
	internal := newServer(cfg.MetricsAddr, metricsHandler(reg))

Full file: main.go, lines 84 to 89

The code comment gives three reasons. /metrics is not part of the public API, and it reveals the program's internals. Finally, reading it must not depend on the public port's rules. Both servers share the same registry, reg, but not the same address. By default, the second listens on 127.0.0.1:9091. The address 127.0.0.1 means the machine itself: nobody else can reach it. The end-to-end test checks that /metrics returns 404 on the public port.

Bound every label

The route pattern already bounds the route label. That leaves the HTTP method, which the client picks freely. A request sent with an invented method, such as BREW, gets the label OTHER: that is the TestUnknownMethodIsBounded test. Every label whose value a client chooses thus needs a closed list of values.

Keep personal data out of logs and traces

An invoice holds the billed person's name and email. That is personal data, and it belongs neither in a log nor in a trace. Cloud-Native Observability with OpenTelemetry says it bluntly, in lesson 55, about PII, the data that identifies a person. "Whenever possible, save yourself the trouble and remove all PII from the telemetry." In plain words: when you can, strip all personal data from the telemetry, and spare yourself the trouble.

Modern REST API Development in Go does the opposite: one of its logs writes the IP address and the user, without a word about what they are. My review of System Programming Essentials with Go by Alex Rios lists what never goes in a log: passwords, personal data.

In the repository, the real defence is not logging those fields. The slog package looks for a LogValue method on any value it is given, and calls it instead of printing the raw value. The one on Invoice keeps three fields only:

func (i Invoice) LogValue() slog.Value {
	return slog.GroupValue(
		slog.String("id", i.ID),
		slog.Int64("amount_cents", i.AmountCents),
		slog.String("currency", i.Currency),
	)
}

Full file: store.go, lines 24 to 30

Neither Customer nor Email appears. The TestLogsCarryTraceIDAndNoPersonalData test creates an invoice in Alice's name and checks that neither "Alice" nor her email shows up in the logs.

A safety net also masks log fields whose name looks like email, password or token. That net only sees field names: a secret slipped into an error message still gets through.

Do not trust a client's traceparent

On the public port, the traceparent header comes from anyone. Trusting it opens two holes.

• A client could reuse the trace_id of another request. Our spans would land in a trace that is not theirs, and the logs of two requests would carry the same number.
• A client could send the 00 flag, "not recorded". Its 500 errors would then vanish from traces and exemplars.

The fix is one otelhttp option, the last one in this block:

	return otelhttp.NewHandler(instrument(mux, m, log), "http.server",
		otelhttp.WithTracerProvider(tp),
		otelhttp.WithPropagators(propagation.TraceContext{}), // W3C traceparent
		// Public port: an incoming traceparent is untrusted. Each request starts
		// a new trace (sampled by our own policy) and only links to the caller's.
		// (WithPublicEndpoint is gone from otelhttp v0.72; this is its equivalent.)
		otelhttp.WithPublicEndpointFn(func(*http.Request) bool { return true }),
	)

Full file: server.go, lines 55 to 62

WithPublicEndpointFn takes a small function, called for each request, that answers "this entry point is public". Here, it always returns true. The comment notes that the old option, WithPublicEndpoint, no longer exists in otelhttp v0.72, and that this one replaces it.

Each request then starts a new trace, sampled by our own rules. The client's trace is only kept as a link: a reference to it, not a parent. Between internal services that trust each other, you want the opposite: follow the header, to keep one trace across services.

Conclusion

The three books' mistakes each fit within a chapter. A badly chosen label, or code that contradicts its own text, shows up when you reread that chapter. The full list is in the sources, just below.

The failures hardest to see are elsewhere, in the seams between the signals. The 500 ms threshold is written twice, in the Go code and in the Collector. If the two values drift apart, each signal stays correct, and only the path from one to the other breaks, silently. The repository's first end-to-end test fell into a seam of the same kind. It counted traces before the Collector had finished sorting. Only a test that follows a trace_id from one end of the stack to the other, at the right moment, sees that kind of failure.