Problem
A dashboard shows p99 latency doubled, but aggregated metrics give no way to see which requests were slow. Searching a trace store by time window finds thousands of irrelevant traces.
observabilityestablished · medium operational burden
Exemplars and Trace-Metric Linking
Also known as: metric-exemplars, metrics-to-traces
Intent
Attach sample trace identifiers to metric data points so an engineer can jump from a latency spike straight to a representative slow trace.
Forces
- •Metrics are aggregated and lose individual request detail
- •Traces are detailed but hard to find without a pointer
- •Storing a trace ID per sample is costly
- •Linking must not add high-cardinality labels
Solution
Overview
Metrics tell you that something is wrong; traces tell you why. Exemplars connect them. An exemplar is a small piece of metadata attached to a metric sample, normally a trace ID, that points to one concrete request that contributed to the value.
How it works
On each observation the instrumentation library looks at the active span. If sampled, it stores the trace ID with the histogram bucket the value falls into.
# OpenMetrics exposition
http_request_duration_seconds_bucket{le="1.0"} 2847 # {trace_id="4bf92f35"} 0.87 1700000000.1
http_request_duration_seconds_bucket{le="2.5"} 2851 # {trace_id="a1c3e8d2"} 2.31 1700000012.4
The metrics backend keeps the latest exemplar per bucket per series and returns them with queries. The dashboard overlays dots on the latency graph; clicking one opens the trace. The workflow becomes: notice the spike, click an exemplar from the high bucket, read the trace to find the slow downstream call.
Why not use trace ID as a label? A label creates a new time series for every value, which is unbounded cardinality. An exemplar sits beside the sample and does not multiply series.
Pitfalls
- Mismatch with sampling. If only one percent of traces are stored, most exemplars point nowhere. Use tail sampling that keeps slow and failing traces, or record exemplars only for retained traces.
- Missing support in the scrape path. Some pipelines strip exemplars; verify end to end.
- Expecting exemplars to represent the distribution. They are samples, not statistics.
- Applying them to counters of unrelated events with no request context.
Key takeaways
- Exemplars link a metric point to a real trace without adding labels.
- They need tracing, histograms, and a backend that stores exemplars.
- Align trace sampling so linked traces exist.
- Treat them as clues, then confirm with more data.
✓ When to Use
- •Services with both histograms and distributed tracing
- •Investigating tail latency or error spikes
- •Teams that want a metrics-first debugging workflow
✗ When Not to Use
- •Systems without tracing
- •Metrics that are not histograms or counters with request context
- •Cases where tail sampling drops the linked traces
Pros
- +Fast jump from symptom to example
- +No extra labels, so no cardinality growth
- +Works with standard OpenMetrics tooling
Cons
- −Requires tracing and metrics integration
- −Exemplar storage adds some memory
- −Links break if traces are not retained
Cost Profile
Infrastructure
Low - small extra storage
Operational
Medium - integration across tools
Cognitive
Medium - needs both pillars mature
Failure Modes
Exemplar points to a trace dropped by sampling, giving a dead link
Trace ID added as a label instead, exploding cardinality
Clock or unit mismatch makes exemplars appear in the wrong bucket
Real-World Examples
Alternatives
Related Patterns
- distributed-tracing
- observability
- correlation-ids
Competency Domains
reliability opsdata state