Distributed Tracing Setup

Distributed tracing records the path of a request as a tree of timed spans. It answers the question that metrics and logs alone struggle with: where, exactly, did this request spend its time or fail?

How it works

When a request enters the system, the first service creates a trace identifier and a root span. Each outbound call carries the context in standard headers, such as traceparent, so downstream services attach their spans to the same trace. Spans record start time, duration, status, and attributes. Applications export them to a collector, which batches, filters, samples, and forwards them to a backend. Queues and background jobs carry context in message headers so asynchronous work remains connected.

Implementation steps

1. Adopt OpenTelemetry SDKs and the W3C Trace Context format
2. Enable auto-instrumentation for HTTP, database, and messaging libraries
3. Add custom spans around key business operations
4. Propagate context through message brokers and async jobs
5. Deploy collectors and configure sampling, such as tail sampling for errors
6. Inject trace identifiers into log lines to connect logs with traces

Common pitfalls

• Context lost at a proxy, queue, or thread boundary, producing orphan traces
• High cardinality or sensitive data in span attributes
• Sampling only a tiny percentage at the head and missing rare failures
• Too many trivial spans that bury the useful ones

Metrics to watch

• Percentage of requests with complete traces
• Orphan span rate
• Collector queue drops and export failures
• Telemetry volume and cost per service
• Time to identify root cause in incidents

Example configuration

# OpenTelemetry collector pipeline
receivers:
  otlp:
    protocols:
      grpc: {}
processors:
  batch: {}
  tail_sampling:
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
exporters:
  otlp:
    endpoint: tracing-backend:4317
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlp]

Trade-offs and adoption

Teams adopt this approach when requests traverse several services or queues; latency problems are hard to localize. It is a poor fit when a single monolith with simple call graphs already has adequate profiling, or when telemetry budget cannot support the added data volume. The central tensions are detail vs telemetry cost; head sampling simplicity vs tail sampling insight; auto-instrumentation convenience vs custom span quality. The main benefits are that it delivers: shows the full path and timing of a request; pinpoints slow or failing dependencies quickly; standards avoid vendor lock-in; links naturally with logs and metrics. The main costs to budget for are: instrumentation effort across many services; data volume and cost require sampling; context propagation breaks across unsupported boundaries; sampling can drop the very traces you need. Governance typically looks like this: platform team owns collector and conventions; service teams own span quality. Expect adoption cost to be medium: instrumentation, collectors, and backend, and a best fit for organizations with several interdependent services. Critics point out that traces rarely tell the full story without matching logs and metrics, and that cost and sampling trade-offs can leave incident-critical traces missing. Treat these concerns as design inputs, and revisit the decision as traffic, team size, and failure history change.

Key takeaways

• Standardize on OpenTelemetry and trace context propagation
• Cover async boundaries as carefully as synchronous ones
• Sample intelligently, keeping errors and slow traces
• Correlate traces with logs and metrics