Observability at Scale

Observability at scale is an economics and governance problem as much as a technical one. The goal is not to store everything but to retain the evidence that lets engineers answer new questions quickly.

How it works

Applications emit telemetry through OpenTelemetry to a fleet of collectors. The collectors enrich data with service and team metadata, drop noise, aggregate high-volume metrics, and sample traces by keeping errors and slow requests. Hot storage holds recent data for fast queries, while cheaper tiers keep aggregates and sampled data longer. Service level objectives with burn-rate alerts replace large numbers of cause-based alerts. Cost is attributed to teams, creating an incentive to keep cardinality and log volume under control.

Implementation steps

1. Standardize naming, labels, and context propagation across services
2. Deploy collector tiers for enrichment, filtering, and sampling
3. Define service level objectives and burn-rate alerts
4. Set cardinality limits and review high-cost metrics regularly
5. Implement tiered retention for logs, metrics, and traces
6. Report cost per team and set telemetry budgets

Common pitfalls

• Unbounded labels such as user identifiers in metrics
• Logging at debug level in production by default
• Dashboards created ad hoc and never maintained
• The telemetry pipeline itself lacking monitoring and capacity planning

Metrics to watch

• Telemetry cost per request or per service
• Active metric series and cardinality growth
• Query latency for common investigations
• Alert volume and actionable alert ratio
• Mean time to detect and diagnose

Example configuration

# Collector processors to control cost
processors:
  filter/drop_debug:
    logs:
      exclude:
        match_type: strict
        severity_texts: [DEBUG]
  attributes/limit:
    actions:
      - key: user.id
        action: delete
  tail_sampling:
    decision_wait: 10s
    policies:
      - { name: slow, type: latency, latency: { threshold_ms: 1000 } }

Trade-offs and adoption

Teams adopt this approach when telemetry bills or query times are growing out of control; many teams produce inconsistent logs, metrics, and traces. It is a poor fit when a small system where a single tool is sufficient, or when organizations without service ownership boundaries. The central tensions are data fidelity vs cost; central standards vs team autonomy; retention length vs query performance. The main benefits are that it delivers: predictable cost through sampling and retention tiers; consistent conventions speed diagnosis; alerts tied to objectives reduce noise; vendor portability through open standards. The main costs to budget for are: central platform requires dedicated investment; sampling and aggregation discard detail; governance can feel restrictive to teams; migration of existing instrumentation is slow. Governance typically looks like this: platform team sets standards and budgets; service teams own signal quality. Expect adoption cost to be high: platform engineering and migration effort, and a best fit for large organizations with many services. Critics point out that platforms often optimize for data collection instead of answering questions, and that cost controls can quietly remove the data needed for rare incidents. Treat these concerns as design inputs, and revisit the decision as traffic, team size, and failure history change.

Key takeaways

• Control cardinality and volume at the collector
• Alert on objectives, not on every symptom
• Tier retention and sample traces intelligently
• Attribute cost to the teams who generate it