Problem
System is a black box. Dashboards show symptoms, not causes. Debugging takes hours.
observabilityestablished · high operational burden
Observability (Three Pillars)
Also known as: three-pillars, metrics-logs-traces, otel
Intent
Understand system internal state through external outputs: metrics (what), logs (why), traces (where).
Forces
- •Complex distributed systems: hundreds of services
- •Unknown unknowns: can't predict all failure modes
- •Need to ask arbitrary questions in production
- •Signal must be affordable (sampling, cardinality)
Solution
✓ When to Use
- •Any production distributed system
- •Incident response: MTTR targets
- •Capacity planning: saturation signals
- •Release validation: error rate, latency
✗ When Not to Use
- •Simple monolith (logs + basic metrics sufficient)
- •Team not ready for instrumentation investment
- •Cost-sensitive: observability can be expensive
Pros
- +Debug arbitrary questions: 'why is p99 high for user X?'
- +Proactive: alerts on SLO burn rate, not symptoms
- +Capacity: utilization trends, saturation prediction
- +Release confidence: canary metrics, error budgets
Cons
- −Cost: high cardinality, trace volume, log volume
- −Instrumentation effort: auto helps, manual needed
- −Cardinality explosion: user_id, request_id as labels
- −Sampling gaps: rare errors missed
Cost Profile
Infrastructure
Medium-High — storage, query, retention
Operational
High — instrumentation, dashboards, alerts
Cognitive
Medium — three pillars, correlation
Failure Modes
High cardinality → storage explosion, query timeout
Sampling misses rare errors (tail-based mitigates)
Clock skew → trace ordering wrong
Missing context propagation → fragmented traces
Alert fatigue: too many, not actionable
Real-World Examples
Alternatives
- logs-only
- metrics-only
- apm-only
- profiling
Related Patterns
- structured-logging
- distributed-tracing
- metrics
- sli-slo
- alerting
- open-telemetry
Competency Domains
reliability opsdistribution communicationdata stateeconomics evolutiondeployment