Stackbook Logo
observabilityestablished · high operational burden

Observability Patterns

Also known as: observability, monitoring, logging, metrics, tracing, sli-slo

Intent

Build comprehensive observability: three pillars + SLI/SLO, alerting, debugging workflows, cost control.

Problem

Systems are black boxes. Dashboards show symptoms, not causes. Alerts are noisy. Debugging takes hours. Need systematic observability.

Forces

  • •Complex distributed systems: hundreds of services
  • •Unknown unknowns: can't predict all failure modes
  • •Need to ask arbitrary questions in production
  • •Signal must be affordable (sampling, cardinality)

Solution

✓ When to Use

  • •Any production distributed system
  • •Incident response: MTTR targets
  • •Capacity planning: saturation signals
  • •Release validation: error rate, latency

✗ When Not to Use

  • •Simple monolith (logs + basic metrics sufficient)
  • •Team not ready for instrumentation investment
  • •Cost-sensitive: observability can be expensive

Pros

  • +Debug arbitrary questions: 'why is p99 high for user X?'
  • +Proactive: alerts on SLO burn rate, not symptoms
  • +Capacity: utilization trends, saturation prediction
  • +Release confidence: canary metrics, error budgets

Cons

  • −Cost: high cardinality, trace volume, log volume
  • −Instrumentation effort: auto helps, manual needed
  • −Cardinality explosion: user_id, request_id as labels
  • −Sampling gaps: rare errors missed

Cost Profile

Infrastructure

Medium-High — storage, query, retention

Operational

High — instrumentation, dashboards, alerts

Cognitive

Medium — three pillars, correlation

Failure Modes

  • High cardinality → storage explosion, query timeout

  • Sampling misses rare errors (tail-based mitigates)

  • Clock skew → trace ordering wrong

  • Missing context propagation → fragmented traces

  • Alert fatigue: too many, not actionable

Real-World Examples

Alternatives

  • logs-only
  • metrics-only
  • apm-only
  • profiling

Related Patterns

  • structured-logging
  • distributed-tracing
  • metrics
  • sli-slo
  • alerting
  • open-telemetry
  • exemplars

Competency Domains

reliability opsdistribution communicationdata stateeconomics evolutiondeployment