Stackbook Logo
← Back to System Architectures

Observability System Design

Observability is not monitoring. Monitoring is observing known knowns. Observability is discovering unknown unknowns.

4 min read

Observability is not monitoring. Monitoring is observing known knowns. Observability is discovering unknown unknowns.

The Three Pillars of Observability

Metrics: Numeric Data Over Time

  • What: Aggregated measurements over time (CPU usage, request rate, error rate)
  • Purpose: Trend detection, alerting, capacity planning
  • Typical tools: Prometheus, Graphite, Datadog
  • Key question: "How is the system behaving over time?"

Logs: Individual Events

  • What: Discrete records of things that happened (timestamp + structured data)
  • Purpose: Debugging, audit, compliance
  • Typical tools: ELK Stack, Loki, Splunk, Datadog Logs
  • Key question: "What happened at this specific time?"

Traces: Request Journeys

  • What: Path of a single request through the system (services visited, timings, errors)
  • Purpose: Debugging distributed systems, performance analysis
  • Typical tools: Jaeger, Zipkin, OpenTelemetry
  • Key question: "Which services did this request go through, and where was it slow?"

Designing for Observability

From the Start

Observability should be designed from day one, not added later. It's much harder to bolt on after the fact.

Key decisions from the start:

  • What metrics to track (don't track everything)
  • What log structure to use (structured logs JSON preferred)
  • What tracing to implement (OpenTelemetry instrumentation from day one)
  • Where data goes (centralized system, not scattered across services)
  • Who owns what (clear responsibility assignment)

Metrics That Matter

Don't track everything. Focus on metrics that answer business and operations questions:

  • Request rate: Traffic volume
  • Error rate: What's failing?
  • Latency percentiles: p95, p99 response times
  • Service health: Up/down status, health check results
  • Business metrics: Revenue, signups, active users (if applicable)

Avoid: Tracking 100+ metrics with no clear purpose. Start with 5-10 key metrics and add more as needed.

Log Structure

Structured logs (JSON preferred) over free-form text. Include at minimum:

{
  "timestamp": "ISO8601",
  "level": "error|info|warn|debug",
  "service": "service-name",
  "instance": "instance-id-or-hostname",
  "trace_id": "correlation-id",
  "span_id": "within-service-span-id",
  "message": "human-readable description",
  "key_value_pairs": {"field": "value"}
}

Avoid: Free-form log messages that are difficult to query and correlate.

Tracing Implementation

OpenTelemetry is becoming the standard. Instrument from day one.

Key decisions:

  • Sampling rate (don't sample 100% in production if you'll have too much data, but don't sample so much you lose insights)
  • Context propagation (ensure trace IDs flow through all services and hottest paths)
  • Sampling decisions at edge vs. core (different strategies may be appropriate)

The honest truth: Tracing is invaluable for debugging distributed systems but adds latency and complexity. Start with critical paths, expand later.

The Dashboard

Create dashboards that answer common questions:

  • Is the system healthy? (uptime, error rates)
  • Is it fast? (latency percentiles)
  • Is it scaling? (request rates, capacity)
  • Is it secure? (authentication failures, unusual patterns)

One dashboard per team that answers their most common questions. Don't create one dashboard to rule them all—different teams have different concerns.

Alerting

Alerts should answer: "Do I need to take action now?"

Good alerts:

  • Error rate exceeds threshold for 5 minutes
  • Latency exceeds p99 threshold for 10 minutes
  • Service down (liveness check fails)

Bad alerts:

  • "CPU usage exceeds 80%" (might be normal for your workload)
  • "Memory usage exceeds 500MB" (might be normal for your instance size)
  • "Fewer than 100 requests per minute" (might be off-peak)

Alert fatigue is real. Start with fewer alerts. Add more as needed.

The Honest Truth

Observability is a journey, not a destination. Start with the basics and mature over time. The goal isn't perfect observability—that's impossible—but sufficient observability to do your job effectively.

Published June 1, 2026 · 13 min read