Designing for Observability

Monitoring tells you that something is wrong. Observability lets you ask why, including about failures you did not predict. In a distributed system, where a single request crosses a dozen services, you cannot debug by logging into a server and reading files. Observability has to be designed in from the start, the same way you design for security or testability, because retrofitting it during an outage is painful and incomplete.

The Three Signals and What Each Is For

Logs, metrics, and traces are complementary, not interchangeable.

| Signal | Answers | Cost profile | Weakness | |---|---|---|---| | Metrics | Is it healthy? How much, how fast? | Cheap, aggregated | Little detail, low cardinality only | | Logs | What exactly happened here? | Grows with volume | Hard to correlate without discipline | | Traces | Where did this request spend its time? | Moderate, usually sampled | Needs propagation everywhere |

A typical investigation flows from metrics (an alert fires), to traces (which service is slow), to logs (the specific error and its context). If the three cannot be linked with a shared identifier, each step becomes guesswork.

Make Logs Structured and Correlated

Free-text logs are for humans; structured logs are for machines, and at scale machines do the reading. Emit JSON with consistent field names:

{
  "ts": "2026-06-01T10:15:32Z",
  "level": "error",
  "service": "checkout",
  "trace_id": "4bf92f3577b34da6",
  "user_id": "u_1842",
  "msg": "payment authorization failed",
  "error_code": "CARD_DECLINED",
  "duration_ms": 412
}

Include a trace or request identifier in every line, and propagate it across service boundaries through headers such as the W3C traceparent. Never log secrets or raw personal data. Decide log levels deliberately: if everything is info, nothing is findable, and if debug logs run in production without sampling, your bill will tell you.

Metrics That Reflect User Experience

Start with the golden signals from Google's SRE practice: latency, traffic, errors, and saturation. For request-driven services, the RED method (rate, errors, duration) is a compact version. For resources such as queues and databases, use USE (utilization, saturation, errors).

Watch two traps:

• Averages hide pain. Track percentiles such as p95 and p99. A mean of 200 ms can coexist with a p99 of eight seconds.
• Cardinality explosions. Labeling a metric with user ID or request ID creates millions of time series and can take down your metrics store. Put high-cardinality detail in logs and traces, and keep metric labels to bounded sets such as route, status class, and region.

Distributed Tracing Needs Architecture Support

A trace is a tree of spans linked by context propagation. It only works if every hop forwards the context, including through message queues and background jobs. Standardize on OpenTelemetry instrumentation so you can change backends without re-instrumenting. Because tracing every request is expensive, sample thoughtfully: head-based sampling is simple, but tail-based sampling lets you keep all slow or failed traces and drop most healthy ones.

Alert on Symptoms and Define SLOs

Alerting on every CPU spike trains people to ignore pages. Instead, define service level objectives on what users feel, for example "99.9 percent of checkout requests succeed within 500 ms over 30 days". The gap between 100 percent and your target is an error budget. Alert when the budget is burning too fast, and page only for conditions that need a human right now.

• Page on symptoms (users failing), investigate causes (disk filling) on dashboards or tickets.
• Attach a runbook link and a dashboard to every alert.
• Review alert quality regularly, and delete alerts nobody acts on.

Design Choices That Make Systems Observable

Observability is an architecture concern, not just tooling:

• Give every request an identifier at the edge and carry it everywhere.
• Emit business events such as "order placed" alongside technical metrics, so you can connect incidents to revenue.
• Add health and readiness endpoints that reflect dependency state, not just process liveness.
• Make state inspectable: expose versions, feature flag values, and configuration hashes.
• Use feature flags and canary releases, and annotate dashboards with deploy markers so changes can be tied to regressions.

Key Takeaways

• Observability is the ability to explain unknown failures; design for it up front.
• Use metrics to detect, traces to localize, and logs to explain, all linked by a shared identifier.
• Prefer structured logs, percentile latencies, and bounded metric labels.
• Standardize on OpenTelemetry and propagate context through every boundary, including queues.
• Alert on user-facing SLO burn, not on every internal fluctuation.