Golden Signals Monitoring

Overview

The four golden signals come from Google SRE practice. If you can only watch four things about a user-facing system, watch these: latency, traffic, errors, and saturation. Together they answer whether users are being served, how much, how well, and how close the system is to its limit.

How it works

Each signal has a precise definition:

• Latency: time to serve a request, tracked as p50, p95, and p99. Record failed requests separately, because a fast error can make latency look healthy.
• Traffic: demand, such as requests per second or sessions.
• Errors: explicit failures (HTTP 500), implicit failures (200 with wrong content), and policy failures (response slower than the SLO threshold).
• Saturation: utilization of the scarcest resource, such as CPU, memory, connection pool, or queue depth. Alert before 100 percent, since latency degrades early.

rate(http_requests_total[5m])                      # traffic
sum(rate(http_requests_total{code=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))             # errors
histogram_quantile(0.99, rate(http_request_duration_bucket[5m]))  # latency
max(db_pool_in_use / db_pool_size)                 # saturation

Page on error ratio and tail latency. Use saturation for warnings and capacity planning.

Pitfalls

• Using the mean for latency. One slow percent of requests can hurt many users while the average looks fine.
• Mixing successful and failed latencies. Fast failures pull the percentile down.
• Picking saturation by habit (CPU) when the real constraint is a thread pool or a downstream rate limit.
• Treating the signals as a replacement for SLOs. They are inputs; an SLO sets the target.

Key takeaways

• Four signals give broad coverage for request-serving services.
• Standardize names and dashboards so any engineer can read any service.
• Alert on symptoms; use other signals to diagnose.
• Measure tail latency and split success from failure.