Burn-Rate Alerting

Overview

Burn-rate alerting, described in the Google SRE Workbook, asks a better question than a threshold: at the current rate of failure, how soon will the error budget run out? It links paging directly to SLO impact.

How it works

Burn rate is error ratio divided by the allowed error ratio (1 minus SLO). For a 99.9 percent SLO, a 1.44 percent error ratio is a burn rate of 14.4, which exhausts 2 percent of a 30-day budget in one hour.

Severity  Burn rate  Long window  Short window  Budget used
page      14.4       1h           5m            2 percent
page      6          6h           30m           5 percent
ticket    1          3d           6h            10 percent

alert when:
  burn(long_window) above rate  AND  burn(short_window) above rate

The long window proves the problem is sustained and meaningful; the short window proves it is still happening, so the alert resets within minutes of a fix. Recording rules precompute error ratios for each window to keep queries cheap.

Pitfalls

• Applying it to low-traffic endpoints. At a few requests per hour, one failure produces a huge ratio. Aggregate across endpoints, add synthetic traffic, or use a ticket-only policy.
• Setting only one window, which causes either flapping or slow clearing.
• Alerting on every service dimension. Page on user-facing aggregates; keep per-shard burn as dashboards.
• Forgetting the multiplier depends on the SLO window; changing from 30 to 28 days changes the budget fractions.

Key takeaways

• Page on budget consumption speed, not raw error rate.
• Use paired long and short windows.
• Pick several tiers: page for fast burn, ticket for slow burn.
• Aggregate or synthesize traffic for sparse services.