Alerting Strategy Framework

Alerts are interruptions, and every interruption must earn its place. This framework defines what deserves to wake someone and how to keep the rule set healthy over time.

How it works

Paging alerts come from SLO burn rate: a fast burn over a short window catches severe outages, while a slower burn over a longer window catches quiet degradation. Both windows must agree before paging to avoid flapping. Resource and cause metrics, such as CPU or queue depth, feed dashboards and low-urgency tickets unless they predict imminent user impact.

A useful test for any proposed page is to ask three questions. Is a user or the business harmed now or imminently? Does it require a human decision that software cannot make? Is there something specific the responder can do? If any answer is no, the signal belongs on a dashboard, in a ticket queue or in an automated remediation job instead. Applying this test during alert reviews steadily shrinks the paging surface while improving the quality of what remains.

Implementation steps

1. Inventory existing alerts and record last-fired date, owner and action taken.
2. Delete alerts that never fire or never lead to action.
3. Define urgency classes: page, ticket and notify, with response expectations for each.
4. Replace threshold alerts on symptoms with burn rate alerts per SLO.
5. Attach to each page a summary, impact, dashboard link and runbook link.
6. Add predictive alerts for hard limits, such as disk fill in four hours.
7. Review the top noisy alerts at every on-call handoff and tune or remove them.

Pitfalls

• Alerting on CPU or memory without evidence they cause impact.
• Missing for duration on rules, so a single scrape spike pages someone.
• Routing everything to one channel with no ownership.
• Alerts without runbooks, forcing responders to rediscover diagnosis.
• Silencing alerts permanently instead of fixing them.

Metrics to watch

• Pages per on-call shift and percentage actionable.
• Alert-to-incident ratio and incidents not caught by alerts.
• Time to detect for customer-impacting events.
• Alerts without an owner or runbook.
• Silence and mute counts per month.

Example Prometheus burn rate rule

groups:
  - name: checkout-slo
    rules:
      - alert: CheckoutErrorBudgetFastBurn
        expr: |
          (sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
            / sum(rate(http_requests_total{job="checkout"}[5m])) > 14.4 * 0.001)
          and
          (sum(rate(http_requests_total{job="checkout",code=~"5.."}[1h]))
            / sum(rate(http_requests_total{job="checkout"}[1h])) > 14.4 * 0.001)
        for: 2m
        labels:
          severity: page
          team: payments
        annotations:
          summary: "Checkout burning error budget at 14x"
          runbook: "https://runbooks.example.com/checkout/errors"

Key takeaways

• Page on symptoms; investigate with causes.
• Every page needs an owner, an action and a runbook.
• Use two windows to catch fast and slow burns without flapping.
• Prune continuously; an alert set decays without review.