Runbook Authoring Framework

A runbook is the shortest reliable path from an alert to a mitigated problem. It is written for a tired reader who has never seen this failure before.

How it works

Each runbook is a short document tied to a specific alert. It starts with symptoms and impact, offers a handful of diagnostic checks ordered by likelihood, then lists mitigation actions, each with the exact command and the expected output. It ends with how to confirm recovery and who to escalate to. Runbooks live in the same repository as the service, so changes to behaviour update the procedure in the same pull request.

Good runbooks follow the way people actually work during incidents: they skim, they copy commands and they distrust anything that looks out of date. Put the most likely cause first, show the expected output next to each command, and keep the page short enough to read on a phone. Treat the runbook as a product with users, and gather feedback after every use by asking responders what was missing, unclear or wrong, then fix it within days while the experience is fresh.

Implementation steps

1. List paging alerts and prioritise those that fired most in the last quarter.
2. Draft a runbook per alert using a fixed template and have a non-author execute it.
3. Use numbered steps, one action each, with copyable commands and safe defaults.
4. Mark dangerous steps clearly and state the rollback of each action.
5. Link the runbook URL in the alert annotation.
6. Add a review date and owner, and flag runbooks unused or stale for a year.
7. After each incident, update the runbook and consider automating the fix.

Pitfalls

• Writing background essays instead of actions.
• Commands that depend on local setup or personal credentials.
• No verification step, so responders cannot tell if the fix worked.
• Documentation separate from code, which drifts.
• Treating the runbook as complete and skipping automation.

Metrics to watch

• Percentage of paging alerts with a linked runbook.
• Runbook usage and median time to mitigate with versus without one.
• Last-verified age per runbook.
• Number of runbook errors reported during incidents.
• Count of runbook steps converted to automation.

Example runbook template

# Runbook: HighQueueLag (orders-consumer)

Owner: orders-team   Last verified: 2026-08-02

## Meaning and impact
Consumers are more than 10 minutes behind. Orders appear late to customers.

## Quick checks
1. Open dashboard: https://grafana.example.com/d/orders
2. Check consumer pods: kubectl get pods -n orders -l app=consumer
3. Look for errors: kubectl logs -n orders -l app=consumer --tail=100

## Mitigation
1. If pods are crash-looping, roll back: kubectl rollout undo deploy/consumer -n orders
2. If healthy but slow, scale: kubectl scale deploy/consumer -n orders --replicas=12

## Verify
Lag below 1 minute for 10 minutes.

## Escalate
Page orders-secondary if not recovered in 30 minutes.

Key takeaways

• One alert, one runbook, one clear owner.
• Commands and expected output beat prose.
• Test with someone who did not write it.
• A repeated manual fix is a candidate for automation.