Failover Runbook Design

A failover runbook is the written procedure for moving service from a failed component to a healthy one. Its quality is judged at the worst moment: tired responder, partial information, customers waiting. Design for that reader.

How it works

A strong runbook has a fixed anatomy so responders know where to look.

• Trigger and scope: the alert or symptom that points to this runbook, and what it covers.
• Decision section: criteria for failing over now, waiting, or escalating, tied to RTO and RPO. For example, fail over if the primary has been unreachable for five minutes and replication lag is under thirty seconds.
• Steps: numbered, one action each, with the command, expected output, and who performs it.
• Verification: objective checks proving traffic and data are healthy on the new primary.
• Rollback and failback: how to undo, and how to return to the original topology safely.
• Communication: who to notify and status page wording.

Implementation steps

1. Start from the architecture and list each failure scenario: instance, zone, database primary, dependency.
2. Write the decision tree first; most delay comes from deciding, not typing.
3. Convert manual commands into scripts with a dry-run mode, and call the script from the runbook.
4. Add guard rails, such as refusing promotion when replication lag exceeds the RPO without confirmation.
5. Keep the runbook in version control next to the code and require updates in changes that alter topology.
6. Run a game day each quarter where someone unfamiliar follows it literally, and record every deviation.
7. Fix gaps and timestamp the last successful rehearsal at the top of the document.

Common pitfalls

• Steps that assume access or credentials that on-call engineers do not have.
• Missing fencing, leading to two primaries and split brain.
• Failing over without checking data freshness.
• No failback plan, leaving the system in a fragile topology for weeks.
• Walls of prose; responders skim, so use short imperative lines.

Metrics to watch

• Measured recovery time in rehearsals versus the RTO.
• Data loss observed versus the RPO.
• Days since each runbook was last rehearsed.
• Number of deviations or fixes found per game day.
• Time from alert to decision.

Example runbook skeleton

## DB primary failover
Last rehearsed: 2026-09-12
Trigger: alert db-primary-unreachable for 5m
Decide: replication lag below 30s? yes -> continue, no -> escalate to DBA
1. Fence old primary: ./fence.sh --host db-a --dry-run
2. Promote replica: ./promote.sh --host db-b
   Expect: "promotion complete, writable=true"
3. Repoint app: ./set-endpoint.sh db-b
4. Verify: write probe succeeds, error rate under 1% for 5m
Rollback: ./set-endpoint.sh db-a only if db-b rejects writes

Key takeaways

• Put the decision criteria in the runbook, not just the commands.
• Script the risky steps and add guard rails.
• Always include verification, fencing, and failback.
• Rehearse regularly; an untested runbook is a hypothesis.
• Keep it short, imperative, and in version control.