Game Days

Overview

A game day is a scheduled exercise in which a team deliberately simulates or injects a failure and practices responding. It differs from automated chaos experiments by focusing on people and process as well as software: can responders find the right dashboard, follow the runbook, reach the right person, and decide quickly?

How it works

A game day has three phases.

Plan     scenario, hypothesis, steady-state metrics, blast radius,
         abort criteria, roles, comms plan, rollback steps
Run      brief -> baseline check -> inject fault -> respond ->
         observe -> abort or restore -> debrief
Review   timeline, what held, what broke, action items with owners

An example hypothesis: if the primary database fails, the replica is promoted within five minutes and error ratio stays below one percent. The injector then stops the primary. The observer notes detection time, first action, mitigation time, and surprises such as an expired credential or an alert that never fired. Results update runbooks, alerts, and architecture.

Start small: tabletop discussions, then staging, then production at low traffic with a narrow scope. Increase scope as confidence grows. Include dependencies, such as what happens when the paging tool itself is unavailable.

Pitfalls

• Skipping abort criteria. Define in advance the metric thresholds that stop the exercise.
• Running only the happy scenario. Include partial failures and slow dependencies, which are harder than clean crashes.
• Treating the exercise as a test of people. Keep it blameless so participants report confusion honestly.
• Recording findings without owners or dates.

Key takeaways

• State a hypothesis and measure against it.
• Limit blast radius and define abort conditions.
• Practice the human response, not only the failover.
• Convert every gap into a tracked fix.