Chaos Game Day Framework

A game day is a planned rehearsal of failure. It tests technology, monitoring and people together, which unit tests and design reviews cannot do.

How it works

The team picks a failure scenario from past incidents or architecture risk reviews. A facilitator runs the session with defined roles: an injector who applies the fault, observers who watch dashboards, and responders who act as they would on call. Results are compared to the written hypothesis, and any surprise becomes a finding.

Maturity grows in stages. Early game days are discussion-based tabletop exercises where the team walks through a scenario on a whiteboard and checks the runbooks. Next come announced experiments in staging, then announced experiments in production with a small blast radius, and eventually automated continuous experiments that run unannounced within strict guardrails. Each stage builds the trust and tooling needed for the next, and skipping stages tends to end in a failed experiment that sets the whole programme back.

Implementation steps

1. Choose one scenario and write a hypothesis, such as: if one zone is lost, error rate stays under 1 percent and failover completes in two minutes.
2. Define steady state using existing SLIs and confirm it holds before starting.
3. Set blast radius, duration and abort criteria, plus a tested rollback of the fault.
4. Notify stakeholders and support teams, and avoid deploys during the window.
5. Run the injection while responders follow the incident process, with or without advance warning.
6. Capture timeline, detection time, response gaps and unexpected behaviour.
7. Hold a short review, file actions with owners and repeat the experiment after fixes.

Pitfalls

• Experimenting without a hypothesis, so nothing is learned.
• Starting in production with broad scope.
• Forgetting an abort button, or one that is slower than the fault.
• Treating findings as embarrassing rather than useful.
• Running a single exercise and never repeating it.

Metrics to watch

• Time to detect and time to mitigate during the exercise.
• Number of findings and share closed within thirty days.
• Percentage of critical dependencies with at least one tested failure.
• Experiments aborted due to guardrail triggers.
• Hypotheses confirmed versus disproved.

Example experiment definition

experiment: database-primary-failover
environment: production
hypothesis: "Replica promotion completes within 90s and checkout error ratio stays below 2%"
steady_state:
  sli: checkout_success_ratio
  threshold: 0.995
blast_radius:
  region: eu-west-1
  traffic_percent: 10
duration: 20m
fault:
  type: terminate-instance
  target: "role=db-primary"
abort_conditions:
  - "checkout_success_ratio below 0.98 for 2m"
  - "facilitator calls stop"
roles:
  facilitator: sre-lead
  injector: platform-engineer
  observers: [payments, support]

Key takeaways

• Write the hypothesis first; the result is the learning.
• Limit blast radius and rehearse the abort.
• Test responders and runbooks, not only infrastructure.
• Repeat after fixes to prove they work.