reliability-opsemerging · high operational burden

Chaos Engineering

Also known as: chaos-testing, failure-injection, game-day

Intent

Proactively inject failures to discover weaknesses before they cause outages, building confidence in system resilience.

Problem

Systems fail in unexpected ways. Post-mortems reveal gaps. Chaos engineering finds gaps before production does.

Forces

•Failure is inevitable; preparation is optional
•Staging doesn't replicate production complexity
•Need safe, controlled failure injection
•Must measure impact, not just inject chaos

Solution

✓ When to Use

•High availability targets (99.99%+)
•Post-incident: prevent recurrence
•Before major launches or migrations
•Regular cadence: monthly game days

✗ When Not to Use

•No observability (blind injection)
•No rollback/autostop mechanism
•Production not resilient enough (will cause real outage)
•Team not bought in (fear-driven)

Pros

+Finds real weaknesses: config, code, architecture
+Improves incident response: practice under controlled conditions
+Builds confidence: 'we know it survives X'
+Cultural: failure is normal, not shameful

Cons

−Risk: can cause real outage if not controlled
−Operational overhead: planning, execution, cleanup
−False confidence: tested scenarios ≠ all scenarios
−Requires mature observability and runbooks

Cost Profile

Infrastructure

Low — tools, sometimes dedicated instances

Operational

High — planning, execution, follow-up

Cognitive

Medium — experimental thinking, hypothesis design

Failure Modes

Uncontrolled blast: experiment takes down production
No observability: can't measure impact
Rollback fails: experiment becomes incident
Team burnout: too frequent, not actionable
Compliance: chaos in regulated env requires approval

Real-World Examples

Alternatives

game-days
fire-drills
fault-injection-testing
resilience-testing

Related Patterns

game-day
observability
circuit-breaker
bulkhead
runbook

Competency Domains

reliability opsdeploymentdistribution communicationeconomics evolution