Static Stability

Overview

Amazon popularized static stability as a design goal. A statically stable system stays in its steady state when something it depends on breaks, rather than needing to take action to survive.

How it works

The data plane serves traffic and the control plane configures it. Failure of the control plane must not affect serving.

control plane (create, scale, config)   <- may be down
        | push, cached locally
data plane (serve requests)             <- keeps running

For capacity, provision so that losing one of three zones leaves the other two able to carry full load, which means running about 150 percent of needed capacity in total. If a zone fails, no new instances need to be launched, so there is no dependence on the compute API during the incident. Health-based routing shifts traffic automatically.

Pitfalls

• A small call to a configuration or identity service per request quietly reintroduces a dependency. Audit runtime calls.
• Cached credentials and configs need a long enough lifetime to outlast an incident, with explicit alerting when they age.
• Static capacity decisions cost money, so size them from real failure domains, not guesses.
• Fail-open vs fail-closed choices on stale data must be explicit, especially for security configuration.
• Test by actually disabling the control plane in a game day.

Key takeaways

• Keep the data plane independent from the control plane.
• Provision headroom ahead of failure instead of scaling during it.
• Cache the last known good configuration.
• Verify the property with fault injection.