Failure Domain Design

Designing for failure is designing for the reality that things will break.

What Are Failure Domains?

A failure domain is a group of components where a failure affects all of them. Designing failure domains means controlling the blast radius.

Network Failure Domains

Availability zone: Single AZ failure takes down all resources in that AZ
Region: Single region failure takes down all resources in that region
Multi-region: Design for region-level failures

Service Failure Domains

Single service: One service goes down, others stay up
Service group: Group of related services go down together
Full system: Complete failure (should be extremely rare)

Data Failure Domains

Single database: All data and traffic dependent on one database
Database cluster: Cluster failure affects all dependent services
Database region: Region-level database failure

Design Principles

Principle 1: Isolate Failure Domains

Design so that failures are contained. Use:

Different AZs for different service instances
Different regions for disaster recovery
Circuit breakers to prevent cascading failures
Bulkheads to isolate resource exhaustion

Principle 2: Assume Failure

Design every component as if it will fail. Not "if it might fail, but when it will fail."

The mindset shift: From "how do we prevent all failures?" to "how do we ensure no single failure is catastrophic?"

Principle 2: Graceful Degradation

When parts of the system fail, the system should remain operational at reduced capacity. Prioritize which features/functions are most important and ensure they stay available.

Example: An e-commerce site might:

Keep product browsing available
Degrade search functionality
Degrade product review display
Decrease checkout capacity but keep it functioning

Principle 3: Automatic Recovery

Design for automatic recovery where possible. Health checks, auto-scaling, service mesh retries and timeouts.

Why manual recovery is insufficient: Humans are slow. Automated systems can react in seconds.

Principle 4: State Management

State is the enemy of availability. Design stateless components where possible. Use external state stores (databases, caches) that are designed for high availability.

The challenge: Some state must be stateful (user sessions, shopping carts). Design for this with session replication, distributed caches, or token-based stateless auth.

Patterns for Resilience

Pattern: Circuit Breaker

Prevent cascading failures. When a downstream service is unhealthy (based on health checks, error rate, latency), stop calling it.

Implementation: Most languages have libraries. Service meshes (Istio, Linkerd) provide this infrastructure.

Pattern: Bulkhead

Isolate resources so failure in one area doesn't exhaust resources elsewhere.

Example: Connection pools per service rather than a shared pool. Thread pools per application rather than shared thread pool.

Pattern: Retry with Backoff

Transient failures should be retried, but with exponential backoff to avoid overwhelming the failing service.

Key: Distinguish between transient failures (network hiccups, temporary overload) and permanent failures (invalid data, deleted resources).

Pattern: Timeout Settings

Set appropriate timeouts for all external calls. Shorter timeouts fail fast. Longer timeouts wait too long.

The balance: Timeouts short enough to fail fast, long enough to not drop legitimate requests under normal load.

Pattern: Health Checks

Regular health checks determine service health. Used by load balancers, auto-scalers, circuit breakers.

Three levels of health checks:

Liveness: Is the process alive? If no, restart.
Readiness: Can the service serve requests? If no, remove from load balancer.
Dependence: Can the service reach its dependencies? If no, circuit breaker should trigger.

Patterns for Resilience (continued)