Designing for Failure First

Most designs begin with the happy path: a request arrives, services cooperate, data is returned. Production does not behave that way. Disks fill, dependencies slow down, deployments overlap and networks partition. If you design failure behavior last, you end up with a system whose real behavior was never chosen by anyone.

Failure is a design input

Treat failure modes as requirements. For every dependency in a diagram, ask three questions: what happens when it is slow, what happens when it is down, and what happens when it returns wrong answers? Slow is usually the worst of the three, because a hung call holds threads, connections and memory while appearing healthy.

A lightweight way to capture this is a failure table next to the architecture diagram:

| Dependency | Failure | User impact | Planned behavior | |---|---|---|---| | Payment gateway | Timeout | Cannot pay | Retry once, then queue and show pending | | Recommendation service | Down | Less relevant page | Serve cached or static list | | Primary database | Failover, 30s | Writes fail | Reads from replica, writes return 503 with retry hint | | Email provider | Rate limited | Delayed receipt | Outbox with backoff |

Notice that the planned behaviors differ. Not every dependency is equally critical, and degrading gracefully is only possible once you have decided what is essential.

Classify what must not fail

Separate your capabilities into tiers:

Critical path. Login, checkout, core reads. These get redundancy, tight timeouts and the most testing.
Degradable. Recommendations, search suggestions, analytics. These fall back to defaults.
Deferrable. Emails, reports, notifications. These go through queues and can lag safely.

The architectural payoff is isolation. A failure in a degradable feature must not be able to consume the resources of the critical path. Bulkheads (separate thread pools, connection pools or even deployments) enforce that.

Core patterns, and what they cost

Each resilience pattern solves one problem and introduces another:

Timeouts. Every remote call needs one, set from the caller's latency budget. Without it, one slow dependency stalls everything upstream.
Retries with backoff and jitter. Helpful for transient errors, harmful during overload. Retrying at every layer multiplies traffic: three layers with three attempts each means up to 27 calls for one user action.
Circuit breakers. Stop calling a failing service so it can recover, at the price of added state and tuning.
Idempotency. Retries are only safe if repeating a request does not repeat its effect. Use idempotency keys for payments and order creation.
Load shedding. Reject excess work early with a clear error rather than degrading everyone.

client -> gateway (timeout 2s) -> orders (timeout 1s) -> inventory
                                          |
                                          +-> circuit breaker -> cached stock level

Test the failure paths

A fallback that has never executed is a hypothesis. Verify with fault injection: kill a replica in staging, add 2 seconds of latency to a dependency, fill a disk. Run game days where the team practices an outage with a defined hypothesis, such as "checkout stays available if search is down". You will usually find that the fallback exists in code but is disabled by configuration, or that a hidden dependency makes it useless.

Also check recovery, not just survival. Systems often fail again when the dependency returns, because a backlog of queued work floods it. Rate limit your own replays.

Make failure visible

Resilience without observability just hides problems. Emit metrics for fallbacks triggered, retries attempted and breakers opened. A system quietly serving stale data for six hours is a defect, not a success. Alert on the degraded mode itself, and define how stale is too stale.

Key takeaways

Design the failure behavior of each dependency before the happy path is finished.
Slow dependencies are more dangerous than dead ones; always set timeouts.
Tier capabilities into critical, degradable and deferrable, and isolate them with bulkheads.
Retries, breakers and shedding each have costs; apply them deliberately and make operations idempotent.
Exercise failure paths with fault injection, and alert when the system runs in degraded mode.