Disaster Recovery Architecture

Disaster recovery, or DR, is the capability to restore a system after a failure large enough to take out an entire site, region, or data store. High availability handles routine failures such as a crashed instance by redundancy. DR handles the rare catastrophic ones: a region outage, a corrupted database, a ransomware attack, or an operator deleting the wrong thing. The central design question is not "how do we avoid disaster" but "how fast must we recover, and how much data can we afford to lose".

Start With RTO and RPO

Two numbers drive every DR decision:

Recovery Time Objective (RTO): the maximum acceptable time from failure to restored service.
Recovery Point Objective (RPO): the maximum acceptable window of data loss, measured backward from the failure.

An RPO of five minutes means you need replication or backups at least that frequent. An RTO of one hour means your restore process, from detection to traffic cutover, must reliably finish within an hour. These should be set per system with the business, not by engineering alone, because each step toward zero costs sharply more. A marketing site and a payments ledger deserve different answers.

The Four Standard Patterns

AWS's well-known framing describes a spectrum from cheap and slow to expensive and fast:

| Pattern | What runs in the DR site | Typical RTO | Typical RPO | Cost | |---|---|---|---|---| | Backup and restore | Nothing; backups stored off-site | Hours to days | Hours | Lowest | | Pilot light | Data replicated; core services off or minimal | Tens of minutes to hours | Minutes | Low | | Warm standby | Scaled-down full stack running | Minutes | Seconds to minutes | Medium | | Active-active | Full capacity in multiple regions serving traffic | Near zero | Near zero | Highest |

Most organizations tier their systems: critical customer-facing services get warm standby or active-active, internal tools get backup and restore. Matching the pattern to the tier prevents overspending on everything and underprotecting what matters.

Architecture Building Blocks

A recoverable system depends on a few design choices made early:

Replicate data across failure domains. Use cross-region replication for databases and object storage. Asynchronous replication gives better latency but a nonzero RPO; synchronous replication approaches zero RPO at a latency cost.
Keep backups isolated and immutable. Replication faithfully copies corruption and deletions. Point-in-time backups, stored in a separate account with write-once retention, protect against both malicious and accidental damage.
Define infrastructure as code. If the DR environment can be recreated from version-controlled templates, recovery is a pipeline run rather than a heroic effort.
Make the application region-agnostic. Avoid hard-coded endpoints, sticky local state, and region-specific dependencies. Store sessions and configuration in replicated stores.
Control traffic with DNS or global load balancing. Health-checked failover routing, with low TTLs, moves users to the healthy region. Keep in mind that DNS caching makes cutover slower than the numbers suggest.

            Global traffic manager (health-checked)
                  /                      \
        Region A (primary)         Region B (standby)
        app tier  <--- async replication --->  app tier (scaled down)
        database primary                       database replica
        backups ----> separate account, immutable storage

Failover Is the Easy Part; Failback and Split Brain Are Not

Plan the return trip too. After failover, the old primary may hold writes that never replicated. Decide how to reconcile or discard them, and ensure the old primary cannot accept traffic and diverge, a condition called split brain. Use fencing, quorum, or an explicit promotion procedure rather than automatic promotion on a flaky signal.

Also inventory dependencies that quietly break DR: secrets managers, identity providers, third-party APIs, licenses, DNS providers, and the CI system that deploys your fix. If your recovery tooling lives in the region that failed, you do not have a DR plan.

Test It or It Does Not Exist

Untested DR plans fail. Backups that were never restored are hopes, not backups. Build a testing ladder:

Regularly restore backups into a scratch environment and verify integrity and timing.
Run tabletop exercises where the team walks through a scenario and its decisions.
Execute game days that fail over a real workload, ideally in production for critical systems.
Measure achieved RTO and RPO against targets, and fix the gaps.

Document runbooks with explicit decision points, contacts, and the authority to declare a disaster. In a real event, nobody wants to debate who may press the button. Review the plan after every architecture change and after each exercise.

Key Takeaways

Define RTO and RPO per system with the business; they determine cost and pattern.
Choose among backup and restore, pilot light, warm standby, and active-active by tier.
Replication is not backup: keep immutable, isolated, point-in-time copies.
Automate environments as code and keep recovery tooling outside the failure domain.
Plan failback and split-brain prevention, and rehearse regularly until achieved times match targets.