Multi-Region Failover

Multi-region failover lets a service continue operating when a whole region becomes unavailable. It combines traffic steering, data replication, and operational discipline. The traffic part is the easy half; the hard half is state.

How it works

Common topologies, in rising cost and complexity:

• Backup and restore: data is copied to another region; recovery takes hours.
• Pilot light: core data is replicated and minimal infrastructure is ready to scale up.
• Warm standby: a scaled-down full stack runs in the second region and takes traffic after scaling.
• Active-active: all regions serve traffic; each user is routed to a nearby region, and writes are partitioned by home region or reconciled by replication.

Data replication determines the recovery point objective. Asynchronous replication gives low write latency but loses the unreplicated tail on failover. Synchronous replication avoids loss but adds cross-region latency to every write.

Traffic moves through global load balancing or DNS with health checks, with TTLs of 30 to 60 seconds, though some resolvers ignore low TTLs.

Implementation steps

1. Set RTO and RPO per service tier, and pick a topology that can meet them.
2. Inventory dependencies: identity, secrets, DNS, queues, third-party services. Anything single-region is a failover blocker.
3. Replicate data and monitor lag continuously; alert when lag exceeds the RPO.
4. Keep infrastructure defined as code so both regions are identical.
5. Define failover criteria and who may decide; use fencing so the old primary cannot accept writes.
6. Capacity-reserve the standby, because other tenants will fail over at the same time.
7. Run quarterly region evacuation drills, including failback, and record RTO and data loss.

Common pitfalls

• Control plane dependencies located in the failing region, blocking the failover itself.
• Standby capacity that cannot be obtained during a regional event.
• Split brain after partial network partition.
• Failing back too quickly, with data written in the secondary not yet reconciled.
• Configuration or secrets differing between regions.

Metrics to watch

• Cross-region replication lag against RPO.
• Measured failover time in drills against RTO.
• Standby capacity as a share of peak load.
• Health check consistency across regions.
• Configuration drift between regions.

Example configuration

global_routing:
  policy: failover
  primary: us-east-1
  secondary: eu-west-1
  health_check:
    path: /readyz
    interval: 10s
    failure_threshold: 3
  dns_ttl: 30
  failover_mode: manual_approval
database:
  replication: async
  max_lag_for_promotion: 30s
  fence_old_primary: true

Key takeaways

• Failover is mostly a data problem; design replication first.
• Hunt single-region dependencies, including the control plane.
• Define clear failover criteria and fence the old primary.
• Reserve standby capacity and keep both regions identical.
• Drill failover and failback regularly; evidence beats assumption.
• Treat each drill as a measurement: record elapsed time, data lost, and every manual step that surprised the team.
• Repeat the exercise after each significant change to the system, because results go stale.