Multi-region and disaster recovery

Key points

• Two numbers drive every DR design: RPO, how much data you may lose, and RTO, how long you may be down.
• Nightly backups give an RPO of hours. A streaming replica in another region gives seconds, for the price of a second cluster.
• Active-passive is the honest default: one region serves, one follows, and you rehearse the promotion. Untested failover does not work.
• Active-active needs conflict resolution, because two regions will accept writes to the same row 100 ms apart, and both are right.
• Pinning each user to a home region avoids most of those conflicts: reads are local everywhere, writes are routed home.
• Detection plus DNS TTL plus promotion is your real RTO. Checks every 10 s and a 60 s TTL have already cost you minutes.

Example

# Route 53 failover: the real RTO is detection + TTL + promotion, added up.
- name: api.example.com
  type: A
  set_identifier: primary
  failover: PRIMARY
  ttl: 60                       # 60 s of clients still aiming at the dead region
  alias: eu-west-1-alb
  health_check:
    path: /actuator/health/readiness
    interval: 10                # 3 failed checks = 30 s just to notice
    failure_threshold: 3

- name: api.example.com
  type: A
  set_identifier: standby
  failover: SECONDARY
  ttl: 60
  alias: us-east-1-alb

# The standby database streams from the primary.
#   async        -> RPO is the replication lag, usually well under a second
#   remote_write -> RPO is ~0, and every commit waits for the ocean crossing
primary_conninfo: host=db.eu-west-1.internal application_name=us-east-1
synchronous_commit: remote_write     # about +90 ms per commit, London to Ohio

Write down RPO and RTO first, buy exactly that much replication, then rehearse the failover for real.