Disaster Recovery: RPO, RTO, Strategies, and Game Day

Disaster Recovery is about restoring service after a catastrophic event. Two KPIs drive design: RPO (Recovery Point Objective) — how much data you can afford to lose — and RTO (Recovery Time Objective) — how quickly you must be back online.

Types of disasters

• Region/data center outage (fire, power, network)
• Corruption (bug writes bad data across fleet)
• Ransomware (attacker encrypts storage)
• Accidental destruction (DROP TABLE in prod)
• Vendor failure (cloud provider, SaaS shutdown)
• Natural disasters (flood, earthquake)

RPO and RTO

• RPO: max tolerable data loss in time (1h = lose up to 1h writes)
• RTO: max tolerable downtime (30 min = serving within 30 min)
• Business sets numbers from revenue, SLAs, compliance
• Engineering picks strategy meeting those at bearable cost

DR vs HA

• HA: within-region failures (AZ loss, instance death)
• DR: region / catastrophic failures
• HA doesn't save you from whole region down

Four canonical strategies

1. Backup & Restore (RPO hours, RTO hours/days, Cost $)

• Daily/hourly snapshots to cross-region storage
• IaC re-creates infra on disaster
• Data from latest valid snapshot
• Best for: non-critical, dev/staging, cost-sensitive
• Quarterly restore test to fresh account/region

2. Pilot Light (RPO min-hrs, RTO tens of min, Cost $$)

• Always-running DB replica in DR region (async)
• App defined in IaC but not running (or minimum)
• Disaster: scale app, promote DB, redirect traffic
• RTO 15-30 min (DB promotion + ECS startup)

3. Warm Standby (RPO min, RTO min, Cost $$$)

• Always-running full replica at reduced capacity (25-50%)
• App serves no traffic or trivial share
• Disaster: scale up, shift traffic, promote DB
• Less scaling needed; continuous validation that DR actually works

4. Active-Active / Multi-Region (RPO sec, RTO sec, Cost $$$$)

• Full capacity in multiple regions; traffic split by latency/routing
• Data: globally consistent (Spanner, CockroachDB, DynamoDB Global Tables, Aurora Global) or eventually consistent
• Challenges: write conflicts, 50-150ms latency, data sovereignty, consistency choices
• Right for: global SaaS, high-availability finance, e-commerce
• Wrong for: strong consistency + low latency + no conflict resolution

Database replication modes

• Synchronous: primary waits for replica ack; RPO=0, write latency penalty
• Asynchronous: primary commits immediately; replica catches up; RPO = lag
• Semi-sync (MySQL): primary waits for one replica; balance
• Multi-region sync = 50-150ms write penalty; usually async with small RPO

Point-in-Time Recovery (PITR)

• Restore to any moment within retention (7-35 days)
• WAL/binlog archived continuously
• Essential for logical errors (bad code dropped data)
• AWS RDS, Aurora, Postgres with pgBackRest, WAL-G

Backup hygiene

• 3-2-1 rule: 3 copies, 2 media, 1 offsite
• Immutable backups: S3 Object Lock, Azure Immutable Blob (attacker can't delete/encrypt)
• Cross-account/cross-region: main account compromise doesn't kill backups
• Tested restoration: untested backups are fictional

Ransomware-specific

1. 1
   Immutable backups (WORM storage)
2. 2
   Versioned object storage (revert encryption)
3. 3
   Cross-account backup with limited access
4. 4
   Air-gapped offline for highest tier
5. 5
   Detection: monitor mass encryption patterns (write spikes)
6. 6
   IR runbook with isolation without destroying evidence

Corruption scenarios (often missed)

• Bad deploy writes invalid data across fleet — DR replica has same bad data
• Solution: retain PITR snapshots ≥ 2x max detection window
• 'Delayed replica' (1 hour behind) as ransomware defense

Spring-specific

• Application state: stateless best (12-factor); state in backing services
• Config: from env/external store (Vault, Spring Cloud Config); reconstitutable in DR
• Build artifacts: cross-region container registry; deploy in DR without primary dependency
• Secrets: Vault cross-region replication or KMS in both
• Certificates: issue for DR region pre-disaster or wildcard

Testing DR: runbooks, game days, chaos

Three levels

1. 1
   Tabletop exercise: walk through runbook in meeting; quarterly
2. 2
   DR drill: execute failover on test env; annual minimum; quarterly for critical
3. 3
   Full game day: simulate real disaster on prod-like traffic; measure actual RTO/RPO

Runbook template

• Incident type, owner, RPO/RTO
• Pre-conditions: access, comms, decision tree
• Steps: confirm disaster, notify stakeholders, failover sequence (scale, promote, DNS, validate)
• Monitor: error rate, latency, KPIs
• Communicate to users
• Failback: primary recovers, reduced capacity, gradual shift, sync data back, decommission DR

Game day patterns

• Unplanned, scheduled: team knows week, not hour
• Increasing complexity: 'DB replica promote' → 'region isolated, 40% cutover'
• Measure: detect, decide, execute, recover time
• Retrospective: blameless; runbook updates

Chaos tools

• AWS Fault Injection Simulator
• Chaos Mesh / Litmus (K8s)
• Netflix Chaos Monkey
• Gremlin (commercial)

Observability

• Dashboard: health by region, replication lag, failover state
• Alerts: lag > threshold, backup age > N hours, RPO/RTO exhausted
• Centralized logs across regions

Choosing a strategy | RPO | RTO | Strategy | Cost multiplier | | 24h | 24h | Backup & Restore | 1x | | 1h | 1h | Pilot Light | 1.2-1.5x | | 5 min | 15 min | Warm Standby | 1.5-2x | | <1 min | <1 min | Active-Active | 2-3x |

• Not every service same tier; Tier-1 (revenue): warm/active-active; Tier-2: pilot light; Tier-3: backup only

Costs Over-provisioning

• Infra cost 50-200% more
• Operational complexity; every deploy works in DR
• Dev distraction; drift risk

Under-provisioning

• Extended outage = revenue loss
• Customer churn
• Regulatory fines (GDPR 72-hour breach)
• Emergency consultants at catastrophic rates

Common mistakes

1. 1
   'We have backups' but never test restore
2. 2
   DR region diverges (fix applied to primary only)
3. 3
   Forgetting DNS TTL (long TTLs delay failover)
4. 4
   Shared dependencies (both regions hit same auth in one region)
5. 5
   'Region failure'-only plans miss corruption, ransomware, operator error
6. 6
   No failback plan (stuck in DR)
7. 7
   Manual runbook for 3 AM (sleepy on-call can't execute 40 steps)
8. 8
   DR not in budget ('add later' = never)
9. 9
   Insufficient testing (annual drill finds problems too late)

DR meets 12-factor

• Stateless services: trivially DR-able
• Config as env: easily re-created
• Backing services DR = most of your DR strategy
• Logs to stdout: centralized aggregation across regions

Emerging patterns

• Zone/cell architecture: partition users by cell; blast radius limited
• Chaos engineering as platform: continuous low-level chaos in prod
• Event sourcing for replay: replay to any region/point
• Serverless: failover often automatic

Strategic take

• DR is insurance, not a project you finish
• Match RPO/RTO to revenue impact, not engineering ambition
• Invest in testing and automation, not just architecture
• Focus on likely disasters: corruption + ransomware + operator error more common than region outages
• Culture: DR is everyone's responsibility; bake into design reviews
• Iterate: annual test finding 10 gaps > untested perfect architecture

Key interview signal

• DR is multi-layered: backup, replication, failover, runbook, testing
• Focus on actually tested DR, not theoretical

Bottom line

• RPO/RTO drives strategy choice
• Four strategies: Backup/Restore → Pilot Light → Warm Standby → Active-Active
• Test with game days; runbook must be rehearsed
• Ransomware + corruption matter as much as region outages
• Untested DR = no DR