Multi-Region Replication

Key Points

• Why multi-region: latency (serve users from a close region), disaster recovery (region outage), data residency (GDPR / sovereignty), regulatory isolation, scale.
• Active-passive: one region writes, others stand by. Simple. Failover is an event.
• Active-active: multiple regions accept writes simultaneously. Hard. Requires conflict resolution.
• Conflict resolution strategies: Last-Write-Wins (LWW, beware clock skew), vector clocks, CRDTs (G-Counter, PN-Counter, OR-Set, LWW-Element-Set), application-level merge.
• Synchronous replication = strong consistency, global-RTT latency on every write (Spanner, CockroachDB).
• Asynchronous replication = low-latency writes, RPO measured in seconds-to-minutes (Cosmos DB Strong-vs-Eventual, Azure SQL geo-replication).
• Cosmos DB offers five consistency levels — pick deliberately, not by default.
• Read close, write to home: most apps route reads to the nearest replica and writes to a designated primary per partition/tenant.
• CRDTs shine for collaborative editing, presence, counters, shopping carts. They are not a free pass for arbitrary state.
• RPO/RTO are the only measurable contract: define them per dataset before choosing topology.

Concepts (deep dive)

Why multi-region

Latency:        user in Sydney hitting a Frankfurt DB → 300ms round-trip.
Sovereignty:    EU citizen data must reside in EU.
DR:             AZ failures are common; region failures are rare but survivable
                only if you have a second region.
Regulatory:     PCI/HIPAA/SOC2 may require regional isolation per tenant.
Scale:          beyond ~5-10× the size of a single region's capacity.

If none of these apply, don't go multi-region. The complexity is enormous.

Active-passive vs active-active

Active-Passive (simpler)              Active-Active (harder)
─────────────────────────             ───────────────────────
Region A: writes + reads              Region A: writes + reads
Region B: replica, reads only         Region B: writes + reads
                                      Region C: writes + reads

Failover: promote B to primary        No failover; survivors keep accepting
                                      writes. Conflict resolution required.

RPO: replication lag                  RPO: 0 in chosen region
RTO: failover time (minutes)          RTO: ~0 (DNS / traffic manager)

Active-passive is what most "geo-replication" features actually deliver (Azure SQL geo-replication, RDS read replicas, Cosmos with single write region).

Active-active requires either synchronous quorum (Spanner / CockroachDB / Cosmos multi-write with custom conflict policy) or explicit conflict-free types (CRDTs).

Replication topologies

1. Primary-replica (single writer)
   ┌────────┐    async       ┌────────┐
   │ Region │ ─────────────▶ │ Region │
   │   A    │                │   B    │  reads only
   │ writes │ ─────────────▶ │ Region │
   └────────┘                │   C    │  reads only
                              └────────┘

2. Multi-primary (writes anywhere)
   ┌────────┐ ◀───── async ────▶ ┌────────┐
   │ Region │                     │ Region │
   │   A    │                     │   B    │
   │ writes │ ◀───── async ────▶ │ writes │
   └────────┘                     └────────┘
        ▲                              ▲
        └────────── conflicts ─────────┘

3. Leader-per-region (sharded by tenant)
   Tenant 1 → home region: A (writes there only)
   Tenant 2 → home region: B
   Tenant 3 → home region: A
   Replicates everywhere for reads / DR.

Topology 3 is the unsung hero: most apps with a tenant model can shard tenants to home regions and avoid conflicts entirely.

Synchronous replication

Every write blocks until quorum across regions confirms. Strong consistency anywhere.

Write in A:
  → A persists
  → ship to B, C
  → wait for ACK from majority (e.g., 2 of 3)
  → return success

Latency: dominated by inter-region RTT (Frankfurt-to-NYC ≈ 80ms one-way).
Every write pays 160ms+ minimum.

Examples: Google Spanner, CockroachDB, FoundationDB. Cosmos DB Strong consistency in multi-write configurations.

This is the "feels like one global database" experience. The cost is fixed and unavoidable: writes pay global latency.

Asynchronous replication

Write returns from local region. Replication ships in background.

Write in A: persists locally → ACK to client
Background: ship change-feed/log to B, C
Lag:        ~1-30 seconds typical, can spike under load

Examples: Azure SQL geo-replication (Active Geo-Replication / failover groups), Cosmos DB Eventual / Session / Bounded-Staleness, AWS RDS read replicas, DynamoDB Global Tables (uses LWW under the hood).

RPO = replication lag. If A dies before lag drains, those last writes are gone.

Cosmos DB consistency levels

Cosmos lets you pick per-account; some clients can opt-down per-request.

Lower levels = lower latency + more availability. Pick per dataset, not per app.

Azure SQL geo-replication

Active Geo-Replication: 1 primary + up to 4 readable secondaries (async).
Auto-Failover Groups:   primary + secondary with managed failover endpoints.
RPO:                    seconds (often < 5).
RTO:                    typically < 60s for managed failover.

Always asynchronous. Failover is explicit (or automatic on policy). The connection string points to the failover-group endpoint, which redirects to whichever is primary.

DynamoDB Global Tables

Active-active across regions. Conflict resolution = LWW by per-item timestamp written by the service. Simple but cannot represent "merge two shopping carts" — last write wins, the other is gone.

Spanner / CockroachDB

Synchronous via Paxos / Raft groups. Each row's leader lives in one region (the region nearest the writer if configured that way). Writes pay leader-quorum latency. Reads can be served stale-locally or fresh-from-leader.

Conflict resolution strategies

Active-active inevitably produces concurrent writes to the same key. You choose how to merge.

LWW (Last-Write-Wins)

A writes value=X at t1
B writes value=Y at t2  (t2 > t1)
Merge: keep Y, discard X.

Simple. Works if time is trustworthy. With clock skew across regions, "later" is ambiguous. Hybrid Logical Clocks (HLC) help but don't fully solve it.

Vector clocks

Each replica increments its own counter on write. Reads see all "concurrent" versions and the application picks (or merges).

A writes:  vc = {A:1}
B writes:  vc = {B:1}    ← concurrent; neither dominates
C reads:   gets both versions, must merge

Used by Riak, original Dynamo paper. Pushes complexity to the app.

CRDTs (Conflict-free Replicated Data Types)

Mathematical types where concurrent updates always merge deterministically. No coordination required.

Yjs and Automerge power most modern collaborative editors (Figma, Linear, Notion-style).

Application-level merge

Sometimes the app has domain rules: "two carts merged = union of items, sum of quantities, latest discount". Implement explicitly. Read both versions; produce a new merged write.

Read-replica routing

// Pseudo: pick connection by intent + closest region
public DbContext GetContext(IntentKind intent)
{
    if (intent == IntentKind.Write) return _primary;
    return _replicas.Closest(_currentRegion);   // by latency or static config
}

Read-from-closest, write-to-primary is the default pattern. Beware: a write followed by a read in another region may not see the write — Session consistency or read-your-write tokens fix this.

RPO and RTO

RPO (Recovery Point Objective): how much data can we lose?
   - Synchronous replication: 0
   - Async replication:        equal to current lag
   - Backups:                  equal to backup interval

RTO (Recovery Time Objective): how long to recover?
   - Active-active failover: seconds (DNS / traffic manager)
   - Active-passive promotion: minutes
   - Restore from backup:    hours

These are business decisions, not engineering preferences. Get them in writing per dataset.

Multi-region SignalR

Real-time pub/sub across regions: use Azure SignalR Service with a shared backplane, or run multiple SignalR services with a message-bus backplane (Redis, Service Bus). See SignalR for the messaging side.

Region A SignalR  ←──── Service Bus Topic ────→  Region B SignalR
       │                                                  │
   clients in EU                                    clients in US

Messages broadcast in A reach US clients via the bus. Watch out: ordering across regions is per-topic-per-partition only.

How it works under the hood

Cosmos DB multi-region write under the covers:

Write to closest region:
  1. Region A leader (for that partition) commits locally.
  2. Replication queue ships the change to B and C.
  3. Local ACK returns immediately.
  4. Async: B and C apply; if conflict, the configured resolver runs.
  5. Conflict feed contains losers (you can query / log them).

Azure SQL failover group on region failure:

1. Health probe detects A failing.
2. (Auto policy or manual) initiate failover.
3. Listener endpoint switches DNS to B.
4. Connections drain; reconnect lands on B.
5. B promotes from secondary to primary.
6. Recent writes from A that hadn't replicated → lost (RPO).

Code: correct vs wrong

❌ Wrong: assume LWW handles merges

// Two regions both update cart concurrently.
// LWW keeps only one — items added in the loser region are lost.
await container.UpsertItemAsync(cart);

✅ Correct: model cart as an OR-Set CRDT or merge in app

// On read, fetch all versions; merge by union of items.
var versions = await container.ReadAllVersionsAsync(id);
var merged = MergeCarts(versions);
await container.UpsertItemAsync(merged);

Cosmos can also be configured to surface conflicts to a feed for explicit handling.

❌ Wrong: write to primary, immediately read from a replica

await primary.SaveChangesAsync();
var fresh = await replica.Query.FirstAsync(...);   // may not see write

✅ Correct: read-your-writes via session or routing

// Cosmos Session consistency uses session token
var token = await primary.UpsertItemAsync(...);
var fresh = await replica.ReadItemAsync(...,
    new ItemRequestOptions { SessionToken = token });

Or route post-write reads to the primary for a short window.

❌ Wrong: time-based LWW with naive timestamps

item.UpdatedAt = DateTime.UtcNow;   // depends on clock — can be wrong by seconds

✅ Correct: HLC or server-assigned ETag

// Let the storage assign the version; let conflict policy decide
await container.ReplaceItemAsync(item, id,
    new ItemRequestOptions { IfMatchEtag = item._etag });

Design patterns for this topic

Pattern 1 — "Tenant home region"

• Intent: shard tenants to a home region; replicate read-only elsewhere; avoid conflicts.

Pattern 2 — "Active-passive with managed failover"

• Intent: simplest viable DR; accept seconds of RPO.

Pattern 3 — "CRDTs for collaborative state"

• Intent: carts, presence, counters, collaborative documents.

Pattern 4 — "Pick consistency per dataset"

• Intent: Cosmos Bounded-Staleness for orders; Eventual for telemetry; Strong only when truly required.

Pattern 5 — "Read close, write to home"

• Intent: lowest read latency; write latency bounded by inter-region link only when crossing home.

Pattern 6 — "Conflict feed reviewer"

• Intent: in active-active, log losers; humans / batch jobs reconcile.

Pros & cons / trade-offs

When to use / when to avoid

• ✅ Use active-passive when latency is acceptable from one region and DR is the goal.
• ✅ Use tenant-sharded multi-region when you have isolated tenants.
• ✅ Use CRDTs for inherently mergeable state (carts, collaborative docs, counters).
• ❌ Avoid active-active LWW for data with merge semantics (carts, sets, balances).
• ❌ Avoid synchronous global replication unless your business requires it — the latency tax is paid every write.
• ❌ Avoid going multi-region "for performance" before you've measured a real geographic latency problem.

Interview Q&A

Q1. Why multi-region? Latency, disaster recovery, sovereignty, regulatory, scale. Pick the actual driver — different drivers point at different topologies.

Q2. Active-active vs active-passive? Active-active accepts writes anywhere; needs conflict resolution. Active-passive has one writer; failover on outage. Most "geo-replication" is active-passive.

Q3. Synchronous vs async replication? Sync = strong consistency, global-RTT writes always. Async = low-latency writes, RPO equal to lag.

Q4. What's RPO vs RTO? RPO = how much data we tolerate losing on failover (sec/min). RTO = how long until back online (sec/hr).

Q5. Cosmos DB consistency levels? Strong, Bounded-Staleness, Session, Consistent-Prefix, Eventual. Pick per dataset; lower = lower latency.

Q6. What's LWW and its risk? Last-Write-Wins picks the latest timestamp. Risks: clock skew picks the wrong winner; merges (e.g., two cart updates) lose data.

Q7. What's a CRDT? A data type with deterministic, commutative, associative merge — concurrent updates always converge without coordination. G-Counter, OR-Set, Yjs, etc.

Q8. When are CRDTs not enough? When semantics aren't expressible (e.g., "max-of-quantities" with re-adds may need OR-Set or custom). Also when you need strong constraints like "balance never negative" — CRDTs are eventually consistent.

Q9. Azure SQL geo-replication is sync or async? Asynchronous. RPO ≈ seconds. Failover groups manage the endpoint redirection.

Q10. Spanner / CockroachDB how? Paxos/Raft quorums per partition leader. Synchronous quorum = strong consistency. Latency tax per write.

Q11. Tenant-sharded multi-region? Each tenant has a home region; writes go there; data replicates read-only elsewhere. Avoids conflicts entirely.

Q12. Read-your-writes across regions? Session tokens (Cosmos), sticky-route post-write to primary briefly, or use stronger consistency for the affected reads.

Q13. Multi-region SignalR? Azure SignalR Service or self-hosted SignalR with a message-bus backplane (Service Bus, Redis) bridging regions.

Q14. How to test multi-region? Chaos failover drills (region down), inject replication lag, verify conflict-feed handling, run latency assertions per region.

Gotchas / common mistakes

• ⚠️ Active-active LWW on shopping carts — silent item loss in conflicts.
• ⚠️ Defaulting to Strong consistency in Cosmos — pays the latency tax on every read everywhere.
• ⚠️ Clock-skew-dependent timestamps — pick HLC or server-assigned versions.
• ⚠️ Forgetting RPO/RTO are per-dataset — you can't have one global number.
• ⚠️ Untested failover — automated failover paths rot fast; drill quarterly.
• ⚠️ Cross-region chatty service calls — three serial calls × 80ms RTT = 240ms baseline.
• ⚠️ "Geo-replication = HA" — replication doesn't replace backups; corruption replicates.
• ⚠️ Ignoring conflict feeds in Cosmos / DynamoDB — losers vanish silently otherwise.
• ⚠️ CRDT misuse — using a counter where you need a constraint (e.g., "non-negative balance") leads to overdraws.

Further reading

• Azure Cosmos DB consistency levels
• Azure SQL Active Geo-Replication
• Cosmos DB conflict resolution
• Google Spanner whitepaper
• CockroachDB multi-region docs
• CRDTs: Shapiro et al.
• Yjs and Automerge for collaborative-editing CRDTs
• Azure SignalR Service