CQRS Read-Model Rebuilding

Key Points

Read models drift — projections have bugs, schemas evolve, new derived views are needed, corruption happens. Rebuild capability is non-negotiable in mature CQRS systems.
Source of truth for rebuild: an event store (replay), a CDC feed from the source DB (Debezium → Kafka), or both.
Strategies: replay-from-store, parallel projection (build new alongside old, swap when caught up), versioned projections (v1/v2 traffic shift), dual-write during cutover.
Snapshotting bounds replay time on long-lived aggregates. Without it, a 1B-event store takes hours to rebuild.
Idempotency at the projector is mandatory: events arrive at-least-once; dedupe by (streamId, version) or eventId.
Versioning projections (orders_v1, orders_v2) is the safest cutover. Run both; route traffic by feature flag; drop the old when confident.
Watermarks and lag monitoring tell you when a rebuild is "caught up". Without them, you ship a half-built read model to prod.
Upstream coordination matters: if upstream Kafka topics have compacted/expired, your downstream rebuild cannot recover what's gone. Plan retention with rebuild SLAs in mind.

Concepts (deep dive)

Why rebuild?

Schema evolution:    new column / new index / new shape.
Projection bug:      projector miscounted; data is wrong.
New view:            product wants a new screen → new derived read model.
Corruption recovery: storage corruption / accidental delete.
Backfill:            historical events finally enriched (e.g., late dim data).

If you can't rebuild a read model, you can't fix a projection bug. You can't add a new view without a downtime window. You're stuck.

CQRS read-model anatomy

Commands → Aggregate → Events ──► Event Store
                            │
                            ├──► Projector A → Read Model A (e.g., OrderListView)
                            ├──► Projector B → Read Model B (e.g., RevenueByDay)
                            └──► Projector C → Read Model C (e.g., SearchIndex)

Queries → Read Model (whatever shape the UI needs)

Each read model is a cache of the events shaped for one query. Multiple read models from the same events are normal.

Source of truth options

1. Event store

The events themselves are persisted (EventStoreDB, Marten on Postgres, MartenDB, Axon, Kafka with infinite retention or compaction).

Replay: read events from offset 0 → re-run projector → new read model

This is the gold-standard CQRS source of truth. If you have one, rebuild is straightforward.

2. CDC (Change Data Capture)

If your write-side is a normal SQL DB, Debezium (or SQL Server CDC, Postgres logical replication) tails the WAL/transaction log and emits changes as Kafka events.

[App] → [SQL Server] ──CDC──► [Kafka topic] ──► [Projectors] ──► [Read models]

Now Kafka holds the change-stream. Rebuild = replay the Kafka topic from offset 0 (or the earliest retained).

CDC is the rescue path for systems that started without an event store. See NoSQL & Event Stores in Data Access.

3. Both

A modern shape: write-side persists in SQL (transactional), CDC publishes to Kafka, projectors materialize read models. CQRS-shaped without forcing event-sourcing on the write side.

Rebuild strategies

Replay-from-store (the basic move)

1. Stop the old projector.
2. Truncate the read model.
3. Read events from offset 0.
4. Apply each to the projector.
5. When caught up, restart subscription.

Simple but stop-the-world — read model is unavailable during the rebuild. For small models, fine; for big models, unacceptable.

Parallel projection (zero-downtime rebuild)

Time → ─────────────────────────────────────────────────────►

       [v1 projector running, serving traffic]
                          │
                          │  start v2 from offset 0
                          ▼
       [v1 projector]    [v2 projector building...]
                                │
                                │  v2 catches up; lag → 0
                                ▼
                          [v2 caught up]
                          flag flip; route reads to v2
                                │
                                ▼
                          [v1 retired]

Run both. The new projector reads the same event stream from offset 0 in parallel. Once it's caught up (lag ≈ 0), feature-flag the read traffic over to v2. Drop v1 a few days later.

Versioned projections

The natural pairing: read models are named orders_v1, orders_v2. Projectors are named the same. Application config controls which version queries hit.

read_model:
  orders:
    active_version: v2
    available_versions: [v1, v2]

Toggle, observe metrics, roll back instantly if v2 misbehaves.

Dual-write during cutover

When you can't do parallel projection (e.g., the new model has additional event sources), have the old + new projector both run live during the transition; reconcile at the end.

Snapshotting

For long event streams, replay from offset 0 is expensive.

Snapshot at version N: { fullState }
Events from N to current: 100 records

Replay = load snapshot + apply 100 events  (vs apply 1,000,000 events)

EventStoreDB has snapshot streams; Marten supports snapshots; for raw Kafka, you persist snapshots to a side store keyed by (aggregateId, version).

Snapshots are an optimization, not a correctness mechanism. The events are still the truth.

Catch-up subscriptions

A "catch-up subscription" reads historical events first, then transparently switches to live tail.

Subscribe(fromOffset=0) →
   replay historical events (millions) →
   reach log end →
   switch to live (push-based) →
   apply live events as they arrive

Both EventStoreDB .NET client and Marten support this directly. Kafka equivalents: auto.offset.reset=earliest + seek(0).

Idempotent projector

Events arrive at-least-once. Without dedup, you'll re-apply the same event after a crash → wrong totals.

public async Task Apply(OrderPlaced e, ProjectionContext ctx)
{
    using var tx = await _db.Database.BeginTransactionAsync();

    if (await _db.AppliedEvents.AnyAsync(x => x.EventId == e.EventId))
        return;

    _db.AppliedEvents.Add(new AppliedEvent { EventId = e.EventId });
    _db.Orders.Add(new OrderRow { Id = e.OrderId, Total = e.Total });
    await _db.SaveChangesAsync();
    await tx.CommitAsync();
}

Or dedupe by (streamId, version) if your event store provides per-stream version numbers (EventStoreDB, Marten).

Watermarks and lag monitoring

watermark = highest event-time successfully applied to read model
lag       = wallclock-now - watermark   (if event-time)
          OR  log-end-offset - committed-offset  (Kafka)

Expose both. Without them you cannot answer "is the read model caught up?" — and a half-built read model in prod is worse than no read model.

// Pseudo metric emission
metrics.Gauge("readmodel.orders.lag_seconds", DateTime.UtcNow - watermark);
metrics.Gauge("readmodel.orders.events_behind", endOffset - committedOffset);

Coordinating rebuilds across services

In event-driven architectures, downstream read models often depend on upstream events that another team owns. If the upstream team has:

Compacted topics (latest-per-key only).
Time-expired events past retention.
Squashed events into checkpoint snapshots.

…then downstream rebuild can't get the original events back. The history is gone.

Upstream  team A: Kafka topic, retention=7 days.
Downstream team B: needs to rebuild a 6-month projection.

Result: B can rebuild only the last 7 days. Older state is gone.

Mitigations:

Negotiate retention SLAs with upstream (e.g., 90 days minimum).
Snapshot the read model at coarse intervals (your own snapshots).
Use long-retention or tiered-storage Kafka topics for shared event streams.
Have upstream maintain a "replay topic" with full history (separate from the operational topic).

Rebuild cost and blast radius

1B events × 0.1ms apply per event ≈ 100,000 seconds ≈ ~28 hours.
Add IO contention, dedup checks, indices: 2-5×.
A real rebuild is hours-to-days for big systems.

Plan accordingly:

Run rebuilds off-peak.
Throttle so you don't starve the write-side.
Use parallel projectors (one per partition / shard).
Build to a separate DB or table; swap atomically when ready.

Anti-patterns and where they bite

Stop-the-world rebuilds in prod

DELETE FROM orders_view;
-- now run replay for 4 hours, no-one can query

The read model is dead until done. Always use parallel projection.

No projection versioning

Update the projector code → re-deploy → silent partial replay → mixed-shape data

Old data has the old shape, new data has the new shape, queries break in subtle ways. Version the read model name; never mutate in place.

Derive-on-read fallback

"If the read model is missing or stale, query the event store directly and compute on the fly." Hides projection bugs; makes performance unpredictable; complicates caching. Don't.

How it works under the hood

A rebuild via Marten catch-up subscription:

1. Operator triggers rebuild for projection "OrderListV2".
2. Marten creates the v2 projection table.
3. CatchUpSubscription opens with fromSequence=0.
4. Historical events streamed in batches (e.g., 1000 at a time).
5. Each batch applied in a transaction:
     - Apply events to projection table.
     - Update projection_progress table (sequence + watermark).
6. When sequence reaches "log end" → subscription switches to live tail.
7. Lag metric drops to ~0 → flip the read flag from v1 to v2.
8. After observation window, drop v1 projection table.

A rebuild from CDC (Debezium → Kafka):

1. Source SQL DB has CDC enabled; Debezium emits change events to Kafka topic "orders.cdc".
2. Topic retains 90 days OR is compacted (latest per row).
3. New projector consumer group "orders-view-v2" subscribes from offset 0.
4. Builds the read model in a side table.
5. Catches up to live; flag flips.

Code: correct vs wrong

❌ Wrong: stop-the-world rebuild

public async Task Rebuild()
{
    await _db.Orders.ExecuteDeleteAsync();
    await foreach (var e in _events.ReadAllAsync())
        await Apply(e);
}
// read model is gone for hours

✅ Correct: parallel projection with catch-up subscription

// Marten example
public class OrderListV2Projection : EventProjection
{
    public void Project(OrderPlaced e, IDocumentOperations ops)
        => ops.Store(new OrderRowV2 { Id = e.OrderId, Total = e.Total, Customer = e.Customer });

    public void Project(OrderShipped e, IDocumentOperations ops)
        => ops.Patch<OrderRowV2>(e.OrderId).Set(x => x.ShippedAt, e.ShippedAt);
}

// Operator command:
await store.Advanced.RebuildSingleProjectionAsync<OrderListV2Projection>(ct);

// Read code reads orders_v1 until flag flips:
var rows = featureFlags.IsOn("orders_v2")
    ? await session.Query<OrderRowV2>().ToListAsync()
    : await session.Query<OrderRowV1>().ToListAsync();

✅ Correct: EventStoreDB catch-up subscription

await using var sub = client.SubscribeToAllAsync(
    FromAll.Start,
    cancellationToken: ct);

await foreach (var msg in sub.Messages.WithCancellation(ct))
{
    if (msg is StreamMessage.Event(var resolved))
    {
        var evt = Deserialize(resolved.Event);
        await ApplyIdempotentAsync(evt, resolved.OriginalEventNumber);
    }
}

❌ Wrong: non-idempotent projector

public async Task Apply(OrderPlaced e)
{
    var row = await _db.Orders.FindAsync(e.OrderId);
    if (row == null) _db.Orders.Add(new() { Id = e.OrderId, Total = e.Total });
    else row.Total += e.Total;   // double-applied on retry → wrong total
    await _db.SaveChangesAsync();
}

✅ Correct: dedupe by event id

public async Task Apply(OrderPlaced e, EventMetadata m)
{
    if (await _db.AppliedEvents.AnyAsync(x => x.EventId == m.EventId)) return;
    _db.Orders.Upsert(new() { Id = e.OrderId, Total = e.Total });
    _db.AppliedEvents.Add(new() { EventId = m.EventId });
    await _db.SaveChangesAsync();
}

❌ Wrong: derive-on-read fallback hides bugs

var orders = await _readModel.GetAsync(...);
if (orders == null) orders = await _eventStore.ReplayAndComputeAsync(...);  // hides drift

✅ Correct: fail loudly when read model lags

if (lag > _maxAllowedLag)
    throw new ReadModelStaleException("orders_v2 lag exceeds SLO");

Design patterns for this topic

Pattern 1 — "Versioned read models"

Intent: never mutate in place; build vN+1 alongside vN; flag-flip; drop vN.

Pattern 2 — "Catch-up subscription"

Intent: historical replay then seamless switch to live tail.

Pattern 3 — "Idempotent projector with dedup table"

Intent: safe at-least-once apply.

Pattern 4 — "Snapshot to bound replay"

Intent: rebuild long aggregates in minutes, not hours.

Pattern 5 — "Parallel projector swap"

Intent: zero-downtime rebuild via feature flag.

Pattern 6 — "Watermark + lag SLO"

Intent: observable readiness; refuse to serve when stale.

Pattern 7 — "Upstream retention contract"

Intent: ensure upstream history exists for the rebuild horizon you need.

Pros & cons / trade-offs

Strategy	Pros	Cons
Replay-from-store	Simple; correct	Stop-the-world for the read model
Parallel projection	Zero-downtime	Twice the storage during cutover
Versioned models	Safe rollback	Code carries v1 + v2 paths
CDC source	Works without event store	Limited to write-side schema; semantics blurred
Snapshots	Fast replay	One more thing to manage; correctness still on events
Stop-the-world	Cheap to implement	Outage during rebuild

When to use / when to avoid

✅ Use parallel projection + versioned read models for production rebuilds.
✅ Use snapshots when individual aggregate streams exceed ~10K events.
✅ Use CDC + Kafka when you don't have a dedicated event store but want CQRS-shaped read models.
✅ Use idempotency tables in projectors always.
❌ Avoid stop-the-world rebuilds in any system with a real SLA.
❌ Avoid mutating read models in place across schema changes.
❌ Avoid derive-on-read fallbacks — they hide projector bugs.
❌ Avoid assuming upstream events are eternal — verify retention.

Interview Q&A

Q1. Why is rebuild capability essential in CQRS? Projections have bugs, schemas evolve, new views are added, corruption happens. Without rebuild, you cannot fix any of these without downtime or data loss.

Q2. Replay-from-store vs CDC? Event store: events are the source of truth; replay is first-class. CDC: tail the source DB's transaction log into Kafka and project from there. CDC is the rescue path for systems without an event store.

Q3. What is a catch-up subscription? A subscription that reads historical events from the start, then transparently switches to live tail. Both EventStoreDB and Marten support it directly.

Q4. How do you avoid downtime during rebuild? Parallel projection + versioned read models. Build vN+1 alongside vN; flip the read flag when caught up.

Q5. Why must projectors be idempotent? At-least-once delivery means events may apply twice on retry/crash. Dedupe by event ID or (streamId, version).

Q6. What's a snapshot? A serialized state of an aggregate at version N. Replay = load snapshot + apply events from N+1. Bounds replay time on long streams.

Q7. How do you know a rebuild is "done"? Watermark / lag = 0 against the live event stream. Without explicit lag metrics, you don't know.

Q8. Risks of stop-the-world rebuilds? The read model is unavailable during rebuild — minutes to hours of outage.

Q9. What if upstream events have expired? You can rebuild only what's still retained. Plan retention SLAs based on rebuild horizon, or maintain your own snapshots.

Q10. Why version read models? Schema-changing rebuilds in place mix old + new shapes mid-rebuild. Versioning gives clean cutover and instant rollback.

Q11. Marten vs EventStoreDB? Marten: Postgres-backed, .NET-friendly, projections + sessions in one library. EventStoreDB: dedicated event store; richer subscription model. Both support catch-up subscriptions for rebuilds.

Q12. Dual-write during cutover — when? When the new projection has additional inputs the old didn't, so you can't simply replay the same events. Run both live; reconcile.

Q13. Anti-pattern: derive-on-read fallback? "If read model missing, compute from events." Hides bugs; unpredictable latency. Always fail loudly when read model lags beyond SLO.

Q14. Cost of rebuild for a 1B-event store? Hours to days. Throttle, parallelize per partition, build to a side table, swap atomically.

Q15. How does CDC interact with rebuilds? Debezium tails WAL into Kafka. Rebuild = consumer group reads from offset 0. Limited by Kafka topic retention.

Gotchas / common mistakes

⚠️ Stop-the-world rebuilds in prod — outage; angry users.
⚠️ No projection versioning — half-rebuilt mixed-shape data.
⚠️ Non-idempotent projector — wrong totals after retries/crashes.
⚠️ Derive-on-read fallback — hides projector drift indefinitely.
⚠️ No watermark / lag — half-built read model serves traffic; users see partial data.
⚠️ Upstream retention too short — rebuild horizon > retention; history gone.
⚠️ No snapshots on long streams — rebuilds take days.
⚠️ Rebuild during peak — write-side starves on IO.
⚠️ Single projector instance — rebuild is single-threaded; partition the projector by shard for parallelism.
⚠️ Mutating projection table in place — no rollback if v2 has a bug; you've corrupted the only copy.
⚠️ Skipping the dedupe table "because it's slow" — silent corruption beats slow correctness every time.