Skip to content

Saga Compensation Patterns — Deep

This file deepens Sagas: Orchestration vs Choreography. Read that first; this topic focuses specifically on compensation — the hardest part of any saga.

Key Points

  • Compensation is semantic, not technical. You cannot "rollback" a payment, an email, or a third-party API call. You issue a new forward action that produces the equivalent business effect (refund, retraction, reverse-shipment).
  • Process Manager is the home of compensation logic. It owns the saga state, knows which steps completed, and decides which compensations to fire and in what order.
  • Compensations must be idempotent. They are retried like any other message. A double-refund is worse than a missed one.
  • Backward recovery (compensate completed steps in reverse) is the default. Forward recovery (proceed past failure with degraded result) is sometimes correct — e.g., partial fulfillment.
  • Pivot points are non-compensable steps. They must succeed before the saga is allowed to proceed (or you accept that the only "comp" is human escalation).
  • Model the saga as a state machine. XState-style or MassTransit StateMachineSaga. Don't hand-roll branching if/else flows in handlers.
  • Timeouts and escalation are first-class. Comp steps that exceed SLA route to a human queue, not a loop.

Concepts (deep dive)

Compensation is not rollback

Database transactions roll back via undo logs. They are technical: the storage engine reverses byte-level changes that no one ever saw (because the transaction never committed).

A saga step is already committed in some external service when the compensation runs. The payment captured. The email sent. The shipping label printed. You cannot un-commit those.

Database TX rollback              Saga compensation
-------------------               -----------------
Atomic, reversible                Forward action with same business meaning
Undo log entries                  RefundPayment, CancelShipment, SendRetraction
Invisible to clients              Visible business event (refund hits statement)
Free                              Costs money / time / customer goodwill

This distinction drives every other rule. The compensation is a new operation that the business has to do anyway — it just happens to be triggered by the saga rather than by a human clicking "refund".

The Process Manager

A Process Manager (PM) is a stateful coordinator that:

  1. Receives events (InventoryReserved, PaymentFailed).
  2. Updates its internal state ("we are in Charging, awaiting PaymentCharged").
  3. Issues commands (ChargePayment, or on failure ReleaseInventory).
  4. Persists its state durably so a crash doesn't lose mid-flight sagas.
                ┌────────────────────────┐
                │   Process Manager      │
                │   (saga state machine) │
                │                        │
                │   completed:           │
                │     - InventoryReserved│
                │   pending: Payment     │
                │   pivot:    Shipping   │
                └─────────┬──────────────┘
                          │ on failure → emit comps in reverse
                          ▼
              ┌─────────────────────┐
              │ ReleaseInventory    │  ← compensation for step 1
              └─────────────────────┘

The PM is the only place that knows the full ordering. Individual services know nothing about the workflow shape.

Backward recovery (default)

Compensate completed steps in reverse order:

Forward:    [1 Reserve] → [2 Charge] → [3 Ship]   ✗ fails
Backward:   [3 nothing] ← [2 Refund] ← [1 Release]

Reverse order matters when steps depend on each other. Releasing inventory before refunding payment is fine in this example, but in workflows where step 2 uses the result of step 1, you must comp 2 first.

Forward recovery

Sometimes "undo everything" is wrong. Examples:

  • Order has 5 line items; one is out of stock. Forward: ship the 4 available, refund only the missing line.
  • Travel booking: flight booked, hotel booked, car rental fails. Forward: notify user, let them keep flight + hotel, retry car offline.
Forward:    [Flight ✓] → [Hotel ✓] → [Car ✗]
Decision:   keep flight & hotel; downgrade goal; user notified

The PM decides per workflow whether failure means "abort everything" or "proceed with degraded outcome".

Pivot points

A pivot is a step that, once committed, cannot be compensated cheaply or at all:

  • Sending a physical letter.
  • Triggering a manufacturing order on the factory floor.
  • Booking a non-refundable flight.
  • Applying a credit to an account where the customer has already withdrawn it.

Pivots split the saga into two phases:

[ Compensable Phase ]  →  [ PIVOT ]  →  [ Retriable Phase ]
   any failure here          must            failure here:
   triggers comps            succeed         retry forever
                                             (or escalate)

Before the pivot, the saga can fail safely. After the pivot, the saga must complete — failures escalate to humans, not to comps.

Compensation idempotency

Compensations run via the same at-least-once messaging you use for forward steps. They will retry. They may double-fire after a crash. Comps must be idempotent.

// ❌ Wrong: refund based on amount
public async Task Handle(RefundPayment cmd)
{
    await _payments.Refund(cmd.OrderId, cmd.Amount);   // double-call → double refund
}

// ✅ Right: refund keyed by saga's compensation ID
public async Task Handle(RefundPayment cmd)
{
    if (await _refunds.ExistsAsync(cmd.CompensationId)) return;
    await _payments.Refund(cmd.OrderId, cmd.Amount);
    await _refunds.RecordAsync(cmd.CompensationId);
}

The PM should generate a stable CompensationId per (saga, step). The downstream service dedupes on it. See Idempotency Keys — Deep.

Compensation timeouts and escalation

A comp can fail too. The refund API is down. The shipping carrier rejects the cancellation. What now?

[Comp fired] → retry with backoff (3-5 attempts)
            → still failing → DLQ + alert + manual intervention queue
            → human resolves → resumes saga via admin command

Never "fire and forget" a compensation. Track its outcome:

During(Compensating,
    When(RefundCompleted)
        .TransitionTo(Failed),
    When(RefundFailed)
        .Schedule(EscalateToHuman, TimeSpan.FromMinutes(15))
        .TransitionTo(WaitingForOps));

State machine modeling

Hand-rolled compensation logic ends up as nested if/else in handlers — unauditable, untestable, fragile.

A state machine forces you to enumerate:

  • States the saga can be in.
  • Events it accepts in each state.
  • Transitions (including which compensations to fire).
  • Terminal states (Completed, Compensated, Failed-Manual).
                    OrderPlaced
                        │
                        ▼
           ┌──────► Reserving ──Reserved────► Charging
           │            │                       │
           │      ReserveFail               ChargeFail
           │            │                       │
           │            ▼                       ▼
           │        Cancelled           Compensating-Inventory
           │                                    │
           │                              InventoryReleased
           │                                    │
           │                                    ▼
           │                                Cancelled

Tools that enforce this shape:

  • MassTransit MassTransitStateMachine<T> — Automatonymous-style.
  • Wolverine sagas with [Saga] attribute.
  • XState (TypeScript, but the modeling discipline applies in any language).
  • Temporal.io / Durable Functions for very long-running.

How it works under the hood

A typical compensation flow under MassTransit + outbox + persistence:

1. PM in state Charging receives PaymentFailed event.
2. PM transition logic:
     - Mark step "Charging" as failed.
     - Look up completed steps: [InventoryReserved].
     - For each, in reverse: enqueue comp command.
     - Transition to state CompensatingInventory.
     - Persist new state + outbox messages in one DB transaction.
3. Outbox dispatches ReleaseInventory(orderId, compensationId=X).
4. Inventory service:
     - Look up compensationId X in dedup table → not found.
     - Release the reservation.
     - Record compensationId X.
     - Publish InventoryReleased(orderId).
5. PM receives InventoryReleased → transition to Cancelled (terminal).
6. If step 4 fails: retry with backoff. After N retries → DLQ + alert.

Key invariants:

  • The saga's state change and the outbox write are in one DB transaction. The bus dispatch happens after commit.
  • The downstream comp handler does idempotency check first, then work, then dedup record — all in one TX.
  • Terminal state is durable; no further events are accepted (or cause a log-and-ignore).

Code: correct vs wrong

❌ Wrong: synchronous "rollback"

try
{
    await inventory.ReserveAsync(...);
    await payments.ChargeAsync(...);
    await shipping.ScheduleAsync(...);
}
catch
{
    // imagined rollback
    await payments.RollbackAsync(...);   // not a real API; only Refund exists
    await inventory.RollbackAsync(...);  // hope the call succeeds; no retry
}

Catch blocks aren't durable. A pod restart loses the comp work entirely.

✅ Correct: comp as a forward command, durably stored

public class OrderSaga : MassTransitStateMachine<OrderState>
{
    public OrderSaga()
    {
        InstanceState(x => x.CurrentState);

        During(Charging,
            When(PaymentFailed)
                .Then(c => c.Saga.FailureReason = c.Message.Reason)
                .Send(c => new ReleaseInventory(
                    c.Saga.OrderId,
                    compensationId: $"{c.Saga.CorrelationId}:release"))
                .TransitionTo(CompensatingInventory));

        During(CompensatingInventory,
            When(InventoryReleased)
                .TransitionTo(Cancelled)
                .Finalize(),
            When(InventoryReleaseFailed)
                .Schedule(EscalateAfter, TimeSpan.FromMinutes(15))
                .TransitionTo(WaitingForOps));
    }
}

❌ Wrong: non-idempotent compensation handler

public async Task Consume(ConsumeContext<ReleaseInventory> ctx)
{
    await _inv.IncreaseStock(ctx.Message.Sku, ctx.Message.Qty);  // double-call → double stock
}

✅ Correct: dedupe on the compensationId

public async Task Consume(ConsumeContext<ReleaseInventory> ctx)
{
    var comp = ctx.Message;
    using var tx = await _db.Database.BeginTransactionAsync();

    if (await _db.AppliedComps.AnyAsync(x => x.Id == comp.CompensationId))
    {
        await _bus.Publish(new InventoryReleased(comp.OrderId, comp.CompensationId));
        return;
    }

    await _inv.ReleaseReservationAsync(comp.OrderId);
    _db.AppliedComps.Add(new AppliedComp { Id = comp.CompensationId });
    await _db.SaveChangesAsync();
    await tx.CommitAsync();

    await _bus.Publish(new InventoryReleased(comp.OrderId, comp.CompensationId));
}

❌ Wrong: fire-and-forget compensations

_bus.Publish(new RefundPayment(orderId));
_bus.Publish(new CancelShipment(orderId));
_bus.Publish(new ReleaseInventory(orderId));
saga.State = "Cancelled";   // we never check if comps succeeded

If the refund fails, the saga has lied about being cancelled.

✅ Correct: track each comp through to terminal

Wait for RefundCompleted, ShipmentCancelled, InventoryReleased events before declaring saga Cancelled.

Design patterns for this topic

Pattern 1 — "Process Manager owns the comp ledger"

  • Intent: centralize knowledge of which steps completed and what to undo.

Pattern 2 — "Compensation as forward command"

  • Intent: model Refund, Release, Cancel as normal commands with their own retries/DLQ.

Pattern 3 — "Pivot point separator"

  • Intent: distinguish compensable phase from must-succeed phase; route post-pivot failures to humans.

Pattern 4 — "Idempotency key per (saga, step)"

  • Intent: safely retry comps without duplicate side effects.

Pattern 5 — "Escalation timer on stuck comp"

  • Intent: never wait forever; surface to ops when SLA exceeded.

Pattern 6 — "Forward recovery for partial success"

  • Intent: don't undo what's already useful; degrade the goal instead.

Pros & cons / trade-offs

Aspect Pros Cons
State machine modeling Auditable; testable Up-front design cost
Backward recovery Predictable; symmetric Wastes work already done
Forward recovery Salvages partial success Decision logic per workflow
Pivot points explicit Clear must-succeed boundary Sometimes forces redesign
Comp idempotency keys Safe retries Extra storage + check
Escalation timers No silent stalls Ops queue must exist

When to use / when to avoid

  • ✅ Use explicit comp modeling when the saga has 3+ steps with side effects.
  • ✅ Use forward recovery when partial outcomes have business value.
  • ✅ Use pivot point modeling when the saga touches non-refundable, non-cancellable, or expensive irreversible operations.
  • ❌ Avoid hand-rolled try/catch rollback outside a state machine.
  • ❌ Avoid assuming the database can rollback what an HTTP call already did.
  • ❌ Avoid compensations that depend on other compensations completing (deadlock risk) — keep them parallel-safe or strictly sequential via the PM.

Interview Q&A

Q1. Why is compensation not the same as rollback? A rollback reverses an uncommitted change in storage. A compensation is a new committed business action (refund, retraction) that produces the same business effect as undoing the original. The original step is already visible to the world; you cannot un-commit it.

Q2. Where does compensation logic live? In the Process Manager / saga state machine. Individual services don't know the workflow shape; they only handle commands and emit events.

Q3. Why must compensations be idempotent? At-least-once delivery means comps may fire twice. A double-refund is a real customer-visible bug. Idempotency keys per (saga, step) are the standard fix.

Q4. Backward vs forward recovery? Backward: undo completed steps in reverse to abort the saga. Forward: accept partial success and degrade the goal. Choose per workflow.

Q5. What is a pivot point? A step that cannot be compensated (or only at huge cost). After the pivot, the saga must complete; failures escalate to humans, not to comps.

Q6. How do you escalate a stuck compensation? Schedule a timeout; on expiry, route to a manual-intervention queue with full context. Don't loop forever.

Q7. Why a state machine over branching if/else? Enforces enumerated states + transitions. Auditable. Testable in isolation. Survives long-running flows across redeploys.

Q8. How does the PM persist state? Saga state row in EF/Mongo/Redis with optimistic concurrency. State change + outbox messages in one TX. Survives crashes.

Q9. What if the comp itself can't be implemented? Either redesign to make the step compensable, or mark it a pivot point and accept human escalation. There is no third option.

Q10. How do you test compensations? Inject failures at every step in tests. Assert the saga reaches a terminal state with the correct comps fired in the right order. MassTransit InMemoryTestHarness makes this practical.

Q11. Comp ordering — does it matter? Yes, when steps build on prior state. Generally: comp in reverse of forward order. Document exceptions explicitly in the state machine.

Q12. How is comp progress observed? Each comp is a tracked message with its own metric (count, latency, failure rate) and DLQ. Treat comps as first-class workflows, not error paths.

Q13. Comp vs retry — when to do which? Retry the same step with backoff for transient failures. Compensate only after retry budget is exhausted, or for permanent failures (e.g., card declined).

Q14. Can a comp itself trigger another saga? Yes — and it often does (a refund saga). Keep nesting shallow; deep saga trees become unobservable.

Gotchas / common mistakes

  • ⚠️ Treating comps as try/catch cleanup — non-durable; lost on crash.
  • ⚠️ Non-idempotent comp handlers — duplicate refunds in production.
  • ⚠️ No pivot point modeling — saga deadlocks when a "comp" fails after an irreversible step.
  • ⚠️ Fire-and-forget comps — saga marked Cancelled while the refund silently failed.
  • ⚠️ No escalation path — stuck comps loop forever, alerts pile up, eventually muted.
  • ⚠️ Comp ordering errors — releasing inventory before saving the cancellation context, then losing the order ID.
  • ⚠️ Hand-rolled state machines in handlers — diverges from documented workflow within months.
  • ⚠️ Forgetting the comp budget — refunds cost money; if you comp on every transient blip, finance will notice.

Further reading