Case: Event-Driven Order Pipeline

Problem

Design an e-commerce order pipeline: place order → reserve inventory → charge payment → schedule shipping → notify customer. Up to 1000 orders/min peak. Zero lost orders. Fault-tolerant; observable.

Walkthrough

Clarify

99.9% availability for placing orders; pipeline can be slower.
Idempotent re-tries OK (don't double-charge).
Inventory eventually consistent OK (within seconds).
Payment provider: third-party (Stripe).
Existing infra: Azure.

Capacity

1000 orders/min = ~17/s peak (modest)
Avg order: 5 line items, 2 KB payload
Storage: 10M orders/year * 5 KB = 50 GB
Events: 1 OrderPlaced * 5 downstream = ~85 events/s

Architecture

[Web/API]
  │
  │ POST /orders (idempotency key)
  ▼
[Order Service]
  │
  │ 1. Save Order (DB) + Outbox row in TX
  ▼
[Outbox Relay]
  │
  │ Publishes OrderPlaced
  ▼
[Service Bus Topic: orders]
  │
  ├──→ [Inventory Worker] → InventoryReserved | InventoryFailed
  ├──→ [Payment Worker]   → PaymentCharged   | PaymentFailed
  ├──→ [Shipping Worker]  → ShippingScheduled
  └──→ [Notification]     → CustomerNotified

Saga state:
[OrderSaga (MassTransit / Wolverine)] tracks per-order progress.
On failures, publishes compensation commands.

Why outbox

DB commit + event publish atomic without distributed transaction.
Retry from outbox on transient failure.
See Outbox & Inbox Patterns.

Saga (orchestration)

public class OrderSagaState : SagaStateMachineInstance
{
    public Guid CorrelationId { get; set; }
    public string CurrentState { get; set; } = "";
    public Guid OrderId { get; set; }
    public bool InventoryReserved { get; set; }
    public bool PaymentCharged { get; set; }
}

public class OrderSagaMachine : MassTransitStateMachine<OrderSagaState>
{
    public State Reserving { get; private set; }
    public State Charging { get; private set; }
    public State Shipping { get; private set; }
    public State Done { get; private set; }
    public State Cancelled { get; private set; }
    /* ... */

    public OrderSagaMachine()
    {
        InstanceState(x => x.CurrentState);

        Initially(
            When(OrderPlaced)
                .Then(c => c.Saga.OrderId = c.Message.OrderId)
                .Publish(c => new ReserveInventory(c.Saga.OrderId))
                .TransitionTo(Reserving));

        During(Reserving,
            When(InventoryReserved)
                .Then(c => c.Saga.InventoryReserved = true)
                .Publish(c => new ChargePayment(c.Saga.OrderId))
                .TransitionTo(Charging),
            When(InventoryFailed)
                .Publish(c => new CancelOrder(c.Saga.OrderId))
                .TransitionTo(Cancelled));

        During(Charging,
            When(PaymentCharged)
                .Then(c => c.Saga.PaymentCharged = true)
                .Publish(c => new ScheduleShipping(c.Saga.OrderId))
                .TransitionTo(Shipping),
            When(PaymentFailed)
                .Publish(c => new ReleaseInventory(c.Saga.OrderId))   // compensate
                .Publish(c => new CancelOrder(c.Saga.OrderId))
                .TransitionTo(Cancelled));

        During(Shipping,
            When(ShippingScheduled)
                .Publish(c => new NotifyCustomer(c.Saga.OrderId))
                .Finalize());
    }
}

Idempotency

Each consumer: - Checks if message already processed (inbox table). - Idempotency key on outbound calls (Stripe charge, inventory reservation). - Conditional updates on DB (optimistic concurrency).

Compensations

Failures must be reversed via forward actions: - ReleaseInventory if payment fails. - RefundPayment if shipping fails (rare: typically orders are committed at shipping).

Resilience

Polly resilience pipeline for outbound HTTP (Stripe):

builder.Services.AddHttpClient<StripeClient>()
    .AddStandardResilienceHandler();

Service Bus retry policy: built-in. DLQ after max-retry.

Observability

Correlation ID = OrderId flows through every event.
Distributed traces via OTel; one trace per order shows full pipeline.
Custom metrics: orders/min by status; saga step durations; failure rates.
Alerts: "InventoryFailed rate > 5% in 5 min" → page ops.

Failure modes

Failure	Recovery
API instance crash post-DB-commit	Outbox relay republishes
Inventory worker crash	Service Bus redelivers; idempotent consumer
Payment timeout	Retry with same idempotency key; Stripe dedupes
Saga state lost	Persisted in EF; resumes on restart
Bad event in queue	DLQ; manual fix; replay

Data model

Orders
├── Id (PK), CustomerId, Status, CreatedAt, Total
├── Items[]: SKU, Qty, UnitPrice
└── (event sourcing optional — see RAG case)

OrderEvents (audit / event log)
├── OrderId, EventType, Payload, OccurredAt

Outbox
├── Id, Type, Payload, OccurredAt, SentAt, RetryCount

Inbox (per consumer)
├── MessageId, ConsumedAt

Throughput math

17 orders/s * 5 events = 85 events/s. Service Bus Standard handles 1000s/s easy.

Multi-region

Primary region for writes.
Service Bus geo-DR for outage continuity.
DB read replicas in other regions (eventual reads of order status).

Trade-offs

Choice	Why	Trade-off
Service Bus	Sessions, DLQ, transactions	Cost vs RabbitMQ
Saga (orchestration)	Visible workflow	Coordinator complexity
Outbox	Atomic	2x DB writes
MassTransit	Ecosystem	License (v8+ commercial)

What we'd skip

Event sourcing: not needed for this CRUD-with-events; standard DB + outbox is enough.
Kafka: not needed at this scale; Service Bus richer features.
CQRS: order read = order write (no separate read model).

What we'd add for higher scale

Kafka for higher throughput streaming.
Materialized order views (CQRS) for dashboards.
Event sourcing for audit + replay.
Multi-region active-active for higher SLO.