Messaging Fundamentals

Key Points

• At-most-once / at-least-once / exactly-once are the three delivery guarantees. Most brokers offer at-least-once by default. Exactly-once is mostly a marketing claim — practical systems use at-least-once + idempotent consumers.
• Idempotency = consuming the same message N times yields the same outcome as 1 time. Achieved via dedup keys, conditional writes, or natural idempotence.
• Ordering is per-partition (Kafka), per-queue/session (Service Bus), or none (broadcast). Total ordering is rare and expensive.
• Backpressure prevents producers from overwhelming consumers. Bounded queues + flow control.
• Poison messages — bad payloads that crash the consumer. Need a dead-letter queue (DLQ) and retry-with-backoff.

Concepts (deep dive)

Delivery guarantees

At-least-once + idempotent consumer is the practical "exactly-once" pattern.

Idempotency

Process the same message twice → same result. Strategies:

// 1. Dedup table (idempotency key)
public async Task Handle(OrderPlaced msg)
{
    if (await _db.ProcessedMessages.AnyAsync(p => p.Id == msg.Id)) return;
    // ... do work ...
    _db.ProcessedMessages.Add(new() { Id = msg.Id });
    await _db.SaveChangesAsync();   // dedup + work in one tx
}

// 2. Conditional write (optimistic)
await _db.ExecuteSqlInterpolatedAsync(
    $"UPDATE orders SET status = {newStatus}, version = version + 1 WHERE id = {id} AND version = {expected}");

// 3. Natural idempotence
await _redis.StringSetAsync("order:" + id, json);   // SET is idempotent

Ordering

Kafka:  ordered per partition  (key → partition → ordered)
Service Bus: ordered per session
RabbitMQ: ordered per queue (single consumer)

Don't assume global ordering. Use partitioning by key (CustomerId, OrderId) so related events stay ordered.

Patterns

Pub/Sub

Publisher → Topic → Sub1, Sub2, Sub3 (each gets a copy)

Multiple consumers process the same event independently. Decouple producers from consumers.

Competing consumers

Producer → Queue → Worker1, Worker2, Worker3 (each msg goes to ONE worker)

Scale horizontally; distribute load.

Request-Reply

Caller → Request Queue → Service
Service → Reply Queue (correlated by ID) → Caller

Async RPC over messaging. MassTransit/Wolverine support natively.

Poison messages

Message → consumer → throws → reject → broker redelivers → throws → ...

Solutions: 1. Retry with backoff: 1s, 5s, 30s, then DLQ. 2. DLQ (Dead-Letter Queue): persistent store for failed messages. Manual intervention. 3. Replay: after fix, replay from DLQ.

// MassTransit retry policy
cfg.UseMessageRetry(r => r.Intervals(TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(30)));
cfg.UseInMemoryOutbox();
cfg.UseDelayedRedelivery(r => r.Intervals(TimeSpan.FromMinutes(5), TimeSpan.FromMinutes(15), TimeSpan.FromHours(1)));

Backpressure

var channel = Channel.CreateBounded<Msg>(new BoundedChannelOptions(1000)
{
    FullMode = BoundedChannelFullMode.Wait   // producer awaits when full
});

Bounded buffers + flow control. Don't let producers run unbounded — eventually OOM.

In broker-land, prefetch limits do the same:

// RabbitMQ
channel.BasicQos(0, prefetchCount: 50, global: false);   // max 50 unacked at once

Message versioning

Schema evolution is hard. Strategies:

1. Add fields, don't remove (forward-compatible).
2. Version in topic name (orders.placed.v2).
3. Schema registry (Confluent Schema Registry, Apicurio) — central typed schemas; producers/consumers compatible.

Outbox pattern

Solving the "did the DB commit AND the message send happen atomically?" problem. Covered in Outbox & Inbox Patterns.

Choosing topology

Messaging vs HTTP

For event-driven, async, reliable communication: messaging. For request-response, low-latency: HTTP.

Event vs Command

• Command (PlaceOrder): one consumer; expects action. Imperative.
• Event (OrderPlaced): many consumers; "this happened". Past-tense; descriptive.

Different topics; different semantics.

Eventual consistency

Messaging implies async. Read-your-writes consistency requires extra plumbing (read-from-write side, or wait for processing). Design UI to show "pending" state.

Distributed transactions

XA / 2PC across DB + broker is dead. Modern: outbox pattern.

1. App writes to DB (business data + outbox row) — single transaction.
2. Background process polls outbox → publishes → marks sent.

Covered in Outbox & Inbox Patterns.

Saga vs orchestration

Multi-step distributed workflows. Covered in Sagas: Orchestration vs Choreography.

CQRS + events

Read side often built from events. Append events to a stream; project into read models. Event sourcing when the events ARE the state.

Headers and correlation

Always include: - MessageId — unique per send (idempotency key). - CorrelationId — links to a request/saga. - traceparent — distributed tracing. - Source — producing service. - Type — event type / command name.

Schema considerations

• JSON — readable; verbose. Default for HTTP-style APIs.
• Protobuf — compact; typed; schema evolution rules. Default for gRPC.
• Avro — compact; schema registry. Default for Kafka.
• MessagePack — JSON-like; smaller. .NET-friendly.

Code: correct vs wrong

❌ Wrong: non-idempotent consumer

public async Task Handle(OrderPlaced msg)
{
    _db.Orders.Add(new() { Id = msg.OrderId, ... });   // duplicate row on redelivery
    await _db.SaveChangesAsync();
}

✅ Correct: dedup + tx

public async Task Handle(OrderPlaced msg)
{
    if (await _db.ProcessedMessages.AnyAsync(p => p.Id == msg.MessageId)) return;
    _db.Orders.Add(...);
    _db.ProcessedMessages.Add(new() { Id = msg.MessageId });
    await _db.SaveChangesAsync();
}

❌ Wrong: assuming global ordering

// Two events, processed by different consumers, may reorder.

✅ Correct: partition by key

producer.Send(new Msg { Key = order.CustomerId, ... });   // Kafka
// Or session-id on Service Bus

Design patterns for this topic

Pattern 1 — "At-least-once + idempotent consumer"

• Intent: practical exactly-once.

Pattern 2 — "Pub/Sub for events; competing consumers for work"

• Intent: match topology to need.

Pattern 3 — "DLQ + replay"

• Intent: don't lose poison messages.

Pattern 4 — "Bounded buffer + prefetch"

• Intent: backpressure.

Pattern 5 — "Outbox over distributed transactions"

• Intent: atomic DB + send.

Pros & cons / trade-offs

When to use / when to avoid

• Use messaging for async, decoupled flows.
• Use HTTP for request-response with low latency.
• Avoid XA/2PC.
• Avoid assuming exactly-once delivery.

Interview Q&A

Q1. Three delivery guarantees? At-most-once, at-least-once, exactly-once.

Q2. Why is "exactly-once" suspicious? End-to-end exactly-once requires consumer + side-effects atomicity. Most "exactly-once" features apply only within a broker. Practical pattern: at-least-once + idempotent consumer.

Q3. Idempotency strategies? Dedup table, conditional writes, natural idempotence (set, upsert).

Q4. How preserve ordering? Partition by key. Kafka: same key → same partition. Service Bus: sessions.

Q5. Poison message — what do? Retry with backoff; DLQ; manual fix; replay.

Q6. Pub/Sub vs competing consumers? Pub/Sub: every consumer gets a copy. Competing: one consumer per message.

Q7. Backpressure? Bounded buffer + flow control. Producer awaits when buffer full.

Q8. Outbox pattern goal? Atomic DB + message-emit. No distributed transaction.

Q9. Event vs command? Event: past-tense; many consumers. Command: imperative; one consumer.

Q10. Schema evolution? Add-only fields; version in topic; schema registry.

Q11. Why messaging over HTTP? Loose coupling; built-in retry; backpressure; broadcast support.

Q12. Eventual consistency UX? Show "pending" state; refresh via polling/events. Don't promise immediate consistency.

Gotchas / common mistakes

• ⚠️ Non-idempotent consumers — duplicate side effects on redelivery.
• ⚠️ Assuming global ordering — partition by key.
• ⚠️ No DLQ — poison messages stuck in retry forever.
• ⚠️ Unbounded buffers — OOM.
• ⚠️ Distributed transaction (XA) — replace with outbox.

Further reading

• Microsoft eBook: Asynchronous messaging patterns
• Confluent: at-least-once vs exactly-once
• Enterprise Integration Patterns (Hohpe & Woolf)
• Jimmy Bogard: Reliable Messaging