WebHooks — Sending & Receiving

Key Points

A webhook = HTTP POST from a system to an arbitrary URL when something happens. Sender retries on 5xx; receiver dedups on idempotency keys.
Sender responsibilities: HMAC signature, retry with exponential backoff + jitter, dead-letter after N failures, observability (signed deliveries, latency, status).
Receiver responsibilities: verify signature, validate timestamp (replay window), check idempotency key, return 2xx fast, process async.
Common HMAC pattern (Stripe/GitHub-style): signature = HMAC-SHA256(secret, timestamp + "." + body); sender sends X-Signature: t=…,v1=….
Don't reinvent — for receivers, libraries exist (Stripe.NET, Octokit, custom HMAC verifier ~30 lines). For senders, use Outbox + a simple HTTP-with-Polly worker.

Concepts (deep dive)

Sender side

event happens → write event to DB (and Outbox table) → background worker reads outbox →
HTTP POST to subscriber → success: mark sent | failure: retry with backoff → exhausted: dead-letter

Key components:

public class WebhookSender(IHttpClientFactory clients, ResiliencePipeline pipeline, ILogger<WebhookSender> log)
{
    public async Task<bool> Send(WebhookSubscription sub, WebhookEvent evt, CancellationToken ct)
    {
        var http = clients.CreateClient("webhook");
        var bodyJson = JsonSerializer.Serialize(evt);
        var ts = DateTimeOffset.UtcNow.ToUnixTimeSeconds();
        var sig = ComputeSignature(sub.Secret, ts, bodyJson);

        return await pipeline.ExecuteAsync(async ct =>
        {
            var req = new HttpRequestMessage(HttpMethod.Post, sub.Url)
            {
                Content = new StringContent(bodyJson, Encoding.UTF8, "application/json")
            };
            req.Headers.Add("X-Signature", $"t={ts},v1={sig}");
            req.Headers.Add("X-Idempotency-Key", evt.Id.ToString());
            req.Headers.Add("X-Event-Type", evt.Type);

            using var resp = await http.SendAsync(req, ct);
            return resp.IsSuccessStatusCode;
        }, ct);
    }

    static string ComputeSignature(string secret, long ts, string body)
    {
        using var hmac = new HMACSHA256(Encoding.UTF8.GetBytes(secret));
        var hash = hmac.ComputeHash(Encoding.UTF8.GetBytes($"{ts}.{body}"));
        return Convert.ToHexString(hash).ToLowerInvariant();
    }
}

Receiver side

app.MapPost("/webhooks/orders", async (
    HttpRequest req,
    IConfiguration config,
    IIdempotencyStore idemp,
    IBackgroundJobQueue jobs,
    CancellationToken ct) =>
{
    if (!req.Headers.TryGetValue("X-Signature", out var sigHeader) ||
        !req.Headers.TryGetValue("X-Idempotency-Key", out var keyHeader))
        return Results.Unauthorized();

    using var reader = new StreamReader(req.Body);
    var body = await reader.ReadToEndAsync(ct);

    if (!VerifySignature(config["Webhooks:Secret"]!, sigHeader!, body, maxAgeSeconds: 300))
        return Results.Unauthorized();

    if (!await idemp.TryReserve(keyHeader.ToString(), TimeSpan.FromHours(24), ct))
        return Results.Ok();   // already processed; idempotent 200

    await jobs.Enqueue(new ProcessWebhookJob(body), ct);
    return Results.Ok();        // return fast; process async
});

static bool VerifySignature(string secret, string header, string body, int maxAgeSeconds)
{
    // X-Signature: t=1714250000,v1=abcdef...
    var parts = header.Split(',').Select(p => p.Split('=')).ToDictionary(p => p[0], p => p[1]);
    if (!long.TryParse(parts["t"], out var ts) ||
        DateTimeOffset.UtcNow.ToUnixTimeSeconds() - ts > maxAgeSeconds) return false;

    using var hmac = new HMACSHA256(Encoding.UTF8.GetBytes(secret));
    var expected = Convert.ToHexString(hmac.ComputeHash(Encoding.UTF8.GetBytes($"{ts}.{body}"))).ToLowerInvariant();
    return CryptographicOperations.FixedTimeEquals(
        Encoding.UTF8.GetBytes(parts["v1"]),
        Encoding.UTF8.GetBytes(expected));
}

Key points: - FixedTimeEquals — constant-time comparison, prevents timing attacks. - Timestamp window (5 minutes) — blocks replay attacks beyond a small clock skew. - Idempotency reservation — same key returns success without re-processing. - Return 2xx fast — heavy work moves to a queue.

Retry strategy

ResiliencePipeline pipeline = new ResiliencePipelineBuilder()
    .AddRetry(new RetryStrategyOptions
    {
        MaxRetryAttempts = 5,
        BackoffType = DelayBackoffType.Exponential,
        Delay = TimeSpan.FromSeconds(2),    // 2, 4, 8, 16, 32 s
        UseJitter = true,
    })
    .AddTimeout(TimeSpan.FromSeconds(30))
    .Build();

Common retry windows: 1m → 5m → 30m → 2h → 12h → dead-letter. Bound total retry duration (e.g., 24 h); after that, the receiver is presumed dead.

Dead-letter queue

After max retries, mark the delivery Failed, store the last response, alert. Operators can replay manually after fixing the receiver.

Outbox pattern (sender side)

To guarantee at-least-once delivery without losing events on crash:

// In the same DB transaction as the business event:
db.Orders.Add(order);
db.Outbox.Add(new OutboxItem { Type = "order.created", Payload = JsonSerializer.Serialize(order) });
await db.SaveChangesAsync();

// Background worker reads outbox, posts webhook, marks sent.

See Outbox & Inbox Patterns.

Receiver-side challenge for slow processing

If your receiver's downstream is slow:

Webhook POST → enqueue → return 200 immediately → worker processes from queue

Don't process inline — sender's timeout is usually 10-30 s.

Replay protection

Attacker captures a valid signed webhook on the wire.
Sends it again later → receiver:
  - Signature still valid (HMAC over same payload)
  - But timestamp is older than maxAgeSeconds → reject

Without timestamp validation, replay is trivial.

Verification helpers

For well-known senders (Stripe, GitHub), use their SDKs which know the exact signature format:

// Stripe
var stripeEvent = EventUtility.ConstructEvent(json, signatureHeader, webhookSecret);

For your own platform, document the signature format clearly: which headers, hash algorithm, encoding, what's included in the signed string.

Observability

Sender metrics:
  - delivery success rate per subscription
  - latency to subscriber (p50, p99)
  - retry queue depth
  - dead-letter rate

Receiver metrics:
  - signature verification failures (potential abuse)
  - idempotency duplicate count
  - processing latency

Tag traceparent on the outbound HTTP so distributed tracing crosses the system boundary.

Code: correct vs wrong

❌ Wrong: comparing signatures with `==`

if (computed == provided) /* timing attack possible */

✅ Correct: constant-time

if (CryptographicOperations.FixedTimeEquals(Encoding.UTF8.GetBytes(computed), Encoding.UTF8.GetBytes(provided))) /* ... */

❌ Wrong: processing inline

app.MapPost("/webhook", async (Body b) =>
{
    await DoExpensiveWork(b);     // takes 90 s; sender times out at 30 s
    return Results.Ok();
});

✅ Correct: enqueue and ack

app.MapPost("/webhook", async (Body b, IBackgroundJobQueue q) =>
{
    await q.Enqueue(new ProcessJob(b));
    return Results.Ok();
});

❌ Wrong: no replay protection

// signature only — no timestamp check → captured payload is replayable forever

✅ Correct: signed timestamp

// signature includes timestamp; receiver rejects if timestamp older than 5 min

Design patterns for this topic

Pattern 1 — Subscription management API

POST /subscriptions    { url, eventTypes, secret? }   → returns id + secret
PUT  /subscriptions/{id}/rotate-secret
DELETE /subscriptions/{id}

Each subscriber has a unique secret stored encrypted at rest.

Pattern 2 — Per-event-type filter on subscription

class WebhookSubscription { public string[] EventTypes; }

// Sender enumerates subscriptions interested in evt.Type
foreach (var sub in subs.Where(s => s.EventTypes.Contains(evt.Type)))
    queue.Enqueue(new SendJob(sub, evt));

Pattern 3 — Visible delivery log

GET /subscriptions/{id}/deliveries
   { id, eventType, timestamp, status, attempts, lastResponse }

Customers can self-diagnose without contacting support.

Pattern 4 — Manual replay UI

POST /deliveries/{id}/replay   → re-queue a previously failed delivery

After fixing receiver-side bug.

Pattern 5 — Skew-tolerant timestamp window

// Allow ±5 minutes for clock drift between sender/receiver
const int MaxSkewSeconds = 300;
if (Math.Abs(now - ts) > MaxSkewSeconds) return Unauthorized();

Pros & cons / trade-offs

Aspect	WebHooks	Polling	Message bus
Setup	Customer's URL	Customer's job	Both apps share infra
Real-time	Yes	No (poll interval)	Yes
Customer side complexity	Public URL + signature verify	API client	Bus client + auth
Reliability	Retries + DLQ	Customer-driven	Built-in
Cost	Per delivery	Per poll	Per message

When to use / when to avoid

Use webhooks for B2B integrations, payment platforms, GitOps notifications, customer-facing automation.
Avoid when you control both ends and a message bus is available — buses give better guarantees.
Avoid for high-frequency events (>100 RPS per subscriber) — bus or streaming better.

Interview Q&A

Q1. How do you authenticate webhook deliveries? A. HMAC signature: HMAC-SHA256(secret, timestamp + "." + body) in a header. Receiver recomputes and compares using constant-time comparison.

Q2. Why include a timestamp? A. Prevents replay attacks. The receiver rejects deliveries older than a small window (5 minutes typical), so a captured signed payload can't be replayed later.

Q3. How do you make webhook delivery reliable? A. Outbox pattern at sender (write event in same transaction as business data); background worker retries with exponential backoff + jitter; dead-letter queue after exhausting retries.

Q4. What's idempotency in webhook receivers? A. Each delivery has a unique ID; the receiver records processed IDs (with TTL) and returns 200 on duplicates without re-processing. Senders may retry a successful delivery if the response wasn't received; idempotency makes this safe.

Q5. Why return 2xx fast? A. Senders treat slow responses as failures (timeouts ~10-30 s). Doing real work inline causes unnecessary retries. Enqueue + ack is the standard pattern.

Q6. How do Stripe/GitHub format their signatures? A. Stripe: Stripe-Signature: t=...,v1=... (HMAC-SHA256 of t.body). GitHub: X-Hub-Signature-256: sha256=... (HMAC-SHA256 of body). Different formats; use their SDK if integrating.

Q7. What's a webhook DLQ? A. Dead-letter queue: deliveries that exceeded retry policy land here for manual investigation. Operators replay after fixing receiver, or notify the customer.

Q8. How do you scale webhooks horizontally? A. Outbox + many sender workers polling the queue with leases (one delivery per worker). Spread retries across workers. Keep per-subscriber concurrency limits to avoid hammering slow customers.

Gotchas / common mistakes

Plain string comparison for signatures — timing attack.
No timestamp validation — replay attack window is forever.
Processing inline — sender timeout, retries, duplicate work.
No idempotency at receiver — duplicate processing on retry.
Storing secrets in plaintext — encrypt at rest, rotate on rotation API.
Unbounded retries — keep delivering forever to a dead receiver.
Tiny timestamp window (<30 s) — clock drift causes false rejects.
Logging full body with signatures — log size + secret leakage risk.
No subscription pause — bad receiver brings down sender's queue throughput.