Idempotency Keys — Deep

Key Points

• At-least-once is the rule, not the exception. Network retries, client retries, message redelivery — every non-idempotent endpoint will be called twice eventually.
• The standard is the Idempotency-Key header (popularized by Stripe). Never put the key in the body — middleware can't see it cleanly, and bodies are diverse.
• Server stores the response, not just "seen this key". On replay, return the exact same status, headers, body. Anything less breaks clients (e.g., missing Location header on 201).
• Two storage choices: Redis with TTL (cheap, fast, ephemeral) or a durable table (idempotency_responses) inside the same DB transaction as the side effect. The durable approach is the only way to make replay-safe writes truly safe.
• Concurrency: two requests with the same key can land at once. First wins; second waits or returns 409 / 425. SETNX-style "claim or fail" is the primitive.
• TTL: 24h is canonical (Stripe's default). Tune to your retry budget.

Concepts (deep dive)

Why idempotency

Without idempotency keys:

client POST /payments {amount=100}   ──► server charges $100, returns 200
   ↑ network blip; client doesn't see 200
client retries POST /payments {amount=100}  ──► server charges $100 AGAIN
result: $200 charged, customer angry

With idempotency keys:

client POST /payments  Idempotency-Key: 7f3a…  ──► server charges, stores response by key
client retries (same key) ──► server returns the stored response; no duplicate charge

GETs are idempotent by definition. The interesting cases are POST / PATCH / DELETE on resources whose action has external side effects (charge a card, send an email, post to a queue).

Where the key lives

Stripe convention: Idempotency-Key: <opaque> HTTP header. Why header, not body:

• Middleware (ASP.NET Core, gateway, mesh) can read headers without parsing the body.
• Logs and traces capture headers naturally.
• Body shape varies per endpoint; header is uniform.
• Streaming requests have no parseable "body" pre-consumption.

Key generation: client picks a UUID (or hash of the logical request). The server treats it as opaque.

POST /payments HTTP/1.1
Host: api.contoso.com
Idempotency-Key: 7f3a8d2c-91b1-4f0e-bb22-3c7c5e1a9d0f
Content-Type: application/json

{"amount": 100, "currency": "usd"}

Server-side mechanics

Two storage choices.

Redis with TTL (ephemeral, fast)

SET idem:{key} <serialized response> NX EX 86400

NX (only-if-not-exists) is the atomic claim. EX 86400 = 24h TTL.

Pros: fast (~ms), cheap, simple. Cons: not transactional with the DB write — if Redis says "claimed" but DB write fails, you've stored a bad response.

Durable table (transactional, safe)

CREATE TABLE idempotency_responses (
    key            TEXT PRIMARY KEY,
    request_hash   TEXT NOT NULL,
    status_code    INT  NOT NULL,
    response_body  JSONB NOT NULL,
    headers        JSONB NOT NULL,
    created_at     TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX idx_idem_created_at ON idempotency_responses(created_at);

The handler runs the business write and the response insert in the same transaction:

BEGIN;
INSERT INTO payments(...) VALUES (...);
INSERT INTO idempotency_responses(key, status_code, response_body, headers) VALUES (...);
COMMIT;

Replay: read by key, return verbatim.

This is the only way to guarantee the stored response and the side effect are coherent.

Idempotency-Key vs request fingerprint

What if the client retries with the same key but a different body? That's a client bug, but the server has to defend.

client A: POST /payments  Key=abc  body={amount: 100}    ──► OK, stored
client A: POST /payments  Key=abc  body={amount: 1000}   ──► what now?

Two doctrines:

• Key trust: same key → same response, ignore the body. Simple, requires client cooperation.
• Fingerprint defense: server hashes (method + path + canonical body) and stores it. Replay with same key but different fingerprint → 422 (Unprocessable Entity) with idempotency_key_reused_with_different_request.

Stripe does fingerprinting. So should you.

var fingerprint = Sha256(
    method + "\n" + path + "\n" + canonicalBody);

Returning the cached response

The replay must be byte-for-byte equivalent enough that clients can't tell the difference. That means:

• Same status code (200 vs 201 matters).
• Same body.
• Same critical headers — Location, ETag, custom X-Request-Id, Retry-After.

// ❌ wrong: just store the body
await cache.SetStringAsync(key, body);

// ✅ right: store status + headers + body
var snapshot = new ResponseSnapshot(StatusCode, Headers, Body);
await cache.SetStringAsync(key, JsonSerializer.Serialize(snapshot), ttl);

A 201 Created with Location: /payments/123 becoming a 200 OK on replay = client misses the new URI.

TTL choice

Stripe: 24h. Why 24h:

• Long enough to cover most retry budgets (network outages, mobile reconnect storms).
• Short enough that storage doesn't grow forever.

If your client retries for 5 minutes, 1h is fine. If your batch job re-drives failed messages 3 days later, you need ≥3d. Match the TTL to the upper bound of the retry window.

Concurrency: two requests, same key, at once

t=0   request 1 arrives, claims key (SETNX) → executes
t=10ms request 2 arrives, claim fails (key exists, response not yet stored)

Three options for request 2:

1. Wait and return the eventual result. Most user-friendly. Hold the request, poll Redis / DB for the response, return when stored. Bound the wait.
2. Return 409 Conflict / 425 Too Early with a hint: "request still processing, retry in N seconds". Simplest server-side; harder for clients.
3. Treat the second as a fresh execution if the first looks dead. Risky — defeats the point. Only valid with a heartbeat-based lock.

key state machine:
   absent ──claim──► claimed (no response yet) ──store──► completed
                          │
                          └─ second request: 425/409 OR wait
   completed ──TTL──► absent again (allow new attempts)

Idempotency for non-mutating GETs

Already idempotent semantically. Don't bother with keys. Use HTTP caching (ETag, If-None-Match) for efficiency, not idempotency.

The interesting cases are POST / PATCH / DELETE. (Yes, DELETE is technically idempotent in HTTP semantics, but real systems often have side effects — webhook fires, email sent — that need keys.)

Idempotency + optimistic concurrency (ETag / If-Match)

These solve different problems:

• Idempotency-Key: "I'm retrying the same intended write."
• If-Match: "I'm intending a write conditional on the resource version."

Combine them on PATCH:

PATCH /orders/123 HTTP/1.1
If-Match: "v7"
Idempotency-Key: 4a2…
Content-Type: application/json

{"status":"shipped"}

If-Match prevents lost-update. Idempotency-Key prevents duplicate-apply on retry. They're orthogonal.

Failure modes — the critical one

server: handle request  ──► write to DB ✓ ──► server crashes ✗
                                                        ▲
                                              response was being written
client retries same key
server: key not in store (write didn't complete) ──► re-executes ──► side effect TWICE

The cure: write the side effect and the idempotency record in one transaction, with the outbox pattern for any external side effects (queue publish, email send).

BEGIN;
INSERT INTO orders(...) VALUES (...);
INSERT INTO outbox(event, payload) VALUES ('OrderCreated', ...);
INSERT INTO idempotency_responses(key, ...) VALUES (...);
COMMIT;
-- separate worker reads outbox and publishes; idempotent at consumer

Now: either everything committed (replay returns stored response, outbox eventually delivers), or nothing committed (replay re-executes safely).

Combination with at-least-once messaging

Same idea applies to consumers. Queue redeliveries → consumer must dedupe. Use the message ID as the idempotency key:

public async Task Handle(OrderPlaced msg)
{
    if (await _dedup.AlreadyProcessed(msg.Id)) return;
    await _db.SaveOrderAsync(msg);
    await _dedup.MarkProcessed(msg.Id, ttl: TimeSpan.FromDays(7));
}

Same transactional rule: dedup mark + side effect in one transaction.

Code: ASP.NET Core middleware sketch

public class IdempotencyMiddleware
{
    private readonly RequestDelegate _next;
    private readonly IDistributedCache _cache;

    public IdempotencyMiddleware(RequestDelegate next, IDistributedCache cache)
    {
        _next = next; _cache = cache;
    }

    public async Task InvokeAsync(HttpContext ctx)
    {
        if (!HttpMethods.IsPost(ctx.Request.Method) && !HttpMethods.IsPatch(ctx.Request.Method))
        { await _next(ctx); return; }

        if (!ctx.Request.Headers.TryGetValue("Idempotency-Key", out var keyVals))
        { await _next(ctx); return; }

        var key = "idem:" + keyVals.ToString();
        var existing = await _cache.GetAsync(key);
        if (existing is not null)
        {
            var snap = JsonSerializer.Deserialize<ResponseSnapshot>(existing)!;
            ctx.Response.StatusCode = snap.StatusCode;
            foreach (var (h, v) in snap.Headers) ctx.Response.Headers[h] = v;
            await ctx.Response.Body.WriteAsync(snap.Body);
            return;
        }

        // Claim
        var claimed = await TryClaim(key);
        if (!claimed)
        {
            ctx.Response.StatusCode = StatusCodes.Status425TooEarly;
            return;
        }

        // Capture response
        var origBody = ctx.Response.Body;
        await using var buf = new MemoryStream();
        ctx.Response.Body = buf;

        await _next(ctx);

        buf.Position = 0;
        var snapshot = new ResponseSnapshot(
            ctx.Response.StatusCode,
            ctx.Response.Headers.ToDictionary(h => h.Key, h => h.Value.ToString()),
            buf.ToArray());

        await _cache.SetAsync(key, JsonSerializer.SerializeToUtf8Bytes(snapshot),
            new() { AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(24) });

        buf.Position = 0;
        await buf.CopyToAsync(origBody);
    }

    private async Task<bool> TryClaim(string key) => /* SETNX equivalent */ true;
    record ResponseSnapshot(int StatusCode, Dictionary<string,string> Headers, byte[] Body);
}

This is the simplified shape. Production needs: - Fingerprint validation. - "Still processing" state separate from "completed". - DB transaction integration (not just Redis) for true safety. - Polly around the cache for resilience.

How it works under the hood

State machine

                    ┌──────────────┐
   first request ──►│  CLAIMED     │  (lock entry exists, no body yet)
                    └──────┬───────┘
        success / failure  │
                           ▼
                    ┌──────────────┐
                    │  COMPLETED   │  (status + headers + body stored)
                    └──────┬───────┘
                           │  TTL
                           ▼
                    ┌──────────────┐
                    │  ABSENT      │  (key forgotten, fresh attempts allowed)
                    └──────────────┘

   concurrent request when CLAIMED → wait or 425
   any request when COMPLETED      → return stored response

SETNX vs Lua

SET key val NX EX 86400 claims the key atomically. If you also want to handle "completed" vs "claimed-but-running" with one round trip, a Lua script:

local v = redis.call('GET', KEYS[1])
if v == false then
    redis.call('SET', KEYS[1], 'CLAIMED', 'EX', ARGV[1])
    return 'CLAIMED'
elseif v == 'CLAIMED' then
    return 'IN_PROGRESS'
else
    return v
end

Storage layout

Redis:
  idem:7f3a-…  →  { state: "completed", status: 200, headers: {...}, body: "..." }   TTL=24h

Postgres:
  idempotency_responses (key PK, request_hash, status_code, response_body jsonb, headers jsonb, created_at)
  partial index on (key) WHERE created_at > now() - interval '24 hours'

A nightly job purges entries older than the TTL.

Code: correct vs wrong

❌ Wrong: idempotency key in body

{ "idempotency_key": "abc", "amount": 100 }

Middleware can't see it; bodies vary per endpoint; logs miss it.

✅ Correct: header

Idempotency-Key: abc

❌ Wrong: store only the body

await cache.SetStringAsync(key, responseBody);
// Replay returns 200 even if first call returned 201; Location header lost

✅ Correct: store status + headers + body

await cache.SetAsync(key, JsonSerializer.SerializeToUtf8Bytes(
    new ResponseSnapshot(statusCode, headers, body)), ttl);

❌ Wrong: separate idempotency store and DB write

await db.SaveAsync(payment);          // committed
await redis.SetAsync(key, response);  // crashes here → next retry duplicates payment

✅ Correct: one transaction (or outbox)

using var tx = db.Database.BeginTransaction();
db.Payments.Add(payment);
db.IdempotencyResponses.Add(new(key, statusCode, headers, body));
await db.SaveChangesAsync();
tx.Commit();

❌ Wrong: ignore body fingerprint

Same key, different body → server replays old response → silent client confusion

✅ Correct: validate fingerprint

if (stored.Fingerprint != Sha256(method + path + body))
    return Results.UnprocessableEntity("idempotency_key_reused_with_different_request");

Design patterns for this topic

Pattern 1 — "Header-based idempotency key (Stripe convention)"

• Intent: uniform, middleware-visible, client-driven.

Pattern 2 — "Snapshot-the-whole-response"

• Intent: byte-for-byte replay; clients don't notice.

Pattern 3 — "DB transaction with idempotency row"

• Intent: side effect and dedup record commit together.

Pattern 4 — "Outbox under same transaction"

• Intent: safe external publish on retried writes.

Pattern 5 — "Fingerprint validation"

• Intent: defend against client misuse of keys.

Pros & cons / trade-offs

When to use / when to avoid

• Use on every non-idempotent public endpoint that has external side effects (charge, email, queue publish, order creation).
• Use message IDs as keys for queue consumers.
• Use combined with outbox for guaranteed coherence.
• Avoid body-based key location.
• Avoid Redis-only storage if a duplicate side effect would be costly — durable + transactional store wins.
• Avoid silently overwriting a stored response when fingerprint mismatches — return 422.

Interview Q&A

Q1. Why are idempotency keys necessary? At-least-once delivery. Network retries, client retries, queue redeliveries. Without keys, retries duplicate side effects.

Q2. Where should the key live? HTTP header, conventionally Idempotency-Key. Header is middleware-visible, log-friendly, body-shape-independent.

Q3. What does the server store? The full response — status code, critical headers, body — plus a state (claimed / completed) and optionally a request fingerprint.

Q4. Idempotency-Key vs request fingerprint? The key is the client's contract for "this is the same logical request". The fingerprint defends against client bugs that reuse a key with different bodies.

Q5. Why store the response, not just "seen this key"? Replays must look identical. A 201 with Location becoming a 200 with no header breaks clients.

Q6. What TTL? 24h is the canonical default (Stripe). Match the upper bound of your retry budget.

Q7. Two concurrent requests with the same key — what happens? First claims the key; second waits or gets 409/425. Use SETNX or a Lua script to do the claim atomically.

Q8. Why a DB-backed store and not just Redis? To make the side effect and the dedup record commit atomically. Redis is fast but not transactional with the DB.

Q9. How do you keep external publishes idempotent on retried writes? Outbox pattern. Insert the outbox row in the same transaction as the business write; a separate worker publishes and dedupes at the consumer.

Q10. Combining with optimistic concurrency? Idempotency-Key handles retries. If-Match handles version control. Use both on conditional updates.

Q11. Idempotency for queue consumers? Use the message ID as the key. Mark "processed" + side effect in one transaction. Idempotent processing = at-least-once becomes effectively-once.

Q12. Common pitfalls? Storing only the body, not status / headers. Redis-only store with non-transactional writes. Reusing keys across different requests. Letting "claimed" entries hang forever — need timeout / cleanup.

Gotchas / common mistakes

• ⚠️ Idempotency key in the body — middleware can't see it cleanly.
• ⚠️ Storing only the body — status / headers diverge on replay.
• ⚠️ Redis claim succeeds, DB write fails — without transactions, you've lied to the client.
• ⚠️ No fingerprint check — client reuses key with different body, server returns wrong cached response.
• ⚠️ TTL too short — slow retries miss the dedup window.
• ⚠️ TTL too long — storage grows; old keys collide.
• ⚠️ No cleanup of "claimed but never completed" entries — second request hangs forever.
• ⚠️ External side effects (email, webhook) without outbox — retries duplicate them even with the key.
• ⚠️ Treating GETs as needing keys — wasted effort; HTTP semantics already cover them.

Further reading

• Stripe: Idempotent requests
• IETF draft: HTTP Idempotency-Key Header
• Brandur Leach: Idempotency keys in API design
• MS Learn: Idempotent operations in messaging