Distributed Locks

Key Points

Distributed locks are dangerous and almost always weaker than they look. Every implementation makes assumptions about clocks, network delivery, and process pause that distributed systems routinely violate.
Martin Kleppmann's RedLock critique is required reading. The fix isn't "don't use Redis"; it's "always pair the lock with a fencing token the protected resource validates."
Mechanism by use case: DB row locks (in-transaction, no orphans), DB advisory locks (named, tied to session), Redis SET NX EX (fast, ephemeral), ZooKeeper / etcd leases (strong consistency), Azure Blob Lease (cloud-native leader election).
A fencing token is a monotonically increasing number returned with every lock acquisition. Clients pass it to the protected resource; the resource accepts only the highest token seen. Survives a paused-then-resumed lock holder.
Often you shouldn't use a distributed lock at all. Partition-by-key, idempotency + dedup, or DB-level serialization are cheaper, simpler, and harder to misuse.
.NET Distributed Lock (madelson) wraps SQL Server / Postgres / Redis / Azure / file-system locks behind one API and is the practical default for .NET shops.

Concepts (deep dive)

Why distributed locks are dangerous

A distributed lock answers "only one process at a time can do X". On a single machine, lock / Mutex / Monitor works because the OS sees every process and can stop one. Across machines, nothing has that authority — and every "lock" is a lease on faith that the holder is still alive and responsive.

The three faults that bite every implementation:

Clock skew — TTLs assume "10 seconds from now" means the same thing on the holder and the lock service. NTP slips happen.
Process pause — a 30-second GC pause, container freeze, or VM live-migration can suspend the lock holder past its TTL without it noticing.
Network partitions — the holder thinks it has the lock; the service has expired it; both believe themselves correct.

The senior insight: a distributed lock is not a lock, it's a lease + a hope that the holder notices when it loses the lease. The fix: don't trust the lease at the protected resource; trust a fencing token.

The pause-and-resume failure scenario

Time   Holder A                Lock Service          Holder B            Resource (DB)
────   ─────────                ────────────          ─────────           ──────────────
t=0    acquires lock(ttl=10s)
t=1    starts work
t=2    [GC pause: 12s] ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
t=10                          lease EXPIRES
t=11                          A acquires lock(ttl=10s)
t=12                                                  starts work
t=13                                                  writes V_B  ──────► resource ← V_B
t=14   [resumes!]
t=15   thinks it still owns the lock
t=16   writes V_A           ──────────────────────────────────────────► resource ← V_A
                                                                          (overwrites B's work)

Without fencing tokens this is silent corruption. With them, the resource rejects V_A's write because A's token is lower than B's.

Patterns by mechanism

Database row-level lock — `SELECT ... FOR UPDATE`

The default choice when the DB is already in your architecture. The transaction holds the row lock; commit/rollback releases it. Crash → DB recovery releases. No orphan locks.

-- Postgres
BEGIN;
SELECT * FROM jobs WHERE id = 'midnight-rollup' FOR UPDATE;
-- ... do the work ...
COMMIT;

-- SQL Server
BEGIN TRAN;
SELECT * FROM jobs WITH (UPDLOCK, HOLDLOCK, ROWLOCK)
WHERE id = 'midnight-rollup';
COMMIT;

Pros: serialization is the DB's job; lock dies with the transaction; reliable; observable in DMVs / pg_locks. Cons: slow under high contention; ties up a DB connection for the work duration; doesn't scale beyond DB throughput.

Database advisory locks

Named locks not tied to a row. Postgres: pg_advisory_lock(key). SQL Server: sp_getapplock. Released at session/transaction end (your choice).

-- Postgres: hold for transaction
SELECT pg_advisory_xact_lock(hashtext('migration-runner'));
-- do work
COMMIT;  -- lock released

-- SQL Server: transaction-scoped
EXEC sp_getapplock @Resource = 'migration-runner',
                   @LockMode = 'Exclusive',
                   @LockOwner = 'Transaction';

The classic use: "only one instance runs the schema migration at startup." DB connection holds the advisory lock; if the instance crashes, the connection drops, the lock releases. Other instances waiting on the lock acquire and discover the migration already happened.

Redis-based locks (RedLock)

SET lock:job123 <random-token> NX EX 30
-- on success: you own it for 30s
-- do work
-- to release: ONLY if we still own it (Lua script for atomicity)

Single-instance Redis lock is probably fine for low-stakes coordination ("which instance sends the daily summary email"). The Redis docs propose RedLock — a multi-instance algorithm acquiring quorum across N independent Redis instances — for stronger guarantees. Martin Kleppmann's critique showed RedLock's safety claims rely on synchronized clocks and bounded process pauses, neither of which hold in real systems.

Atomic release in Lua (release-only-if-mine):

-- KEYS[1] = lock key, ARGV[1] = our random token
if redis.call("GET", KEYS[1]) == ARGV[1] then
    return redis.call("DEL", KEYS[1])
else
    return 0
end

Without this script, a process that paused past its TTL might DEL a lock another process now owns.

For long-running work: lease renewal. Periodically SET lock:job123 <token> XX EX 30 to extend, but watch out — if the renewal fails (network blip), you've lost the lock and don't know it. Either crash the work or have the resource validate via fencing token.

Fencing tokens

The senior's required addition — and the part most off-the-shelf "distributed lock" wrappers omit.

Lock Service                                         Resource (DB / Service)
─────────────                                         ────────────────────────
acquire(key) → token=42                              accepts writes only with token ≥ max_seen
acquire(key) → token=43  (after 42 expired)

Every acquisition returns a monotonically increasing token. Clients send the token with every protected operation. The resource keeps max_token_seen and rejects anything lower.

// Resource side
public async Task ApplyChange(Change c, long fencingToken)
{
    var max = await _store.GetMaxToken(c.Resource);
    if (fencingToken < max)
        throw new StaleLockException($"token {fencingToken} < max seen {max}");
    await _store.UpdateMaxToken(c.Resource, fencingToken);
    await _store.Apply(c);
}

Now Holder A's pause-and-resume write fails because B's token (43) is already recorded. No silent corruption.

Verify before trusting: most "Redis distributed lock" libraries do not return fencing tokens. ZooKeeper and etcd do (zxid / mod_revision).

ZooKeeper / etcd-based locks

Strong-consistency-backed leases. ZooKeeper's recipes (Curator's InterProcessSemaphoreMutex) and etcd's Lease + Election APIs give you fencing-token-equivalent guarantees because every operation has a monotonic version (zxid / revision) the protected resource can use.

Higher operational cost — you're running Zoo or etcd. Rare in pure .NET shops; more common when Kubernetes is heavy or there's polyglot infrastructure.

Azure Blob Lease

A blob can be leased for 15-60 seconds (or infinite, with explicit renewal). Lease ownership is enforced by Azure Storage; only the lease holder can write. Auto-renewal is the holder's job.

var blob = container.GetBlobClient("leader.lock");
var lease = blob.GetBlobLeaseClient();
var leaseId = (await lease.AcquireAsync(TimeSpan.FromSeconds(60))).Value.LeaseId;

// Periodic renewal (every ~30s)
await lease.RenewAsync();

// Use leaseId on writes that require leadership
await blob.UploadAsync(content, conditions: new BlobRequestConditions { LeaseId = leaseId });

This is the "leader election" primitive used by Cosmos DB Change Feed Processor and EventProcessorClient under the hood. Cross-link: see Azure Storage.

Cosmos DB optimistic concurrency / DistributedLock pattern

A single document acts as the lock; _etag provides CAS.

// Acquire: read, set OwnerId + ExpiresAt, replace with ETag
var doc = await container.ReadItemAsync<LockDoc>(id, partition);
if (doc.Resource.ExpiresAt > DateTime.UtcNow && doc.Resource.OwnerId != me)
    throw new LockHeldException();

doc.Resource.OwnerId = me;
doc.Resource.ExpiresAt = DateTime.UtcNow.AddSeconds(30);
await container.ReplaceItemAsync(doc.Resource, id, partition,
    new ItemRequestOptions { IfMatchEtag = doc.ETag });   // 412 if someone raced us

Same pattern works in any document store with ETag/CAS semantics. _ts or a counter inside the doc serves as a poor-man's fencing token.

.NET libraries

DistributedLock (madelson) — the well-respected unified API. Wraps SQL Server, Postgres, MySQL, Redis, Azure, file-system, and ZooKeeper providers. Idiomatic using syntax:

var @lock = new SqlDistributedLock("migration-runner", connStr);
await using (var handle = await @lock.AcquireAsync(timeout: TimeSpan.FromSeconds(30)))
{
    // run migration
} // disposed → lock released

AcquireAsync blocks (with a timeout) until acquired; TryAcquireAsync returns null immediately if held.

Native client libraries: - StackExchange.Redis — for raw SET NX EX patterns. - Npgsql — SELECT pg_advisory_lock(...) directly. - Microsoft.Data.SqlClient — sp_getapplock via SqlCommand. - Azure.Storage.Blobs — BlobLeaseClient.

⚠️ Note: As of 2026, a "Microsoft.Extensions.DistributedLock" hasn't shipped as a first-party Microsoft package; the community library above is the de-facto standard. Verify current state of any abstraction at write time.

When NOT to use a distributed lock

The senior's "should I lock?" decision tree:

Can I partition the work by key so only one node ever owns each key?
   yes → done. No lock needed. (Best.)
   no  ↓

Is the operation idempotent? (See idempotency-keys-deep.md)
   yes → at-least-once + dedup is simpler than locking.
   no  ↓

Can I serialize via the database? (UPDATE WHERE, SELECT FOR UPDATE, advisory lock)
   yes → use the DB. No external lock service.
   no  ↓

Is the cost of a brief duplicate run < cost of running a lock service?
   yes → don't lock.
   no  ↓

Use a distributed lock with fencing tokens.

Most "we need a distributed lock" requirements dissolve once you ask the partitioning or idempotency question. Locks are the answer when you genuinely have a single shared mutable resource with non-idempotent operations and no natural partition key.

Lock anti-patterns

No fencing tokens. Silent corruption when a holder pauses. The Kleppmann scenario.
TTL > expected work duration with no recovery on holder crash. Other instances wait forever.
TTL < work duration. Race during work; second holder sees a "free" lock.
Holding a lock across an external API call. Now the lock duration depends on a third party's latency. Lease expiry mid-call is common.
Reentrancy across processes. lock(this) is reentrant; distributed locks usually aren't. Don't assume.
Releasing a lock you no longer own. Without the Lua compare-and-delete, a paused holder's DEL evicts the new owner.
Heartbeat renewal that silently fails. Renewal failure ≠ "we're fine"; it means you may have lost the lock. Either fail the work or check via fencing token.
Using a lock for what should be optimistic concurrency. ETag-based CAS is cheaper and inherently fencing-equivalent on the same resource.

Leader election as a special case

Many "distributed lock" needs are really leader election: "exactly one of N replicas should be the writer at a time." Patterns:

Azure Blob Lease — the single-writer pattern Cosmos Change Feed uses.
etcd Election API — clientv3.Election with explicit Resign/Observe.
ZooKeeper ephemeral sequential nodes — classic leader-election recipe.
Database sp_getapplock / pg_advisory_lock held for the process lifetime.

Leader election + fencing token = a clean leader-only-writes pattern. The leader writes with its token; replicas read freely.

How it works under the hood

Postgres advisory lock

pg_advisory_lock(key):
   1. Hash key to bucket in shared memory.
   2. CAS to set "held" bit; on conflict, wait on PG_LOCK signal.
   3. Held state is in-memory; lost on Postgres crash (which crashes connection).
   4. Released by pg_advisory_unlock OR session end OR transaction end (xact variant).

The lock lives entirely in Postgres process memory — there's no replication. In Postgres HA configs (Patroni, etc.), advisory locks do not fail over. If you need lock survival across primary failover, use a table lock (with row inserts) instead.

Redis SET NX EX flow

Client                       Redis
──────                        ─────
SET lock:k <token> NX EX 30   ──►  if key absent, set value+ttl, reply OK
                              ──►  else reply nil

NX = "only if not exists". EX 30 = TTL in seconds. Both atomic in one round trip. The token is your random value (UUID / GUID); you'll need it on release.

Release uses the Lua script shown above to make GET-and-DEL atomic.

For RedLock across N Redis instances:

Client                  Redis A    Redis B    Redis C    Redis D    Redis E
──────                  ───────    ───────    ───────    ───────    ───────
acquire each in seq    OK         OK         OK         FAIL       OK     → 4/5 = quorum, locked
total time T            ≈ 5×RTT
validity = TTL - T - clockDrift

Kleppmann's critique: validity calculation depends on bounded clock drift and bounded request time. Real systems regularly violate both. Conclusion: RedLock is usable for efficiency-only locks (not safety-critical) — unless you also pair with fencing tokens, in which case the underlying lock service barely matters.

Azure Blob Lease internals

LeaseClient.Acquire(duration):
   POST blob?comp=lease  x-ms-lease-action: acquire, x-ms-lease-duration: 60
        → returns LeaseId GUID

Storage service:
   - Records lease holder + expiry on the blob's metadata in the storage stack.
   - Any write to the blob requires x-ms-lease-id matching.
   - Lease auto-expires; renewal extends.
   - On region failover / service replication: lease state replicates per blob.

The "fencing token equivalent" for Blob Lease is the lease ID itself, but it's a GUID, not monotonic — so multiple expired-then-reacquired leases are not orderable. For strict fencing across leaders, layer your own monotonic token (e.g., increment-on-acquire counter inside another blob).

Fencing token — the resource-side check

Resource state:        max_token_seen[resource_id] = N

Operation arrives:     (resource_id, fencing_token, payload)
  if fencing_token < max_token_seen[resource_id]:
      reject (StaleLock)
  else:
      max_token_seen[resource_id] = fencing_token
      apply(payload)

This must be atomic with the operation — typically a single SQL UPDATE with a WHERE clause:

UPDATE accounts
SET balance = @new, fencing_token = @token
WHERE id = @id AND fencing_token <= @token;
-- if 0 rows affected: stale token, reject

The lock service produces tokens; the resource enforces them; the lock infrastructure no longer needs to be perfectly safe.

Code: correct vs wrong

❌ Wrong: in-process lock for cross-process work

private static readonly object _gate = new();

public void RunNightlyJob()
{
    lock (_gate)             // only locks this instance; other replicas race
    { /* work */ }
}

✅ Correct: Postgres advisory lock via Dapper

public async Task RunNightlyJob()
{
    using var conn = new NpgsqlConnection(_cs);
    await conn.OpenAsync();
    using var tx = await conn.BeginTransactionAsync();

    var got = await conn.ExecuteScalarAsync<bool>(
        "SELECT pg_try_advisory_xact_lock(hashtext('nightly-job'))", transaction: tx);
    if (!got) return;        // someone else has it; skip this run

    await DoWork(conn, tx);
    await tx.CommitAsync();  // lock auto-released
}

❌ Wrong: Redis lock without fencing token

var token = Guid.NewGuid().ToString();
await db.StringSetAsync("lock:k", token, TimeSpan.FromSeconds(30), When.NotExists);

await ProtectedWrite(payload);   // long pause here → lock expires → another holder writes
                                 // we resume and overwrite their work

✅ Correct: Redis lock + fencing token to a CAS resource

// Acquire lock + bump fencing token atomically (Lua)
var script = @"
    if redis.call('SET', KEYS[1], ARGV[1], 'NX', 'EX', ARGV[2]) then
        return redis.call('INCR', KEYS[2])
    else
        return -1
    end";
long token = (long)await db.ScriptEvaluateAsync(script,
    new RedisKey[] { "lock:k", "fence:k" },
    new RedisValue[] { holderId, 30 });
if (token < 0) return;        // didn't acquire

// Resource enforces: only accepts higher tokens
await ProtectedWrite(payload, fencingToken: token);

Resource side:

UPDATE shared_resource
SET value = @v, fencing_token = @token
WHERE id = @id AND fencing_token < @token;

❌ Wrong: Redis SET-then-DEL release (not atomic)

var v = await db.StringGetAsync("lock:k");
if (v == myToken) await db.KeyDeleteAsync("lock:k");
// race window: lock expired between GET and DEL; we just deleted someone else's lock

✅ Correct: Lua compare-and-delete

const string release = @"
    if redis.call('GET', KEYS[1]) == ARGV[1] then
        return redis.call('DEL', KEYS[1])
    else
        return 0
    end";
await db.ScriptEvaluateAsync(release,
    new RedisKey[] { "lock:k" }, new RedisValue[] { myToken });

✅ Correct: Azure Blob Lease for leader election

var blob = _container.GetBlobClient("leader.lock");
await blob.UploadAsync(BinaryData.FromString(""), overwrite: false);  // ensure exists
var lease = blob.GetBlobLeaseClient();

while (!ct.IsCancellationRequested)
{
    try
    {
        var resp = await lease.AcquireAsync(TimeSpan.FromSeconds(60), cancellationToken: ct);
        var leaseId = resp.Value.LeaseId;

        // I'm the leader. Renew every 30s while doing work.
        using var renewer = StartRenewer(lease, ct);
        await DoLeaderWork(leaseId, ct);
    }
    catch (RequestFailedException ex) when (ex.Status == 409) // already leased
    {
        await Task.Delay(TimeSpan.FromSeconds(5), ct);          // back off
    }
}

✅ Correct: DistributedLock (madelson) for convenience

var @lock = new PostgresDistributedLock(new PostgresAdvisoryLockKey("nightly-job"), connStr);
await using var handle = await @lock.TryAcquireAsync(TimeSpan.FromSeconds(30));
if (handle is null) return;       // another instance has it
await DoWork();

Design patterns for this topic

Pattern 1 — "Partition by key, no lock"

Intent: route work so each key has exactly one owner; eliminate the lock entirely.

Pattern 2 — "DB advisory lock"

Intent: simple, reliable, no orphans; for jobs/migrations when DB is already there.

Pattern 3 — "Lease + fencing token"

Intent: every lock acquisition returns a monotonic token; the protected resource validates.

Pattern 4 — "Leader election via Blob Lease"

Intent: exactly-one-writer over time, cloud-native, hands-off operations.

Pattern 5 — "Heartbeat renewal with kill-switch"

Intent: for long-running work; renew lease periodically; if renewal fails, abort work.

Pattern 6 — "ETag-based CAS instead of a lock"

Intent: when contention is on a single resource, optimistic concurrency beats pessimistic locking.

Pros & cons / trade-offs

Mechanism	Pros	Cons
DB row lock	Reliable; no orphans; observable	Slow under contention; ties up DB connection
DB advisory lock	Named; cheap; transaction-scoped	Doesn't survive failover (Postgres)
Redis SET NX EX	Fast; cheap; familiar	No fencing built-in; clock-skew sensitive
RedLock (multi-Redis)	Quorum across instances	Kleppmann's critique applies; complexity
ZooKeeper / etcd	Strong consistency; built-in fencing	Operational cost
Azure Blob Lease	Cloud-native; auto-failover	Lease ID isn't monotonic; renewal required
Cosmos ETag CAS	No lock infrastructure	Single-resource only; reads still cost RU
.NET DistributedLock lib	Unified API	Verify whether your provider returns fencing tokens
Fencing token + any lease	Provably safe	Resource must enforce — extra plumbing

When to use / when to avoid

✅ Use DB advisory locks for migrations, scheduled jobs, and "only-one-of-N" coordination when a DB is already in the architecture.
✅ Use Blob Lease for leader election on Azure (Change Feed Processor, EventProcessorClient pattern).
✅ Use ETag/CAS for single-resource contention — cheaper and naturally fencing-equivalent.
✅ Use fencing tokens with every lock when correctness matters.
❌ Avoid a distributed lock when you can partition by key.
❌ Avoid a distributed lock when idempotency + dedup solves the same problem (cross-link: see Idempotency Keys — Deep).
❌ Avoid RedLock for safety-critical work without fencing tokens.
❌ Avoid holding any lock across an external API call.
❌ Avoid trusting that your library returns fencing tokens — verify.

Interview Q&A

Q1. Why is RedLock controversial? Martin Kleppmann showed RedLock's safety claims rely on synchronized clocks and bounded process pauses. Real systems violate both. RedLock is fine for efficiency-only ("don't run twice if avoidable") but not for safety-critical correctness — unless you pair it with fencing tokens enforced at the resource.

Q2. What's a fencing token and why does it matter? A monotonically increasing number returned with every lock acquisition. Clients pass it on every protected operation; the resource records max_token_seen and rejects lower ones. Survives a paused-then-resumed lock holder and turns "trust the lock service" into "trust a single number".

Q3. When does a Postgres advisory lock beat a Redis lock? When a Postgres connection is already in your code path. Advisory locks are session/transaction-scoped, so a crashed holder loses the lock automatically when the connection drops — no orphan locks. Redis needs a TTL and explicit release.

Q4. How do you renew a lease for long-running work? Background timer that periodically extends the lease (e.g., every TTL/3 seconds). If renewal fails, you must assume the lock is lost — abort the work or rely on a fencing token to keep the resource safe.

Q5. What's the leader-election pattern? Exactly one of N replicas owns a lease at a time. Common implementations: Azure Blob Lease (Cosmos Change Feed uses this), ZooKeeper ephemeral nodes, etcd Election API. Combined with a fencing token, the leader's writes are unambiguous.

Q6. When should you skip distributed locks entirely? When you can partition work by key (each node owns its keys), when operations are idempotent (at-least-once + dedup is simpler), when the DB can serialize via UPDATE with WHERE, or when a brief duplicate run is cheaper than the lock infrastructure.

Q7. What does a Blob Lease give you that a Redis SET doesn't? Lease state managed by Azure Storage with cross-zone replication; explicit fail-fast on lost lease (writes return 412 if the lease ID doesn't match); built-in for leader-election scenarios; no network of Redis nodes to operate. Redis is faster and cheaper for ephemeral coordination.

Q8. Why must Redis lock release use Lua? The naive sequence GET → if-mine → DEL has a race window: between GET and DEL, the lock can expire and be reacquired by another holder. Lua makes the compare-and-delete atomic on the Redis side.

Q9. What's the difference between pg_advisory_lock and a SELECT FOR UPDATE? pg_advisory_lock is a named lock not tied to any row — useful for "only one runner of this job". SELECT FOR UPDATE locks an actual row in a transaction; meant for serializing access to data.

Q10. Reentrancy across processes — what's the gotcha? lock and Mutex in .NET are reentrant — the same thread can re-acquire. Distributed locks are typically not reentrant; the second Acquire from the same process either blocks or sees the lock as held. Don't assume.

Q11. How does sp_getapplock work in SQL Server? Named application lock with optional transaction or session scope, configurable lock mode (Exclusive, Shared, Update, IntentExclusive, IntentShared). Released on transaction commit/rollback or session end. Excellent for coordination tasks where the DB is already in the picture.

Q12. What's the safest fencing-token pattern in EF Core / Cosmos? Use the storage's native version primitive: _etag in Cosmos with IfMatchEtag, xmin or RowVersion in EF Core with [ConcurrencyCheck]. The store enforces ordering for free, and you don't need an external fencing-token field.

Gotchas / common mistakes

⚠️ Assuming RedLock is correct without fencing tokens — silent corruption on pause.
⚠️ Clock-skew assumptions in TTL math.
⚠️ Lease TTL longer than expected work but no recovery on crash → stalled jobs.
⚠️ Lease TTL shorter than expected work → second holder runs concurrently.
⚠️ Holding a lock across an external API → unbounded duration, lease expiry mid-call.
⚠️ Reentrancy expectations — distributed locks usually aren't.
⚠️ Non-atomic release in Redis without Lua — deletes someone else's lock.
⚠️ Heartbeat renewal failure ignored — you've lost the lock and don't know it.
⚠️ Using a "lock library" without checking for fencing-token support — most don't include it.
⚠️ Locking when you should partition — adds complexity for no gain.
⚠️ Locking when the DB already serializes — duplicate effort, more failure modes.
⚠️ Lock starvation — long fair-queue waits; consider time-bounded TryAcquire.
⚠️ Postgres advisory locks across HA failover — they don't survive primary fail.