Resilience: Polly v8

Key Points

• Polly v8 introduced ResiliencePipeline — replaces older Policy<T> API. More performant, composable, OpenTelemetry-friendly.
• Microsoft.Extensions.Resilience is the official wrapper — first-class with IHttpClientFactory, DI, IOptions, telemetry.
• Strategies: retry, circuit breaker, timeout, rate limiter, fallback, hedging, chaos.
• Compose with care: order matters. Typical: outer (timeout for whole op) → retry → circuit breaker → inner (per-attempt timeout).
• Don't retry everything: 4xx (mostly), validation errors, auth failures don't recover with retry. 5xx, transient network errors, 408/429 do.

Concepts (deep dive)

Setup

<PackageReference Include="Microsoft.Extensions.Http.Resilience" Version="9.*" />

builder.Services.AddHttpClient("api", c => c.BaseAddress = new Uri("https://api/"))
    .AddStandardResilienceHandler();   // default policies for HTTP

AddStandardResilienceHandler adds: total timeout (30s), retry (3 with backoff), circuit breaker, attempt timeout (10s).

Custom pipeline

var pipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
    .AddTimeout(TimeSpan.FromSeconds(30))            // overall timeout
    .AddRetry(new RetryStrategyOptions<HttpResponseMessage>
    {
        MaxRetryAttempts = 3,
        BackoffType = DelayBackoffType.Exponential,
        UseJitter = true,
        Delay = TimeSpan.FromSeconds(1),
        ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
            .Handle<HttpRequestException>()
            .HandleResult(r => r.StatusCode == HttpStatusCode.ServiceUnavailable)
    })
    .AddCircuitBreaker(new CircuitBreakerStrategyOptions<HttpResponseMessage>
    {
        FailureRatio = 0.5,
        MinimumThroughput = 10,
        SamplingDuration = TimeSpan.FromSeconds(30),
        BreakDuration = TimeSpan.FromSeconds(15)
    })
    .AddTimeout(TimeSpan.FromSeconds(10))            // per-attempt timeout
    .Build();

var response = await pipeline.ExecuteAsync(async ct => await _http.GetAsync(url, ct));

Strategies

Retry

.AddRetry(new()
{
    MaxRetryAttempts = 3,
    Delay = TimeSpan.FromSeconds(1),
    BackoffType = DelayBackoffType.Exponential,   // 1s, 2s, 4s
    UseJitter = true                                // randomized
})

Always use jitter — prevents thundering herd when many clients retry simultaneously.

Don't retry: - 4xx (except 408 timeout, 429 rate-limit, sometimes 401 after refresh). - Non-idempotent operations on uncertain failure (POST without idempotency key). - Validation errors.

Circuit Breaker

.AddCircuitBreaker(new()
{
    FailureRatio = 0.5,           // 50% of calls failed
    MinimumThroughput = 20,        // ...within 20 calls
    SamplingDuration = 30s,        // ...in last 30s
    BreakDuration = 15s            // open circuit for 15s
})

States:

Closed → (failures) → Open → (after BreakDuration) → Half-Open → (success) → Closed
                                                             → (fail) → Open

Open circuit fails fast — saves load on failing dependency.

Timeout

.AddTimeout(TimeSpan.FromSeconds(10))

Pessimistic timeout cancels the operation. Pair with cooperative cancellation in your code.

Rate Limiter

.AddRateLimiter(new SlidingWindowRateLimiter(new() { PermitLimit = 100, Window = TimeSpan.FromSeconds(1) }))

Limits operations per time window — protects you (the client) from overwhelming downstream.

Fallback

.AddFallback(new FallbackStrategyOptions<HttpResponseMessage>
{
    FallbackAction = args => Outcome.FromResultAsValueTask(
        new HttpResponseMessage(HttpStatusCode.OK) { Content = new StringContent("[]") }),
    ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
        .Handle<HttpRequestException>()
})

Returns a default value when the operation fails after all retries.

Hedging

.AddHedging(new HedgingStrategyOptions<HttpResponseMessage>
{
    MaxHedgedAttempts = 2,
    Delay = TimeSpan.FromMilliseconds(100)
})

Issues parallel requests after a delay. First successful response wins. Useful for tail-latency reduction (some calls ~100ms, occasional ~5s outliers).

Chaos (testing)

.AddChaosLatency(0.1, TimeSpan.FromSeconds(5))   // 10% chance, +5s latency
.AddChaosFault(0.05, () => new HttpRequestException())

Inject controlled failures. Run in non-prod. Helps test resilience.

Composition order

[outer] Timeout (30s total) →
[mid]   Retry (3x) →
[inner] Circuit Breaker →
[innermost] Per-attempt Timeout (10s)

• Outermost timeout caps the whole operation.
• Retry sits inside total timeout.
• Circuit breaker inside retry — opens on aggregate failures.
• Inner timeout per attempt.

Order matters: putting circuit breaker outside retry means each retry doesn't see breaker state.

Standard resilience handler vs custom

.AddStandardResilienceHandler()                   // sensible defaults
.AddStandardResilienceHandler(o => { o.Retry.MaxRetryAttempts = 5; })   // tweak

For 95% of HTTP cases, defaults are great. Custom for special needs.

Hedging strategy

.AddStandardHedgingHandler()

Pre-built hedging for HTTP. Reduces tail latency at cost of doubled traffic (worst case).

OTel integration

Polly v8 emits Microsoft.Extensions.Resilience metrics: - resilience.execution.duration - resilience.execution.attempts - resilience.circuit_breaker.state

Auto-instrumented by OpenTelemetry.Instrumentation.Http.Resilience.

Per-context configuration

var ctx = ResilienceContextPool.Shared.Get();
ctx.Properties.Set(new ResiliencePropertyKey<string>("operationName"), "GetUser");

await pipeline.ExecuteAsync(async ctx => await Op(ctx), ctx);

ResilienceContextPool.Shared.Return(ctx);

Lets strategies make context-aware decisions.

Idempotency caveat

Retrying non-idempotent operations is dangerous. POST without idempotency key + retry on uncertain failure = duplicate side effect.

Always: - Add idempotency keys to non-idempotent operations. - Or use sagas/outbox.

Polly v7 vs v8

• v7: Policy.Handle<>().RetryAsync(). Reflection-heavy; slower.
• v8: ResiliencePipelineBuilder. Source-gen options; faster; better OTel.

Migrate v7 → v8. Both can coexist during migration.

Don't reinvent

Polly is mature. Don't write your own retry loop with for + await Task.Delay. Polly handles cancellation, telemetry, jitter, edge cases.

Code: correct vs wrong

❌ Wrong: retry without jitter

.AddRetry(new() { Delay = TimeSpan.FromSeconds(1), BackoffType = DelayBackoffType.Exponential })

Thundering herd risk.

✅ Correct: jitter on

.AddRetry(new() { Delay = ..., BackoffType = ..., UseJitter = true })

❌ Wrong: retry on everything

.AddRetry(new() { MaxRetryAttempts = 3 })   // retries 4xx errors that won't recover

✅ Correct: ShouldHandle

.AddRetry(new() { ShouldHandle = new PredicateBuilder().Handle<HttpRequestException>().HandleResult(r => (int)r.StatusCode >= 500) })

❌ Wrong: retry on non-idempotent POST

.AddRetry(new()).Execute(() => _http.PostAsync(...));   // duplicate orders!

✅ Correct: idempotency key

req.Headers.Add("Idempotency-Key", Guid.NewGuid().ToString());

Design patterns for this topic

Pattern 1 — "Standard handler for HTTP"

• Intent: sensible defaults; minimal config.

Pattern 2 — "Outer-to-inner: timeout → retry → CB → attempt timeout"

• Intent: correct order.

Pattern 3 — "Jitter on every retry"

• Intent: avoid thundering herd.

Pattern 4 — "Hedging for tail latency"

• Intent: reduce p99 at cost of doubled traffic.

Pattern 5 — "Idempotency key for non-idempotent retries"

• Intent: safe retries.

Pros & cons / trade-offs

When to use / when to avoid

• Always retry + circuit breaker + timeout for outbound HTTP.
• Use hedging for tail-latency-critical reads.
• Avoid retry on non-idempotent without idempotency key.
• Avoid retry on 4xx (with exceptions).

Interview Q&A

Q1. Polly v8 vs v7? v8: ResiliencePipelineBuilder; faster; OTel-native; preferred. v7: Policy classes; reflection-heavy.

Q2. Why jitter? Without it, all clients retry simultaneously after a backoff — thundering herd. Jitter randomizes delays.

Q3. Composition order? Outer: total timeout. Mid: retry. Inner: circuit breaker. Innermost: per-attempt timeout.

Q4. When NOT retry? 4xx (most), validation errors, auth failures, non-idempotent without key.

Q5. Circuit breaker states? Closed → Open → Half-Open → Closed (or back to Open).

Q6. Hedging — how works? Issue parallel requests after delay. First success wins. Reduces tail latency.

Q7. Standard resilience handler? Pre-built HTTP pipeline with sensible retry/CB/timeout. Minimal config.

Q8. Why per-attempt + total timeout? Per-attempt: don't waste time on hung call. Total: cap whole op including retries.

Q9. Idempotency for retried POSTs? Idempotency-Key header. Server dedups by key.

Q10. Polly OTel integration? Built-in metrics; auto-traced when OTel registered.

Q11. Chaos testing in Polly? AddChaosLatency, AddChaosFault — inject controlled failures. Non-prod only.

Q12. Rate limiter use? Self-throttle: don't overwhelm downstream. Different from server-side rate limiting.

Gotchas / common mistakes

• ⚠️ Retry without ShouldHandle — retrying unrecoverable errors.
• ⚠️ No jitter — thundering herd.
• ⚠️ Wrong composition order — circuit breaker outside retry useless.
• ⚠️ Retry non-idempotent POST — duplicates.
• ⚠️ Per-attempt timeout > total timeout — total wins; retry pointless.

Further reading

• Polly docs
• MS Learn: Microsoft.Extensions.Resilience
• MS Learn: Standard resilience handler
• Stephen Cleary: Async retries