Health Checks

Key Points

Two probe types (Kubernetes terminology, broadly applicable):
Liveness — "is the process alive?" Restart on failure.
Readiness — "ready to serve traffic?" Remove from load balancer on failure; don't restart.
ASP.NET Core's Microsoft.Extensions.Diagnostics.HealthChecks ships built-in. Register checks → MapHealthChecks endpoints.
Use Predicate to filter checks per endpoint (/health/live shows only liveness; /health/ready shows readiness).
AspNetCore.HealthChecks.* community packages cover SQL Server, Redis, RabbitMQ, etc. with prebuilt checks.
Don't expose internal details in public health endpoints; use a separate auth-protected detailed endpoint if needed.

Concepts (deep dive)

Liveness vs readiness

Container starts ──► startupProbe (k8s)
                       │
                       ▼
                    livenessProbe (k8s)
                       │ (fail → restart pod)
                       ▼
                    readinessProbe
                       │ (fail → remove from service)
                       ▼
                    serving traffic

Liveness fails: orchestrator restarts the pod. Reserve for "the process is broken and a restart will fix it" (deadlocked, OOMing, app heap corrupt).
Readiness fails: orchestrator removes the pod from the service load balancer until it passes again. Use for "downstream dependency unhealthy" — the app itself is fine, just can't serve right now.

Setup

builder.Services.AddHealthChecks()
    .AddCheck("self", () => HealthCheckResult.Healthy(), tags: new[] { "live" })
    .AddSqlServer(connStr, name: "db", tags: new[] { "ready" })
    .AddRedis(connStr, name: "cache", tags: new[] { "ready" })
    .AddCheck<CustomCheck>("custom", tags: new[] { "ready" });

var app = builder.Build();

app.MapHealthChecks("/health/live", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("live")
});

app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("ready"),
    ResponseWriter = HealthCheckResponseWriter.WriteJson   // optional: detailed JSON
});

Custom check

public class DiskSpaceHealthCheck(IClock clock) : IHealthCheck
{
    public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext ctx, CancellationToken ct)
    {
        var disk = new DriveInfo("/");
        var freePercent = (double)disk.AvailableFreeSpace / disk.TotalSize;

        return Task.FromResult(freePercent switch
        {
            < 0.05 => HealthCheckResult.Unhealthy($"Disk space below 5%: {freePercent:P}"),
            < 0.20 => HealthCheckResult.Degraded($"Disk space below 20%: {freePercent:P}"),
            _ => HealthCheckResult.Healthy($"Disk space {freePercent:P}")
        });
    }
}

HealthCheckResult: Healthy, Degraded (yellow; still serve), Unhealthy (red).

Response writer (detailed JSON)

app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
    Predicate = c => c.Tags.Contains("ready"),
    ResponseWriter = async (ctx, report) =>
    {
        ctx.Response.ContentType = "application/json";
        var result = new
        {
            status = report.Status.ToString(),
            duration = report.TotalDuration.TotalMilliseconds,
            checks = report.Entries.Select(e => new
            {
                name = e.Key,
                status = e.Value.Status.ToString(),
                duration = e.Value.Duration.TotalMilliseconds,
                description = e.Value.Description
            })
        };
        await JsonSerializer.SerializeAsync(ctx.Response.Body, result);
    }
});

The AspNetCore.HealthChecks.UI.Client NuGet package provides a built-in JSON writer plus a UI dashboard.

Kubernetes wiring

livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  failureThreshold: 3

Pod startup:

startupProbe:                    # gives slow-starting apps grace
  httpGet: { path: /health/live, port: 8080 }
  failureThreshold: 30           # 30 attempts × 10s = 5 min before kill
  periodSeconds: 10

Cache the readiness result

Don't ping every dependency on every probe — they fire every few seconds:

builder.Services.AddHealthChecks()
    .AddCheck<DbCheck>("db", tags: new[] { "ready" })
    .AddCheck("cache", () => HealthCheckResult.Healthy(), tags: new[] { "ready" });

// Apply caching at the response layer:
app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
    Predicate = c => c.Tags.Contains("ready")
}).CacheOutput(b => b.Expire(TimeSpan.FromSeconds(5)));

Or implement caching inside each check (MemoryCache.GetOrCreateAsync).

Avoid leaking implementation details

Production health endpoints should return:

{ "status": "Healthy" }

Detailed info (which check failed, error messages) is useful in dev/staging but a security risk in prod (info disclosure). Either gate detailed endpoint behind auth, or use the simple WriteMinimalPlaintext writer.

app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
    Predicate = c => c.Tags.Contains("ready"),
    ResponseWriter = HealthCheckResponseWriter.WriteMinimalPlaintext
});

Code: correct vs wrong

❌ Wrong: liveness depends on database

.AddSqlServer(connStr, tags: new[] { "live" });
// DB temporarily down → pod restarted → eventually all replicas restart → service down

✅ Correct: liveness checks the process; readiness checks dependencies

.AddCheck("self", () => HealthCheckResult.Healthy(), tags: new[] { "live" })
.AddSqlServer(connStr, tags: new[] { "ready" });

❌ Wrong: probing every check on every probe

.AddSqlServer(connStr)   // hit DB every 5 seconds

✅ Correct: cache

app.MapHealthChecks("/health/ready", new() { Predicate = ... })
   .CacheOutput(b => b.Expire(TimeSpan.FromSeconds(5)));

Design patterns for this topic

Pattern 1 — "Two endpoints: live + ready"

Intent: correct K8s probe semantics.

Pattern 2 — "Tag-based filtering"

Intent: one set of checks, multiple filtered endpoints.

Pattern 3 — "Cache results to bound dependency probing rate"

Intent: avoid hammering DB / Redis on every probe.

Pattern 4 — "Minimal response in prod; detailed in non-prod"

Intent: info-disclosure defense.

Pattern 5 — "Custom check for app-specific concerns"

Intent: disk space, queue depth, cache hit rate.

Pros & cons / trade-offs

Aspect	Pros	Cons
Built-in checks	Simple	One per dependency
Tag filtering	Multiple endpoints, one set	Tag mistakes leak checks
Caching probes	Bounded probe load	Slightly stale view
Detailed JSON	Diagnostic	Info disclosure

When to use / when to avoid

Always expose liveness + readiness as separate endpoints.
Always cache readiness probes to bound dependency load.
Avoid liveness depending on external dependencies.
Avoid detailed responses in public-internet endpoints.

Interview Q&A

Q1. Difference between liveness and readiness probes? Liveness: "process alive?" Failure → restart. Readiness: "ready to serve?" Failure → remove from load balancer; don't restart.

Q2. Why shouldn't liveness depend on the database? DB outage → all pods fail liveness → all restart → cascade. Liveness should be process-only; DB belongs in readiness.

Q3. What are the three result statuses? Healthy, Degraded, Unhealthy.

Q4. How do you filter checks per endpoint? Tag checks; filter via Predicate = c => c.Tags.Contains("ready").

Q5. What does AspNetCore.HealthChecks.UI.Client provide? A JSON response writer for detailed health info, plus a UI dashboard option.

Q6. How do you avoid hammering dependencies on each probe? Cache the response (output cache) or cache inside the check itself.

Q7. What's a startupProbe? Kubernetes probe for slow-starting apps. Gives grace before liveness/readiness probes start.

Q8. Should health endpoints require auth? Liveness/readiness from K8s usually no (within-cluster). Detailed/internal — yes, gate behind auth.

Q9. How do you write a custom check? Implement IHealthCheck; register with AddCheck<T>("name", tags: ...).

Q10. When does Degraded fail readiness? Depends on ResultStatusCodes config. Default: Degraded → 200 OK. K8s-friendly: Healthy + Degraded → 200; Unhealthy → 503.

Gotchas / common mistakes

⚠️ Liveness includes DB — cascade restarts.
⚠️ No probe caching — overload DB on probe rate.
⚠️ Detailed JSON to public — info leak.
⚠️ Forgetting startup probe for slow-starting apps — premature liveness fail.
⚠️ Treating Degraded as healthy without thought — silent dependency issues.