Load Balancing Strategies

Key Points

• Algorithms: round-robin (RR), weighted RR, least-connections, least-response-time, IP-hash / consistent-hashing, power-of-two-choices (P2C). Pick based on backend homogeneity and call shape.
• Layer 4 vs Layer 7: L4 routes on TCP/UDP (IP + port) — fast, opaque. L7 routes on HTTP (path, header, host, method) — required for canary, A/B, header-based routing.
• Health checks make or break the LB: active probes detect dead pods quickly; passive (outlier ejection) catches partial failures. Slow-start ramps new instances.
• Sticky sessions are an anti-pattern in stateless services — they pin load to one instance and break elastic scaling. Use a shared session store instead.
• Connection draining during shutdown is mandatory: stop accepting new requests, finish in-flight ones, then exit. Without it, every deploy spikes 5xx.
• In .NET: Kestrel sits behind nginx / Application Gateway / Front Door / YARP / Kubernetes Service. Each layer can do its own balancing — know which one is authoritative.

Concepts (deep dive)

Why load balancing exists

A single instance is a single point of failure and a single throughput cap. Once you scale horizontally, something has to decide which instance handles each request. That something is the load balancer (LB). It also acts as a stable address for clients (DNS → LB VIP → many backends).

   client ──► [   LB   ] ──┬──► instance A
                           ├──► instance B
                           └──► instance C

Algorithms

Round-robin (RR)

Cycle through backends in order. Simple. Fair when backends are identical and requests cost the same.

req1 → A    req2 → B    req3 → C    req4 → A ...

Fails when backend latency varies — slow instance still gets 1/N traffic.

Weighted round-robin

Each backend has a weight; bigger machine gets more requests.

A weight=3, B weight=1  →  A A A B A A A B ...

Useful for heterogeneous fleets (mix of Standard_D2 and Standard_D8 nodes) or canary (90/10 splits).

Least-connections

Pick the backend with the fewest active connections. Self-balancing under variable workloads — slow instance accumulates connections, gets fewer new ones.

A: 12 conns    B: 3 conns    C: 7 conns   →   next req → B

Best default for general HTTP services where requests have variable cost.

Least-response-time

Combines connection count with EWMA of response time. More accurate than least-connections but requires the LB to track latency. Envoy and HAProxy support it.

IP-hash / consistent-hashing

Hash the source IP (or a routing key) → pick a backend deterministically. Same key always lands on same backend.

hash(client_ip) % N → backend index

Used for cache locality (same key → same node → warm cache) and as a workaround for in-process state. Consistent hashing minimizes reshuffling when N changes (only ~1/N keys move, not all of them).

Power-of-two-choices (P2C)

Pick two random backends, route to the one with fewer active connections. Empirically near-optimal with O(1) cost. Used by Linkerd, Twitter Finagle, Envoy.

candidates = sample(backends, 2)
choose argmin(active_connections)

Avoids the herd-on-best-backend problem of strict least-connections (where stale telemetry sends every concurrent decider to the same "best" node).

Layer 4 vs Layer 7

L7 sees the HTTP request and can do smart things (rewrite path, add headers, route by JWT claim). It costs more CPU and terminates TLS. L4 is faster and protocol-agnostic but can only see the 5-tuple.

A common pattern: L4 in front for raw distribution, L7 inside the cluster for application routing.

Health checks

                   ┌───────────────┐
   active probe ──►│  backend pod  │── HTTP 200 ──► healthy
                   └───────────────┘

Active health checks

LB calls /health (or /healthz) on each backend every N seconds. Configurable thresholds: e.g., 2 consecutive failures → mark unhealthy; 3 successes → mark healthy.

ASP.NET Core:

builder.Services.AddHealthChecks()
    .AddDbContextCheck<AppDb>()
    .AddRedis(redisCs);

app.MapHealthChecks("/health/live",  new() { Predicate = _ => false });   // liveness
app.MapHealthChecks("/health/ready", new() { Predicate = c => c.Tags.Contains("ready") });

Liveness vs readiness: - Liveness: am I alive? (failing → restart pod) - Readiness: should I receive traffic? (failing → remove from pool, don't restart)

Passive health checks (outlier ejection)

LB watches real traffic. If a backend's error rate / latency exceeds threshold over a window, eject it temporarily. Detects partial failures the active probe misses (e.g., DB pool exhausted but /health still 200s).

Envoy outlier_detection:
  consecutive_5xx: 5
  interval: 10s
  base_ejection_time: 30s
  max_ejection_percent: 50

Slow-start

Newly-healthy backend gets ramped traffic over N seconds, not full share immediately. Lets it warm up JIT, fill caches, prime connection pools. Without it, a new pod gets hit by 1/N of full prod load and falls over.

Half-open

After ejection, send a single probe request before fully restoring. Same idea as a circuit breaker half-open state.

Sticky sessions

Pin a client to one backend, usually via cookie (AWSALB, ARRAffinity) or source IP.

When you actually need it: - In-process WebSocket / SignalR connection state without a backplane. - Legacy app with in-memory session and no time to refactor.

Why mostly avoid: - Hot pods get hotter (sticky users keep coming back to the same instance). - Deploys / autoscale events break sticky clients. - Defeats the point of stateless horizontal scale.

Better: shared session store (Redis, Cosmos), or SignalR Redis backplane / Azure SignalR Service.

Connection draining (graceful shutdown)

1. Pod receives SIGTERM
2. Health endpoint flips to "not ready"
3. LB stops sending new requests (after one health-check interval)
4. In-flight requests complete (drain timeout, e.g., 30s)
5. Process exits

ASP.NET Core handles this if you respect IHostApplicationLifetime:

builder.Host.ConfigureHostOptions(o => o.ShutdownTimeout = TimeSpan.FromSeconds(30));

In Kubernetes:

terminationGracePeriodSeconds: 60
lifecycle:
  preStop:
    exec:
      command: ["sh","-c","sleep 10"]   # let Service endpoints update

Hosting context for .NET

Kestrel behind nginx

nginx terminates TLS, does L7 routing, forwards to Kestrel via HTTP. Common for self-hosted / on-prem.

upstream app { least_conn; server app1:5000; server app2:5000; }
server { listen 443 ssl; location / { proxy_pass http://app; } }

Behind Azure Application Gateway

Regional L7 LB with WAF. Path/host routing, end-to-end TLS, cookie affinity if needed.

Behind Azure Front Door

Global L7 LB at the Microsoft edge. Anycast routing, geo-balancing, caching, WAF. Use for multi-region failover and edge presence.

client → Front Door (edge POP) → App Gateway (region) → AKS / App Service

Behind YARP

YARP runs in-process or as a sidecar / gateway. Code-configurable routes; great for BFF and platform proxies. See YARP Reverse Proxy.

Kubernetes (Service + kube-proxy)

Service is a stable virtual IP. Behind it: Endpoints / EndpointSlices listing healthy pods.

• iptables mode (default): random pick per connection. Roughly RR over time, no awareness of load.
• IPVS mode: kernel-level LB; supports RR, least-connection, hash. Faster at scale.
• eBPF / Cilium: programmable dataplane; replaces kube-proxy with eBPF maps. Lower latency, better observability.

For L7 inside the cluster: an Ingress controller (nginx-ingress, Contour, Traefik) or a service mesh.

Service mesh (Istio / Linkerd)

Sidecar proxies (Envoy / linkerd2-proxy) handle L7 LB transparently. Per-service policies — RR / least-request / P2C / consistent hash, retries, circuit breaking, mTLS — without app code changes.

Use when: many services, want uniform policy, need mTLS / fine-grained traffic shifting. Skip when: 3 services and a tight ops budget — the mesh is non-trivial.

Algorithm selection guide

How it works under the hood

Connection lifecycle through an LB

client ──TCP SYN──► LB:443 ──TCP SYN──► backend:5000
       ◄─SYN/ACK─       ◄─SYN/ACK─
       ──ACK──►          ──ACK──►
       ──TLS─►  (terminate at LB or pass through)
       ──HTTP─► [LB picks backend per connection or per request]

• Per-connection (L4): client TCP connection pinned to one backend for its lifetime. HTTP/1.1 keep-alive → multiple requests on same backend.
• Per-request (L7): each HTTP request can go to a different backend. Better balance for long-lived connections (HTTP/2, gRPC) — otherwise one client's H2 connection multiplexes everything to a single pod.

HTTP/2 and gRPC pitfall

HTTP/2 multiplexes many requests over one TCP connection. With L4 LB, all those requests stick to one backend. Result: severe imbalance.

Fix: terminate HTTP/2 at an L7 LB so it can balance per-request. (Application Gateway, Envoy, YARP, ALB all do this.)

How a Kubernetes Service routes

Service (ClusterIP 10.96.1.5)
   │
   └─► EndpointSlice [10.0.1.10, 10.0.1.11, 10.0.1.12]  (healthy pods)
          │
          └─► kube-proxy installs iptables / IPVS rules on every node
                 │
                 └─► DNAT to a random endpoint

Pod startup → readiness probe passes → kubelet → API server → EndpointSlice updated → kube-proxy reconfigures iptables on every node (eventually). The "eventually" is why preStop sleep matters.

Consistent hashing internals

ring = [hash(node_A), hash(node_B), hash(node_C)]    sorted around 2^32 ring
key  = hash(request_key)
node = first ring position >= key (wrap around)

Adding node D: only the ~¼ of keys whose nearest position is now D get reassigned. RR or modulo hashing would reshuffle nearly everything.

Code: correct vs wrong

❌ Wrong: gRPC behind L4 LB without H2 termination

# Service is plain ClusterIP / NLB; gRPC client opens one H2 connection
# all RPC calls land on a single pod

✅ Correct: terminate H2 at L7

# Use an Ingress / mesh that speaks HTTP/2 and balances per-request,
# or client-side LB (gRPC's round_robin policy) with headless Service
service:
  clusterIP: None     # headless → DNS returns all pod IPs

var channel = GrpcChannel.ForAddress("dns:///orders", new()
{
    ServiceConfig = new() { LoadBalancingConfigs = { new RoundRobinConfig() } }
});

❌ Wrong: no draining on shutdown

// SIGTERM → process exits immediately → in-flight requests get RST → 5xx

✅ Correct: respect lifetime

builder.Host.ConfigureHostOptions(o => o.ShutdownTimeout = TimeSpan.FromSeconds(30));

app.MapHealthChecks("/health/ready", new()
{
    Predicate = c => c.Tags.Contains("ready")
});
// readiness flips false on SIGTERM; LB drains; then exit

❌ Wrong: sticky sessions for stateless API

upstream app { ip_hash; server a; server b; server c; }   # pins everyone

✅ Correct: stateless + shared session

builder.Services.AddStackExchangeRedisCache(o => o.Configuration = redisCs);
builder.Services.AddSession();

❌ Wrong: only liveness probe

livenessProbe: { httpGet: { path: /health } }
# DB outage → /health 500 → kubelet restarts pod → restart storm

✅ Correct: liveness + readiness

livenessProbe:  { httpGet: { path: /health/live  } }   # process alive?
readinessProbe: { httpGet: { path: /health/ready } }   # deps healthy?

Design patterns for this topic

Pattern 1 — "RR for uniform pods, P2C otherwise"

• Intent: simple default, robust under variable load.

Pattern 2 — "Consistent hashing on tenant ID"

• Intent: cache locality without sticky sessions.

Pattern 3 — "Liveness + readiness split"

• Intent: restart only on process failure; drain on dependency failure.

Pattern 4 — "Slow-start + outlier ejection"

• Intent: protect cold pods, eject silent failers.

Pattern 5 — "Front Door (global) → App Gateway (regional) → mesh (cluster)"

• Intent: layered LB; each tier picks its own algorithm for its scope.

Pros & cons / trade-offs

When to use / when to avoid

• Use L7 LB whenever you have HTTP/2 or gRPC.
• Use P2C or least-connections when request cost varies.
• Use consistent hashing for cache-locality scenarios (per-tenant warmth).
• Avoid sticky sessions for stateless APIs — fix the statelessness instead.
• Avoid liveness-only probes — split into liveness and readiness.
• Avoid rolling without connection draining.

Interview Q&A

Q1. Round-robin vs least-connections? RR cycles backends; least-connections picks the one with fewest active conns. Least-conns adapts to variable request cost; RR doesn't.

Q2. Why is gRPC traffic often imbalanced? HTTP/2 multiplexes on one TCP conn; an L4 LB pins that conn to one backend, so all RPCs go there. Fix: L7 LB or client-side LB with headless DNS.

Q3. L4 vs L7? L4: TCP/UDP, fast, opaque. L7: HTTP-aware, can route on path/header/host, do canary, terminate TLS. L7 costs more CPU.

Q4. Liveness vs readiness probes? Liveness: alive? (fail → restart). Readiness: ready for traffic? (fail → remove from pool, don't restart). Always split them.

Q5. What is power-of-two-choices? Sample 2 random backends, pick the one with fewer active connections. Near-optimal balance without strict-least-conn herding.

Q6. Why slow-start? New pods need to warm up (JIT, caches, pools). Hitting them with full share at t=0 causes brownout. Slow-start ramps over N seconds.

Q7. When are sticky sessions OK? SignalR / WebSocket without a backplane, or legacy in-memory session. Otherwise use shared session store and stay stateless.

Q8. Consistent hashing — what problem does it solve? Adding/removing a node in modulo hashing reshuffles nearly all keys. Consistent hashing reshuffles ~1/N. Critical for distributed caches.

Q9. Connection draining flow? SIGTERM → readiness flips false → LB stops new traffic after probe interval → in-flight finishes within drain timeout → process exits.

Q10. Outlier ejection? Passive health: watch error rate / latency on real traffic; eject backends that exceed threshold. Catches partial failures active probes miss.

Q11. Service mesh role in LB? Sidecars (Envoy / linkerd2-proxy) do L7 LB per service: RR / least-request / P2C, retries, circuit breaking. Uniform policy via control plane.

Q12. Front Door vs Application Gateway? Front Door: global, edge, anycast, multi-region. App Gateway: regional, WAF, intra-region routing. Often layered together.

Gotchas / common mistakes

• ⚠️ HTTP/2 / gRPC behind L4 LB — all calls hit one pod.
• ⚠️ Liveness-only probes — restart storms on dependency outages.
• ⚠️ No terminationGracePeriodSeconds — every deploy = 5xx burst.
• ⚠️ Sticky sessions on stateless API — hot pods, broken autoscale.
• ⚠️ No outlier ejection — partial failures (DB pool exhausted, GC pause) silently drag the cluster.
• ⚠️ Modulo hashing on a cache cluster — adding a node reshuffles everything.
• ⚠️ Health check that doesn't check dependencies for readiness — pod marked ready while DB is down.

Further reading

• MS Learn: Application Gateway overview
• MS Learn: Azure Front Door
• MS Learn: Health checks in ASP.NET Core
• Envoy: load balancers
• Linkerd: Why P2C with EWMA
• Kubernetes: Service & EndpointSlices