Service Discovery

Key Points

Static config breaks under autoscaling. Pods get dynamic IPs and lifetimes; clients need a way to find healthy instances at call time.
Two patterns: client-side discovery (client queries registry, picks an instance, calls direct) vs server-side (LB / gateway queries registry, client calls LB).
Registries: DNS (cheap, TTL pitfalls), Consul, etcd, ZooKeeper (legacy), Eureka (JVM-leaning), Kubernetes Services + EndpointSlices (built-in).
Microsoft.Extensions.ServiceDiscovery (Aspire) abstracts schemes (https+http://orders) over pluggable resolvers — config, DNS SRV, Kubernetes — and integrates cleanly with HttpClientFactory.
Stale registries are the silent killer. TTLs, gossip delays, and missing health propagation mean the registry can list dead instances or miss live ones.
Service mesh sidesteps explicit discovery code — Envoy / linkerd2-proxy intercept egress and handle resolution + LB transparently.

Concepts (deep dive)

Why discovery exists

In a static world: orders.prod.contoso.com → fixed IP → done. In a dynamic world:

deploy v2 → 5 new pod IPs       autoscale +3 → 8 IPs
deploy v3 → 5 new pod IPs       crash pod   → 7 IPs
deploy rollback → 5 IPs         k8s reschedule → different node, new IP

Hard-coding IPs guarantees outages. Discovery is the indirection that maps a logical name (orders) to current healthy instances.

Client-side discovery

   client ──(1) query──► registry
          ◄─(2) list of instances─
          ──(3) pick + call──► instance B

Client owns LB choice (RR / P2C / consistent hash). Saves a network hop. Used by Netflix Ribbon (legacy), gRPC's dns:/// resolver, Aspire's ServiceDiscovery.

Pros: no extra infra; fewer hops; smart, app-aware LB. Cons: every language needs a client library; clients carry registry credentials; harder for polyglot fleets.

Server-side discovery

   client ──► [ LB / gateway ] ──► instance B
                  ▲
                  └─── registry (LB updates pool)

Client just calls a stable LB endpoint. The LB watches the registry and updates its backend pool. Kubernetes Service, ELB target groups, Application Gateway backend pools, YARP with destinations are all server-side.

Pros: language-agnostic clients; centralized policy; simple from client perspective. Cons: extra hop; LB is on the data path.

Registries

DNS

The OG service discovery. Lookup orders.svc.cluster.local → IP(s).

$ dig orders.default.svc.cluster.local
;; ANSWER SECTION:
orders.default.svc.cluster.local. 30 IN A 10.0.1.10

DNS SRV records carry port + priority + weight:

_grpc._tcp.orders  IN  SRV  10 5 5000  pod-a
_grpc._tcp.orders  IN  SRV  10 5 5000  pod-b

DNS pitfalls: - TTL caching. Resolvers (and .NET's HttpClient) cache. After scale-in, dead IPs linger until TTL expires. - A record returns one or more IPs, but no health info — caller has to retry on connection failure. - No push: client polls; updates lag. - HttpClient and connection pooling: connections opened to an old IP keep going there until the connection closes or PooledConnectionLifetime triggers a re-resolve.

// Force periodic re-resolution
builder.Services.AddHttpClient("orders")
    .ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
    {
        PooledConnectionLifetime = TimeSpan.FromMinutes(2)
    });

Consul

KV store + health checking + multi-DC + DNS interface. Each service registers itself (or sidecar registers it) with health checks. Consul queries return only healthy instances.

GET /v1/health/service/orders?passing=true

Strong DC-aware story; gossip protocol (SWIM) for membership. Heavier than DNS, lighter than ZK.

etcd

Strongly consistent KV (Raft). Backs Kubernetes itself. Rarely used as a direct discovery API by apps — usually through Kubernetes.

ZooKeeper

Legacy. Hadoop / Kafka / older JVM stacks. Strongly consistent. Ops complexity is high. Avoid for greenfield.

Eureka

Netflix OSS. AP-favoring. Self-preservation mode keeps stale registrations during partitions — debatable design. JVM-leaning; rarely a fit for new .NET work.

Kubernetes Services + EndpointSlices

The default registry inside K8s.

Service "orders" (ClusterIP 10.96.4.20)
   │
   └─► EndpointSlices (per-pod entries, ready / not-ready, addresses, ports)
           │
           └─► CoreDNS resolves orders.<ns>.svc.cluster.local

EndpointSlices scale better than the legacy Endpoints object (sliced for large services, more efficient watch).
Headless service (clusterIP: None): DNS returns all pod IPs — for client-side LB.
publishNotReadyAddresses can include not-ready pods (rare; for stateful sets needing peer discovery).

Cloud-native: Azure Service Fabric, Consul on Azure, Azure Service Discovery features

Service Fabric has its own naming service. App Service / Container Apps expose stable URLs (the platform handles discovery internally).

.NET ecosystem: Microsoft.Extensions.ServiceDiscovery

Shipped with .NET Aspire. Abstraction layer for resolving logical names to physical endpoints.

builder.Services.AddServiceDiscovery();

builder.Services.AddHttpClient<OrdersClient>(c =>
{
    c.BaseAddress = new("https+http://orders");   // logical scheme
})
.AddServiceDiscovery();

The scheme https+http://orders says: "prefer https, fall back to http; resolve orders via configured resolvers."

Resolvers (composable, in order):

builder.Services.AddServiceDiscovery(o =>
{
    o.RefreshPeriod = TimeSpan.FromSeconds(10);
})
.AddConfigurationServiceEndpointProvider()    // appsettings: "Services:orders" 
.AddDnsServiceEndpointProvider()              // standard A records
.AddDnsSrvServiceEndpointProvider();          // SRV records (K8s headless, Consul DNS)

appsettings.json:

{
  "Services": {
    "orders": {
      "https": ["orders.prod.contoso.com"],
      "http":  ["orders.internal:5000"]
    }
  }
}

Aspire orchestration injects these for local dev so https+http://orders Just Works between projects.

Health-aware discovery

The registry must reflect health, not just presence. Otherwise discovery returns IPs that won't answer.

Kubernetes: readiness probe failure removes the pod from EndpointSlices.
Consul: registered TTL/HTTP health checks; only passing instances returned.
Aspire / Microsoft.Extensions.ServiceDiscovery: integrates with health checks via the resolver's filtering logic; underlying transport (HTTP) does retry on dead endpoints.

Without health-awareness:

client → resolver → list [A, B(dead), C]
client → A ✓ → C ✓ → B ✗ → retry … wasted RTTs

With it:

registry already excluded B; client sees [A, C]

Stale registry (the silent killer)

event:           registry update:        observed by clients:
pod B dies   →   ~5s (probe + ttl)   →   could be 30s with HTTP keep-alive
new pod C    →   ~10s (rolling)      →   could be 60s with DNS cache

Defenses: 1. Short TTLs (10–30s) on DNS. Trade against query load. 2. Push-based registries (Consul watches, K8s informers, Eureka deltas) propagate faster. 3. Client retry on connection failure with re-resolution. 4. SocketsHttpHandler.PooledConnectionLifetime so HTTP connection reuse doesn't outlast the IP. 5. Outlier ejection at LB / mesh to remove silent failers fast.

Comparison to mesh-based discovery

Service mesh (Istio, Linkerd, Consul Connect) injects a sidecar that intercepts outbound traffic.

   app code                       mesh-data plane
   --------                       ----------------
   GET https://orders/...   ──►   envoy sidecar
                                       │ (knows endpoints from xDS / control plane)
                                       └──► healthy orders pod

App code is unchanged. The mesh handles discovery, LB, retries, mTLS. Centralizes traffic policy in the control plane (Istio Pilot, Linkerd destination service).

Trade-off: mesh = more moving parts. For 3 services, overkill. For 30 services across teams, often the right answer.

How it works under the hood

Kubernetes Service resolution end-to-end

1. Pod starts → readiness probe passes → kubelet PATCHes pod status
2. EndpointSlice controller updates EndpointSlice for the matching Service
3. kube-proxy on every node watches EndpointSlices
   → installs iptables / IPVS rules pointing ClusterIP at pod IPs
4. CoreDNS watches Services (not EndpointSlices) → returns ClusterIP for the name
5. Client resolves orders.svc.cluster.local → ClusterIP
6. Packet to ClusterIP → kube-proxy DNAT → random pod IP

Two control planes are at play: DNS for the name, kube-proxy for the routing. Watches are async; eventual.

Aspire ServiceDiscovery resolver pipeline

HttpClient sends GET https+http://orders/api/v1/items
    │
    ▼
ServiceDiscoveryHttpMessageHandler intercepts
    │  resolves "orders" via providers in order:
    │    Configuration → DNS → DNS SRV
    │  picks scheme (https preferred), picks endpoint (round-robin or random)
    ▼
rewrites URL → https://orders-pod-7.contoso.com/api/v1/items
    │
    ▼
delegating handler chain → SocketsHttpHandler → wire

Endpoints are cached per RefreshPeriod; HttpClient retries on transient failures via Microsoft.Extensions.Http.Resilience.

DNS SRV-based discovery

$ dig SRV _orders._tcp.contoso.com
_orders._tcp  10 IN  SRV  10 50  5000 pod-a.contoso.com
_orders._tcp  10 IN  SRV  10 50  5000 pod-b.contoso.com

Resolver returns priority + weight + port + target. Aspire's DNS SRV provider, gRPC's xDS resolver, and Consul's DNS interface all use SRV.

Code: correct vs wrong

❌ Wrong: hard-coded URL in config

{ "OrdersApi": "https://orders-pod-7.contoso.com:5000" }

Pod gets rescheduled → 5xx forever.

✅ Correct: logical name + discovery

builder.Services.AddServiceDiscovery();
builder.Services.AddHttpClient<OrdersClient>(c => c.BaseAddress = new("https+http://orders"))
    .AddServiceDiscovery();

❌ Wrong: long-lived `HttpClient` connection pool with no resolution refresh

services.AddSingleton<HttpClient>();   // pool keeps using the same dead IP

✅ Correct: bounded connection lifetime

services.AddHttpClient<OrdersClient>()
    .ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
    {
        PooledConnectionLifetime = TimeSpan.FromMinutes(2)   // forces re-resolve
    })
    .AddServiceDiscovery();

❌ Wrong: no health filter

// Round-robin over all known instances, including ones that crashed 30s ago

✅ Correct: registry / mesh that filters on health

# K8s: readiness probe drives EndpointSlice membership; only ready pods receive traffic
readinessProbe:
  httpGet: { path: /health/ready, port: 5000 }
  periodSeconds: 5

❌ Wrong: relying on DNS cache for fast failover

TTL=300 → pod replaced → clients try dead IP for 5 minutes

✅ Correct: short TTL + connection rotation + retries

Service DNS TTL = 30s
Client PooledConnectionLifetime = 2m
Polly retry-on-connection-failure with re-resolve

Design patterns for this topic

Pattern 1 — "Logical names + Aspire ServiceDiscovery"

Intent: decouple code from environment; let resolvers handle the rest.

Pattern 2 — "Headless service + client-side LB"

Intent: per-request fairness for HTTP/2 / gRPC; cache-locality with consistent hash.

Pattern 3 — "Mesh-managed discovery"

Intent: uniform policy across heterogeneous services; sidecars do the work.

Pattern 4 — "Short TTL + connection lifetime + retry"

Intent: bound staleness without pummelling DNS.

Pattern 5 — "Health-driven membership"

Intent: registry returns only ready instances; clients don't carry the burden.

Pros & cons / trade-offs

Approach	Pros	Cons
Client-side discovery	No extra hop; smart LB	Per-language libs; clients hold creds
Server-side (LB)	Language-agnostic	Extra hop; LB on data path
DNS	Universal; simple	TTL staleness; no health
Consul	Multi-DC; rich health	Operate it
Kubernetes Services	Built-in	Cluster-bound
Aspire ServiceDiscovery	First-class .NET; pluggable	Newer; .NET-only client
Service mesh	Transparent; uniform	Operational complexity

When to use / when to avoid

Use Kubernetes Services + Aspire ServiceDiscovery for .NET on AKS — it's the path of least resistance.
Use Consul for multi-DC, mixed (non-K8s) fleets.
Use a service mesh when you have many services with shared traffic policy needs.
Avoid ZooKeeper / Eureka for new .NET work.
Avoid DNS-only discovery without thinking about TTLs and connection pools.
Avoid writing your own registry — solved problem.

Interview Q&A

Q1. Why is service discovery needed? Dynamic IPs, autoscaling, deploys. Hard-coded endpoints break. Discovery maps logical names → current healthy instances.

Q2. Client-side vs server-side discovery? Client-side: client queries registry, picks instance directly (smart LB, no extra hop). Server-side: client calls a stable LB; LB queries registry (language-agnostic, extra hop).

Q3. Why is DNS alone insufficient for fast failover? TTL caching; HTTP connection reuse to dead IPs; no health info — DNS just returns the A records. You need short TTLs, bounded connection lifetimes, and retry logic.

Q4. What does Microsoft.Extensions.ServiceDiscovery give you? A https+http://name scheme over pluggable resolvers (config, DNS, DNS SRV, K8s) integrated with HttpClientFactory. Aspire wires it in for dev and prod.

Q5. Kubernetes EndpointSlices vs Endpoints? EndpointSlices are sharded — much better at scale. The legacy Endpoints object had to fit a whole service in one record.

Q6. What's a headless Service? clusterIP: None. DNS returns all pod IPs instead of a single VIP. Used for client-side LB and stateful peer discovery.

Q7. How does a service mesh handle discovery? Sidecar (Envoy / linkerd-proxy) intercepts egress, gets endpoints from the control plane (xDS / destination service), and balances. App code unchanged.

Q8. What's the "stale registry" problem? Registry can list dead instances or miss live ones due to TTLs, watch lag, or HTTP connection reuse. Mitigations: short TTLs, push-based registries, retry-with-reresolve, outlier ejection.

Q9. SRV records — why use them? They carry port, priority, weight — multi-instance + ordering info DNS A records can't. Used by Consul DNS, K8s headless services, gRPC.

Q10. Eureka vs Consul? Eureka: AP, Netflix-OSS, JVM-leaning, self-preservation mode. Consul: CP-ish, multi-DC, polyglot. For .NET, Consul or K8s Services beat Eureka.

Q11. How do you keep HTTP clients from sticking to dead IPs? Set SocketsHttpHandler.PooledConnectionLifetime to a short value (1–5 min) so connections get rebuilt and DNS re-resolved.

Q12. When is a service mesh overkill for discovery? Few services, single team, simple traffic policy. The mesh's operational complexity outweighs benefits below a certain scale.

Gotchas / common mistakes

⚠️ Long DNS TTLs + long-lived HTTP connections — clients call dead IPs long after pods die.
⚠️ Singleton HttpClient without lifetime control — same problem.
⚠️ No readiness probe — registry lists "running" pods that aren't actually ready.
⚠️ Putting credentials in the registry path — clients shouldn't authenticate to discover.
⚠️ Cross-cluster / cross-region discovery via cluster DNS — won't resolve; use Front Door / Traffic Manager or a federated mesh.
⚠️ Watching for "all instances" instead of "healthy instances" — you'll route to dead pods.
⚠️ Different services with the same name in different namespaces / DCs — fully-qualify or scope them.