Kubernetes for .NET

Key Points

Liveness, readiness, startup probes are essential. .NET endpoint: /healthz, /healthz/ready, /healthz/live.
Graceful shutdown: handle SIGTERM → drain in-flight requests → finish background work → exit. ASP.NET Core does this with IHostedService.StopAsync. Set Kubernetes terminationGracePeriodSeconds: 30+.
Resource limits: requests + limits for CPU and memory. Without limits, one pod can OOM the node. .NET's GC respects container limits via DATAS in .NET 8+.
Configuration: ConfigMaps for non-secret; Secrets for secrets (encrypt etcd; consider External Secrets / Sealed Secrets / Azure Key Vault CSI driver).
Common mistakes: no probes; no graceful shutdown; logging to file (containers should log stdout/stderr).

Concepts (deep dive)

What Kubernetes is

Kubernetes (K8s) is a container orchestrator: it takes a fleet of machines (the cluster), a set of declared workload specs (pods, deployments, services), and reconciles "what's running" toward "what was declared." You don't tell K8s how to start a container on which host — you tell it what should exist, and the control plane figures out the placement, restarts failed pods, reschedules them when nodes die, and routes traffic to the survivors.

           ┌────────────────────────────────────────────────────────┐
           │                  control plane                         │
           │  ┌──────────┐  ┌──────────────┐  ┌─────────────────┐  │
           │  │ api server│  │ scheduler    │  │ controllers     │  │
           │  │ + etcd   │  │ (placement)  │  │ (reconcile)     │  │
           │  └────┬─────┘  └──────┬───────┘  └────────┬────────┘  │
           └───────┼───────────────┼───────────────────┼───────────┘
                   │               │                   │
              kubectl apply   places pods         restarts/replaces
                   │
                   ▼
           ┌─────────────────────────────────────────────────────┐
           │           worker nodes (the data plane)             │
           │  ┌──────────┐   ┌──────────┐   ┌──────────┐         │
           │  │ kubelet  │   │ kubelet  │   │ kubelet  │         │
           │  │ pods…    │   │ pods…    │   │ pods…    │         │
           │  └──────────┘   └──────────┘   └──────────┘         │
           └─────────────────────────────────────────────────────┘

The smallest unit is a pod — one or more containers that share network and storage namespaces and are scheduled together. A pod is mortal; a Deployment is the desired state ("3 replicas of this pod template, always"). A Service is a stable virtual IP and DNS name that load-balances across whichever pods currently match its label selector. An Ingress is the cluster's HTTP front door.

Why this matters for .NET shops: K8s gives you self-healing, declarative scaling, and a uniform deployment shape across cloud vendors — at the cost of a steep operational learning curve. For most .NET apps that don't need that much control, Azure Container Apps or App Service (Azure App Service & Functions, Azure Container Apps & AKS) is the right choice; this file is about the layer underneath.

The pod lifecycle (and why probes exist)

A pod's life is a small state machine, and most production K8s bugs come from misunderstanding the transitions:

   pending ──► creating ──► running ──► terminating ──► gone
                              │ ▲
            startupProbe ─────┘ │
            (until ready)       │
            readinessProbe ─────┘ (re-evaluated continuously)
            livenessProbe ──────► restart on failure

Startup probe runs first — until it succeeds, the other probes don't run. This is for slow-starting apps (warmup, large data loads); without it, a long startup can be misread as a liveness failure and the pod restart-loops.
Readiness probe decides whether the pod receives traffic from its Service. Failing readiness pulls the pod out of rotation without restarting it — useful for "I'm temporarily overwhelmed" or "my DB is briefly unreachable."
Liveness probe decides whether the pod is fundamentally broken and should be restarted. Liveness failure kills and recreates the container.

The single most important rule: liveness should be cheap and local. If your liveness probe checks the database, a brief DB outage restarts every pod in the cluster — and the pods have nothing wrong with them. Readiness can be deep (DB, downstream APIs); liveness should be "is the process responsive at all."

Probes

Three probe types, each declared as an HTTP endpoint K8s polls on a schedule. ASP.NET Core's health-checks framework is built to back them: register checks with tags (ready, live), then map each tag to its own endpoint.

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: api
        image: myapp:1.0
        ports: [{ containerPort: 8080 }]
        startupProbe:
          httpGet: { path: /healthz/startup, port: 8080 }
          failureThreshold: 30
          periodSeconds: 10
        readinessProbe:
          httpGet: { path: /healthz/ready, port: 8080 }
          periodSeconds: 5
        livenessProbe:
          httpGet: { path: /healthz/live, port: 8080 }
          periodSeconds: 10
          initialDelaySeconds: 30

In ASP.NET Core:

builder.Services.AddHealthChecks()
    .AddDbContextCheck<AppDb>(tags: new[] { "ready" })
    .AddCheck("self", () => HealthCheckResult.Healthy(), tags: new[] { "live" });

app.MapHealthChecks("/healthz/ready", new() { Predicate = c => c.Tags.Contains("ready") });
app.MapHealthChecks("/healthz/live", new() { Predicate = c => c.Tags.Contains("live") });
app.MapHealthChecks("/healthz/startup");

Probe	Failure means
Startup	App never came up; Kubernetes restarts pod
Readiness	App not ready (DB down, warming up); pod removed from Service routing
Liveness	App is broken; Kubernetes restarts

Key rule: liveness should NOT check downstream deps. Readiness can.

Graceful shutdown

When K8s wants to remove a pod (deploy, scale-down, node drain), it sends SIGTERM and gives the container terminationGracePeriodSeconds to exit cleanly. ASP.NET Core catches SIGTERM, stops accepting new requests, lets in-flight requests finish, runs IHostedService.StopAsync on every hosted service, then exits. If the process is still alive after the grace period, K8s sends SIGKILL and drops everything.

The subtle part: the pod is removed from the Service's load balancer asynchronously with the SIGTERM, so for a brief window the LB may still send new requests to a SIGTERM'd pod. The preStop sleep 5 is there to delay the SIGTERM long enough for the LB to actually stop routing.

1. K8s sends SIGTERM
2. Pod's preStop hook runs (optional)
3. Pod removed from Service endpoints (eventually)
4. K8s waits terminationGracePeriodSeconds
5. SIGKILL if still running

ASP.NET Core handles SIGTERM:

builder.Host.ConfigureHostOptions(o =>
{
    o.ShutdownTimeout = TimeSpan.FromSeconds(30);
});

App stops accepting new requests; in-flight finish; IHostedService.StopAsync runs.

spec:
  terminationGracePeriodSeconds: 60
  containers:
  - lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "-c", "sleep 5"]   # let LB drain

Resource limits

Two numbers control how much CPU and memory a pod can use, and they mean different things. The request is what the scheduler reserves for the pod — the node guarantees this is available. The limit is the hard ceiling: exceed the memory limit and the kernel OOM-kills the container; exceed the CPU limit and the kernel throttles it. Without limits, one runaway pod can consume the whole node and starve everything else.

For .NET, the GC needs to know the memory ceiling so it can size the heap. Modern .NET (8+) reads the cgroup limit automatically via DATAS (Dynamic Adaptation To Application Sizes) and resizes the heap as pressure changes. Pre-.NET 8 needed DOTNET_GCHeapHardLimit set manually.

resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"
    cpu: "1000m"

request: scheduler reserves; pod gets at least this.
limit: max usage; OOM-killed beyond memory; throttled beyond CPU.

For .NET 8+: GC adapts to container memory limit (DATAS — Dynamic Adaptation To Application Sizes). Set MaxRAMPercentage if needed:

DOTNET_GCDynamicAdaptationMode=1     # default in .NET 8+
DOTNET_GCHeapHardLimitPercent=75

Sidecars

A sidecar is a second container that runs in the same pod as your app container, sharing its network namespace and (optionally) its volumes. The pattern exists because pods are the unit of scheduling: anything you put in the same pod is colocated, lifecycled together, and reachable on localhost. Sidecars handle cross-cutting concerns (telemetry, service-mesh proxying, log shipping, secret refresh) without modifying the main app image.

spec:
  containers:
  - name: api
    image: myapp:1.0
  - name: otel-collector
    image: otel/opentelemetry-collector

Common sidecars: - OpenTelemetry collector - Service mesh proxy (Envoy/Linkerd) - Log shipper (Fluent Bit) - Secrets agent (Vault Agent)

ConfigMaps + Secrets

K8s separates non-secret config (ConfigMaps) from secret config (Secrets) so RBAC and audit can treat them differently. Both can be mounted into pods as files or projected as environment variables; the choice usually comes down to whether the app reads IConfiguration (env vars are easy) or expects a config file (file mount).

apiVersion: v1
kind: ConfigMap
metadata: { name: myapp-config }
data:
  appsettings.Production.json: |
    { "Logging": { "LogLevel": { "Default": "Information" } } }

---
apiVersion: v1
kind: Secret
metadata: { name: myapp-secrets }
type: Opaque
data:
  ConnectionStrings__Default: <base64>

Mount as files or env vars:

envFrom:
- configMapRef: { name: myapp-config }
- secretRef: { name: myapp-secrets }

Secrets in Kubernetes

K8s Secrets are base64, not encrypted by default. Best practices:

Enable etcd encryption-at-rest.
RBAC to limit secret reads.
External Secrets Operator or Azure Key Vault CSI Driver — secrets pulled from cloud KMS at runtime.
Sealed Secrets — encrypted in git; decrypted in cluster.

HPA (Horizontal Pod Autoscaler)

HPA watches a metric (CPU, memory, or a custom signal) and adjusts the replica count of a Deployment to keep that metric near a target. It scales horizontally (more pods), not vertically (bigger pods) — sized for stateless web/API workloads. The trade-off: HPA needs a metric that actually correlates with load, and cpu averageUtilization is a poor proxy for many .NET workloads where the bottleneck is I/O wait, not CPU.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: myapp }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: myapp }
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource: { name: cpu, target: { type: Utilization, averageUtilization: 70 } }

Custom metrics (RPS, queue depth) via metrics-server + Prometheus adapter.

KEDA (event-driven autoscaling)

HPA scales on metrics the pod itself emits. KEDA scales on metrics outside the pod — queue depth, topic lag, schedule, external HTTP endpoints. For a queue worker, the right metric is "messages waiting," not CPU; KEDA reads that from the source (Service Bus, Event Hubs, Kafka, Redis Streams, …) and scales the Deployment accordingly. It also enables scale-to-zero: zero replicas when the queue is empty, ramp up when work appears.

For queue-based or event-driven workers:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
spec:
  triggers:
  - type: azure-servicebus
    metadata:
      queueName: orders
      messageCount: "5"   # scale up if backlog

Scale to zero when no work; scale up on backlog. Used heavily by Azure Container Apps.

Service & Ingress

A Service is K8s's internal load balancer: a stable virtual IP + DNS name that fronts a fluctuating set of pods (selected by labels). Pods come and go; the Service IP doesn't. An Ingress is the HTTP-aware entry point from outside the cluster — host- and path-based routing, TLS termination, sometimes WAF — delegating to Services internally. The Ingress is implemented by an Ingress controller (NGINX, Traefik, Application Gateway Ingress Controller) that you install in the cluster.

apiVersion: v1
kind: Service
metadata: { name: myapp }
spec:
  selector: { app: myapp }
  ports: [{ port: 80, targetPort: 8080 }]

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: { name: myapp }
spec:
  ingressClassName: nginx
  rules:
  - host: api.contoso.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service: { name: myapp, port: { number: 80 } }

Logging

Log to stdout/stderr as JSON. K8s captures; ship to Loki/Elasticsearch/CloudWatch via DaemonSet.

builder.Logging.AddJsonConsole();

Don't write to files in container.

Networking essentials

NetworkPolicy for pod-to-pod restriction.
Service mesh (Linkerd, Istio) for mTLS, retries, observability.
Ingress controller (NGINX, Traefik, Application Gateway Ingress).

Workload identity (Azure AD)

The pre-workload-identity story was ugly: store an Entra ID client secret in a K8s Secret, mount it into the pod, hope nobody leaked it. Workload identity replaces that with a token exchange: the pod authenticates as its K8s service account, which is federated to an Entra ID identity, which is granted RBAC on Azure resources. No secret ever lives in the cluster; the federation trusts the service account's projected token.

metadata:
  annotations:
    azure.workload.identity/client-id: <aad-app-id>
spec:
  serviceAccountName: myapp-sa

Pods authenticate to Azure AD via service account → no secrets in pod.

Common pitfalls

No probes → flapping pods.
Liveness checks downstream → cascading restarts.
No terminationGracePeriodSeconds → in-flight requests dropped.
No resource limits → noisy neighbor.
Secrets in env vars → leaked in process listings.

.NET Aspire deployment

Aspire (Aspire — Cloud-Native) generates K8s manifests via azd or Bicep — saves manual YAML.

Code: correct vs wrong

❌ Wrong: liveness checks DB

livenessProbe:
  httpGet: { path: /healthz/full }   # DB outage → restart pod → cascading

✅ Correct: liveness simple, readiness deep

livenessProbe:  { path: /healthz/live }   # is process alive?
readinessProbe: { path: /healthz/ready }  # can it serve?

❌ Wrong: ungraceful shutdown

// Default 5s shutdown timeout

Long requests cut off.

✅ Correct: longer shutdown + K8s grace

o.ShutdownTimeout = TimeSpan.FromSeconds(30);
// terminationGracePeriodSeconds: 60

Design patterns for this topic

Pattern 1 — "Three probes (startup, readiness, liveness)"

Intent: correct lifecycle.

Pattern 2 — "Graceful shutdown with grace period"

Intent: drain in-flight.

Pattern 3 — "Workload identity over secrets"

Intent: zero-secret cloud auth.

Pattern 4 — "KEDA for event-driven scale"

Intent: scale to zero.

Pattern 5 — "Sidecars for cross-cutting"

Intent: OTel, mesh, secrets agent.

Pros & cons / trade-offs

Aspect	Pros	Cons
K8s	Powerful; portable	Operational complexity
Probes	Self-healing	Mistuned = flapping
KEDA	Cost-efficient	Cold start
Sidecars	Modular	Resource overhead

When to use / when to avoid

Use K8s for multi-service, scaling needs.
Use Container Apps / App Service if K8s is overkill.
Avoid rolling your own k8s for a single app.

Interview Q&A

Q1. Three probe types? Startup (initial), readiness (in service routing), liveness (restart if dead).

Q2. Why liveness shouldn't check downstreams? DB outage cascades to pod restarts → no recovery; mass restart.

Q3. Graceful shutdown sequence? SIGTERM → preStop → drained from Service → ShutdownTimeout → SIGKILL if still alive.

Q4. Resource requests vs limits? Request: reserved. Limit: max. Memory limit exceeded = OOM kill; CPU = throttle.

Q5. .NET GC in containers? .NET 8+ DATAS adapts heap to container memory limit. No tuning needed for most apps.

Q6. ConfigMap vs Secret? ConfigMap: non-secret config. Secret: sensitive data; base64 (NOT encrypted).

Q7. Workload identity? Pod's service account ↔ cloud IAM (AAD/AWS IRSA). No secret stored in pod.

Q8. KEDA? Event-driven autoscaler. Scales on queue depth, custom metrics. Scale-to-zero.

Q9. Logging in K8s? stdout/stderr as JSON; cluster log shipper aggregates.

Q10. Sidecar use cases? OTel collector, service mesh proxy, secrets agent, log forwarder.

Q11. Why HPA over manual scaling? Auto-respond to load. CPU/memory-based default; custom metrics possible.

Q12. Service mesh — when? Many services + mTLS / observability / fine-grained routing needs.

Gotchas / common mistakes

⚠️ No probes — uncontrolled.
⚠️ Liveness on downstream — cascades.
⚠️ No grace period — dropped requests.
⚠️ Secrets as env vars vs files — leaked easily.
⚠️ Missing resource limits — noisy neighbor.