Diagnostics Tools

Key Points

dotnet-counters — live metrics (CPU, GC, threadpool, custom Meter). Free; built-in.
dotnet-trace — collect ETW/EventSource traces; analyze in PerfView/Speedscope.
dotnet-dump — process dumps for post-mortem; analyze with WinDbg/dotnet-dump CLI.
dotnet-gcdump — heap snapshot; find leaks.
dotnet-monitor — sidecar; HTTP API for diagnostics in containers.
PerfView (Windows) — deep ETW analysis. JetBrains dotMemory / dotTrace — commercial, polished.
Threadpool starvation is the most common production .NET pathology — diagnose with dotnet-counters (threadpool-thread-count, threadpool-queue-length).

Concepts (deep dive)

dotnet-counters

dotnet tool install -g dotnet-counters
dotnet-counters monitor --process-id 1234 --counters System.Runtime,Microsoft.AspNetCore.Hosting

Live counter values:

[System.Runtime]
    CPU Usage (%)                                       12
    Working Set (MB)                                  1024
    GC Heap Size (MB)                                  256
    Gen 0 GC Count (Count / 1 sec)                       2
    ThreadPool Thread Count                              16
    ThreadPool Queue Length                              0
    ThreadPool Completed Work Item Count / sec        4523

[Microsoft.AspNetCore.Hosting]
    Requests/sec                                      1200
    Total Requests                                  500000
    Failed Requests                                     12

Threadpool queue length climbing = starvation. Likely causes: sync-over-async, blocking I/O, lock contention.

dotnet-trace

dotnet-trace collect --process-id 1234 --duration 00:00:30 \
  --providers System.Runtime,Microsoft-Windows-DotNETRuntime

Outputs .nettrace — open in PerfView, Visual Studio, or Speedscope (web).

For CPU profiling:

dotnet-trace collect -p 1234 --profile cpu-sampling --duration 00:00:30

dotnet-dump

dotnet-dump collect -p 1234
dotnet-dump analyze ./core_xxx

Analyze interactively — clrstack, dumpheap -stat, dso (dump stack objects), syncblk.

> dumpheap -stat
              MT    Count    TotalSize Class Name
00007ffac3a4d8d0  120000     19200000 System.String
00007ffac3a4f490   84000     13440000 System.Byte[]
...

dotnet-gcdump

dotnet-gcdump collect -p 1234

Smaller than full dump; just managed heap. Open in Visual Studio or dotnet-gcdump report.

Useful for finding memory leaks: take two snapshots over time; diff; find growing types.

dotnet-monitor

# As a sidecar in K8s:
- name: monitor
  image: mcr.microsoft.com/dotnet/monitor
  ports: [{ containerPort: 52323 }]

REST API for triggering dumps, traces, counters from outside the container — ideal for production.

GET /processes
POST /trace?pid=1
POST /dump?pid=1
GET /counters?pid=1

EventSource / EventListener

Custom EventSource for instrumenting your code:

[EventSource(Name = "MyApp.Orders")]
public sealed class OrderEvents : EventSource
{
    public static readonly OrderEvents Log = new();

    [Event(1, Level = EventLevel.Informational, Message = "Order {0} placed")]
    public void OrderPlaced(string orderId) => WriteEvent(1, orderId);
}

OrderEvents.Log.OrderPlaced("ORD-123");

Pickup with dotnet-counters / dotnet-trace via provider name.

Common pathologies

Threadpool starvation

Symptom: requests queue up; latency climbs; CPU not maxed.

Diagnose: threadpool-queue-length rising.

Cause: sync-over-async (.Result, .Wait()), blocking I/O, lock contention.

Fix: full async/await; remove .Result; use SemaphoreSlim.WaitAsync.

GC pressure

Symptom: CPU bursts; latency spikes during GC.

Diagnose: gc-heap-size, gen-2-gc-count high.

Cause: too many allocations, large objects.

Fix: pooling (ArrayPool, ObjectPool), Span<T>, structs over classes for ephemeral data.

Memory leak

Symptom: working set climbs; eventually OOM.

Diagnose: dotnet-gcdump over time; find type with growing count.

Cause: event handler not unsubscribed, static cache without eviction, captured DI scope.

Fix: dispose properly; use WeakReference for caches; check static collections.

Connection leak

Symptom: SqlException timeout under load.

Diagnose: connection-pool counters; dotnet-counters Microsoft.Data.SqlClient.EventSource.

Cause: unawaited using, exception path skipping Dispose.

Fix: await using for IAsyncDisposable; ensure all paths dispose.

Lock contention

Symptom: CPU low but throughput poor.

Diagnose: dotnet-trace profile cpu-sampling; look for Monitor.Enter hotspots.

Cause: hot lock; bad cache invalidation pattern.

Fix: lock-free structures (ConcurrentDictionary), ReaderWriterLockSlim, partitioning.

PerfView (Windows)

Most powerful — but Windows-only and complex. Captures ETW (Event Tracing for Windows) traces with full stack info. Used to debug GC pauses, JIT compilation, threadpool issues.

JetBrains dotTrace / dotMemory

Polished commercial tools. dotTrace for CPU profiling, dotMemory for heap analysis. Worth it for serious perf work.

Visual Studio Profiler

Bundled with VS Enterprise. Diagnostic tools window during debugging shows live counters.

Application Insights / Datadog APM

Production observability. Live metrics; transaction profiler captures slow requests automatically with stacks.

Crash dumps in production

Configure auto-dump on unhandled exception:

# environment variables in container
DOTNET_DbgEnableMiniDump=1
DOTNET_DbgMiniDumpType=4   # FullDump
DOTNET_DbgMiniDumpName=/dumps/dump_%d.dmp

Volume-mount /dumps to persistent storage.

Logging vs profiling

Logs: events with context; cheap; always on.
Metrics: aggregated numbers; very cheap; always on.
Traces: causal flow; sampled.
Profiles: CPU/memory deep-dive; on-demand only (expensive).

Don't always-on profile production.

Diagnostics in containers

# Attach to running container:
kubectl exec -it pod-xxx -- bash
dotnet-counters monitor -p 1

Or use dotnet-monitor sidecar — HTTP-driven; no exec.

Heap analysis tips

# Find large strings
> dumpheap -stat -mt <System.String MT>
> dumpheap -mt <MT> -min 1000

# Find roots holding objects alive
> gcroot <addr>

Async stack diagnostics

dotnet-stack (preview): captures async-aware stacks for live processes.

dotnet-stack report -p 1234

Shows live tasks and their pending awaits — invaluable for "why is my app stuck?".

Code: correct vs wrong

❌ Wrong: ignoring counters

"Latency is bad" → restart pod → fixed?

✅ Correct: investigate

dotnet-counters monitor -p $(pgrep -f myapp) --counters System.Runtime
# Spot threadpool queue rising → sync-over-async culprit

❌ Wrong: heap dump under load

A full dump pauses the process. In production, use dotnet-monitor triggers; collect during low-traffic windows or on-event.

✅ Correct: dotnet-monitor automated dump on memory threshold

{
  "CollectionRules": {
    "OnHighMemory": {
      "Trigger": { "Type": "EventCounter", "Settings": { "ProviderName": "System.Runtime", "CounterName": "working-set", "GreaterThan": 2048 } },
      "Actions": [{ "Type": "CollectDump" }]
    }
  }
}

Design patterns for this topic

Pattern 1 — "Counters first, then trace, then dump"

Intent: cheapest tool first.

Pattern 2 — "dotnet-monitor sidecar"

Intent: production diagnostics without exec.

Pattern 3 — "Auto-dump on crash"

Intent: post-mortem without repro.

Pattern 4 — "EventSource for domain events"

Intent: structured custom counters.

Pattern 5 — "Threadpool watchdog"

Intent: alert on queue length spike.

Pros & cons / trade-offs

Tool	Pros	Cons
dotnet-counters	Live; cheap	Live only
dotnet-trace	Detailed	Tooling required to read
dotnet-dump	Full state	Pauses process
dotnet-monitor	Container-friendly	Sidecar overhead
PerfView	Most powerful	Windows-only; complex
dotMemory	Polished	Commercial

When to use / when to avoid

Always counters in production (dotnet-monitor / metrics export).
Use trace for CPU/perf investigations.
Use dump only when needed (pauses).
Avoid PerfView unless on Windows + complex problems.

Interview Q&A

Q1. How diagnose threadpool starvation? dotnet-counters — threadpool-queue-length rising indicates queued work waiting for threads. Common cause: sync-over-async.

Q2. dotnet-counters vs dotnet-trace? Counters: live aggregated values (cheap). Trace: detailed event stream (collected for analysis).

Q3. What's dotnet-gcdump? Captures managed heap. Smaller and faster than full dump. Diff over time → find leaks.

Q4. Production diagnostics in containers? dotnet-monitor sidecar exposes HTTP API for traces, dumps, counters.

Q5. Common .NET production pathology? Threadpool starvation from sync-over-async. Diagnose: queue-length counter. Fix: full async/await.

Q6. Auto-dump on crash? DOTNET_DbgEnableMiniDump=1 env var. Mount volume for /dumps.

Q7. EventSource use case? Custom domain events emitted to ETW/dotnet-trace. Aggregate via dotnet-counters.

Q8. dotnet-stack? Preview tool. Captures async stack of live tasks — debug hangs.

Q9. Lock contention diagnosis? dotnet-trace cpu-sampling; look for Monitor.Enter in hot stacks.

Q10. When dump vs trace? Trace for "what is happening over time". Dump for "what is the current state".

Q11. PerfView vs dotnet-trace? Same data (ETW). PerfView reads richer; dotnet-trace is x-plat collection.

Q12. dotnet-monitor security? Disable global metrics collection; restrict actions; use auth on HTTP endpoints.

Gotchas / common mistakes

⚠️ Reaching for dump first — counter may answer cheaper.
⚠️ Full dump in prod — pauses; use mini-dump.
⚠️ Profilers always-on in production.
⚠️ Forgetting dotnet-monitor auth — exposed control plane.
⚠️ Heap snapshot once — leaks visible only with diff.