Profiling Tools

Key Points

• Two profiler types: sampling (statistical; cheap; overview) and instrumentation (every call; expensive; precise). Sampling for production-safe overviews; instrumentation for narrow hotspots.
• Free tools: dotnet-trace (CPU sampling, ETW), PerfView (Windows; deepest), Visual Studio Diagnostic Tools.
• Commercial: JetBrains dotTrace (CPU; sampling+tracing), dotMemory (allocations + retention), Datadog Continuous Profiler.
• Where to start: dotnet-counters → identify symptom → dotnet-trace → identify hot stack → fix.
• Don't profile in Debug. Don't always-on profile production (continuous profiling at low rate is OK).

Concepts (deep dive)

Sampling vs instrumentation

• Sampling: pauses the process every N ms, records the call stack. Statistical: hot methods show up most often. Low overhead (~1–5%). What you want in production.
• Instrumentation: rewrites code to record every call/return. Precise but slow (10–100x). Useful in dev for finding all callers, exact counts.

dotnet-trace

dotnet-trace collect -p $(pgrep myapp) --duration 00:00:30 --profile cpu-sampling

Collects ETW traces. Output .nettrace opens in PerfView (Windows), VS, or Speedscope (web — drag-and-drop).

# convert to speedscope
dotnet-trace collect ... --format Speedscope

Visual Studio Diagnostic Tools

While debugging: built-in CPU usage, memory snapshots, allocation tracking. Easy entry point.

For prod-realistic analysis, attach to a release build with --launchProfile matching production.

PerfView (Windows)

The deepest free tool. Captures ETW with full stack info — GC, JIT, threadpool, lock contention, file I/O, all in one trace.

Common views: - CPU Stacks — sample-based; flame view. - GC Stats — pause times, gen counts. - Any Stacks (with start time) — what's allocating. - Thread Time — per-thread; reveals blocking, lock waits.

PerfView's UI is dense; learning curve is real. But once you know it, no other free tool comes close.

JetBrains dotTrace

Polished UI; sampling + tracing modes; timeline view; per-thread breakdown. Continuous profiling mode for prod-light overhead.

JetBrains dotMemory

Heap snapshots; object retention graphs; comparison between snapshots. The standard for memory leak hunts on Windows.

Datadog Continuous Profiler / Pyroscope

Always-on, low-overhead production profiling. Aggregated flame graphs across all instances. Worth it for production-grade systems.

dotnet-counters (refresher)

dotnet-counters monitor -p PID --counters System.Runtime,Microsoft.AspNetCore.Hosting,MyApp.Counters

Live counters (cheap). First step in any investigation — what's the symptom?

dotnet-stack (preview)

dotnet-stack report -p PID

Async-aware live stack. Shows pending tasks and what they're awaiting. Great for "process is hung" debugging.

Tools by question

CPU profile workflow

1. Symptom: high CPU, slow latency.
2. dotnet-counters: confirm CPU pegged.
3. dotnet-trace for 30s: collect.
4. Open in Speedscope: flame graph; find the hot stacks.
5. Identify hot methods consuming most CPU.
6. Fix and retest.

Memory profile workflow

1. Symptom: working set climbing.
2. dotnet-counters: working-set, allocation-rate.
3. dotnet-gcdump (T1).
4. Wait or repro.
5. dotnet-gcdump (T2).
6. Diff in VS: types/objects growing → roots → why retained.

Lock contention

dotnet-counters monitor -p PID --counters System.Runtime monitor-lock-contention-count

If high: profile with dotnet-trace (collect locks contention provider). Find hot lock; refactor.

Threadpool starvation

threadpool-queue-length rising. Cause is almost always sync-over-async. git grep -E '\.Result|\.Wait\(\)|GetAwaiter\(\).GetResult' to find culprits.

When profiling lies

• Sampling underrepresents very fast hot paths — they may not be sampled. Use instrumentation for those.
• Cold start vs steady state look different. Profile after warmup.
• JIT compilation dominates first calls. Wait for JIT to settle (or use ReadyToRun/AOT).

Continuous profiling in production

# Datadog auto-instrumentation for .NET
- DD_PROFILING_ENABLED=true
- DD_PROFILING_CPU_ENABLED=true
- DD_PROFILING_ALLOCATIONS_ENABLED=true

Low overhead (~1–2%). Aggregates flame graphs across instances. Identify systemic issues (e.g., one method consumes 8% of CPU across the fleet).

Profiling AOT'd apps

Native AOT removes the JIT — same profilers work. Symbolication may need extra config (dotnet publish with -p:DebugType=embedded).

"Don't optimize without profiling"

The mantra. The hot paths are rarely where you think. A profile takes 10 minutes; speculation can waste days.

Reading flame graphs

[Total: 100%]
└─ Main [80%]
   └─ ProcessRequest [70%]
      ├─ ParseJson [50%] ← biggest bar
      └─ Validate [10%]
   └─ Idle [10%]

Wider = more time. Look at the widest leaf — that's the hottest method.

Code: correct vs wrong

❌ Wrong: optimizing without profile

"This loop seems slow; let me parallelize it." → Half the time, makes it slower.

✅ Correct: profile first

profile shows 80% time in JsonSerializer → switch to source-gen → fix.

❌ Wrong: profiling Debug build

dotnet run    # Debug — JIT optimizations off; numbers wrong

✅ Correct: Release

dotnet run -c Release

Design patterns for this topic

Pattern 1 — "Counters → Trace → Fix"

• Intent: narrow from symptom to cause cheaply.

Pattern 2 — "Continuous profiling in prod"

• Intent: detect emerging hotspots.

Pattern 3 — "Snapshot diff for leaks"

• Intent: find growing types.

Pattern 4 — "Speedscope for sharing flame graphs"

• Intent: team-readable flame view.

Pattern 5 — "PerfView for everything Windows"

• Intent: single tool deep-dive.

Pros & cons / trade-offs

When to use / when to avoid

• Use counters first.
• Use trace for CPU/perf investigations.
• Use continuous profiling for production trends.
• Avoid instrumentation profiling on prod.
• Avoid profiling Debug builds.

Interview Q&A

Q1. Sampling vs instrumentation profilers? Sampling: statistical; low overhead. Instrumentation: every call; high overhead. Sampling for prod overviews.

Q2. First tool when investigating slow service? dotnet-counters — confirms symptom (CPU, GC, threadpool) cheaply.

Q3. Then? dotnet-trace for 30s; open flame graph in Speedscope; identify hottest stacks.

Q4. Memory leak workflow? dotnet-gcdump at T1 and T2; diff; find growing types; trace roots.

Q5. PerfView vs dotnet-trace? Same data (ETW). PerfView reads richer; dotnet-trace is x-plat collection.

Q6. Speedscope? Web flame graph viewer; .nettrace export support. Browser-based; shareable.

Q7. Continuous profiling? Always-on, low-overhead production profiling. Aggregates across instances. Datadog/Pyroscope.

Q8. Why not always-on instrumentation? 10–100x slowdown. Production-unsuitable.

Q9. dotnet-stack? Live async-aware stack. Pending tasks; what they're awaiting. Hung-process diagnosis.

Q10. Sampling underrepresents what? Very fast hot loops. Statistically may not be hit between samples.

Q11. Profile in Debug? Don't. Optimizations off; results don't reflect prod.

Q12. JIT impact on first profiles? First calls dominated by JIT. Profile after warmup or use ReadyToRun/AOT.

Gotchas / common mistakes

• ⚠️ Profile Debug — meaningless.
• ⚠️ Optimize without profile — wrong target.
• ⚠️ Sampling on rare events — misses.
• ⚠️ No baseline before optimizing — can't measure improvement.
• ⚠️ Always-on instrumentation in prod.

Further reading

• MS Learn: dotnet diagnostic tools
• Speedscope
• PerfView (microsoft/perfview repo, includes User's Guide)
• Datadog Continuous Profiler
• JetBrains dotTrace