BenchmarkDotNet

Key Points

• BenchmarkDotNet (BDN) is the .NET microbenchmark library. Handles JIT warmup, GC stabilization, statistical noise, multi-target framework runs, allocation tracking.
• Use it for code-level perf comparisons (algorithm A vs B, allocation tuning, pooling impact). NOT a load tester for HTTP services — use bombardier/k6 for those.
• Setup: separate Console project; reference your library; BenchmarkRunner.Run<MyBenchmarks>(). Run in Release.
• [MemoryDiagnoser] for allocation tracking. [Params] for parameter sweeps. [Baseline] for relative comparison.
• Read the output: Mean, Error, StdDev, Median, Allocated. Check Ratio vs baseline. Statistical significance matters.

Concepts (deep dive)

Setup

// Separate console project: MyApp.Benchmarks.csproj
// <PackageReference Include="BenchmarkDotNet" Version="0.14.*" />
// <PropertyGroup><Configuration>Release</Configuration></PropertyGroup>

using BenchmarkDotNet.Running;

public class Program { public static void Main() => BenchmarkRunner.Run<StringBenchmarks>(); }

[MemoryDiagnoser]
public class StringBenchmarks
{
    private string[] _words = Enumerable.Range(0, 100).Select(i => $"word{i}").ToArray();

    [Benchmark(Baseline = true)]
    public string Concat()
    {
        var s = "";
        foreach (var w in _words) s += w;
        return s;
    }

    [Benchmark]
    public string StringBuilder()
    {
        var sb = new System.Text.StringBuilder();
        foreach (var w in _words) sb.Append(w);
        return sb.ToString();
    }

    [Benchmark]
    public string StringJoin() => string.Join("", _words);
}

Run: dotnet run -c Release

Output

| Method        | Mean       | Error    | StdDev   | Ratio | Allocated |
|-------------- |-----------:|---------:|---------:|------:|----------:|
| Concat        | 28.412 us  | 0.241 us | 0.226 us |  1.00 |   42.4 KB |
| StringBuilder |  1.234 us  | 0.012 us | 0.011 us |  0.04 |    1.2 KB |
| StringJoin    |  0.867 us  | 0.009 us | 0.008 us |  0.03 |    0.9 KB |

Ratio makes intent obvious. Allocated shows the GC pressure difference.

Critical: Release configuration

BDN refuses to run Debug builds (warning printed). Always Release. Disable inlining/optimizations in your test code is wrong — measure what runs in production.

[Params] for parameter sweeps

[Params(10, 100, 1000, 10000)]
public int N { get; set; }

[Benchmark]
public int Sum()
{
    int s = 0;
    for (int i = 0; i < N; i++) s += i;
    return s;
}

Each Params value runs separately — see how performance scales with input size.

Multi-runtime targeting

[SimpleJob(RuntimeMoniker.Net80)]
[SimpleJob(RuntimeMoniker.Net90)]
public class MultiRuntimeBenchmarks { /* ... */ }

Compare across .NET versions. Useful for migrations.

Diagnosers

[MemoryDiagnoser]                // allocation tracking
[ThreadingDiagnoser]             // lock contention
[HardwareCounters(HardwareCounter.BranchMispredictions, HardwareCounter.CacheMisses)]
[NativeMemoryProfiler]

[GlobalSetup] / [IterationSetup]

[GlobalSetup]
public void Setup()
{
    _data = LoadOnceData();
}

[IterationSetup]
public void Each()
{
    _state = new();    // reset before each iteration
}

Don't put expensive setup in benchmark methods — measures setup, not the operation.

What NOT to benchmark

• Trivial expressions that the JIT folds — [Benchmark] public int Add() => 1 + 1; returns nanoseconds of nothing.
• Cold-cache scenarios without IterationSetup — first run pays JIT.
• HTTP requests — use bombardier/k6/JMeter.

Common BDN traps

Dead-code elimination

[Benchmark]
public void NoOp() { var x = ComputeStuff(); }   // x unused → JIT may eliminate

Always return the value:

[Benchmark]
public int NoOp() => ComputeStuff();

BDN consumes the return → forces evaluation.

Loop overhead dominates

[Benchmark]
public void Loop()
{
    for (int i = 0; i < 100; i++) _list.Add(i);
}

Includes loop iteration cost in the measurement. Use [Benchmark(OperationsPerInvoke = 100)] if amortizing.

Variability

Mean: 1.234 us  Error: 0.011 us  StdDev: 0.012 us

Look at error/StdDev — small relative to mean = stable. Large = noise (other processes, thermal throttling).

[ParamsSource] for complex inputs

public IEnumerable<int[]> ArrayCases() =>
    new[] { new[] { 1 }, new int[100], new int[10_000] };

[ParamsSource(nameof(ArrayCases))]
public int[] Data { get; set; }

Statistical analysis

BDN runs many iterations, computes mean/StdDev/median, performs outlier detection. Default settings produce reliable numbers.

[ShortRunJob] / [LongRunJob] adjust the time/iteration count.

Disassembly

[DisassemblyDiagnoser]

Outputs the assembly the JIT produced for your benchmark. Killer feature for low-level perf work.

Comparing against baseline

[Benchmark(Baseline = true)]
public int Reference() { /* current */ }

[Benchmark]
public int Optimized() { /* candidate */ }

Ratio column shows multiplier — 0.5 = twice as fast, 2.0 = twice as slow.

Jobs and runtime configuration

[SimpleJob(RuntimeMoniker.Net90, baseline: true)]
[SimpleJob(RuntimeMoniker.NativeAot80)]
public class MyBenchmarks { }

Compare AOT vs JIT, server vs workstation GC, etc.

Allocation analysis

| Method | Allocated | Gen 0 | Gen 1 | Gen 2 |
|--------|----------:|------:|------:|------:|
| A      |   1.0 KB |   2.0 |   0.0 |   0.0 |
| B      |   0.0 B  |   0.0 |   0.0 |   0.0 |

Zero allocations is achievable for hot paths with Span<T>/ArrayPool/struct-based code.

Benchmarking async

[Benchmark]
public async Task<int> AsyncMethodAsync() => await SomeAsync();

BDN handles async — measures task completion time. But: ValueTask vs Task perf differences matter.

Comparing implementations vs measuring "fast enough"

BDN tells you A is N% faster than B. It does NOT tell you "fast enough for the use case". For end-to-end perf, profile your actual app under realistic load.

Code: correct vs wrong

❌ Wrong: benchmarking in Debug

dotnet run    # implicit Debug

BDN warns; numbers are meaningless.

✅ Correct: Release

dotnet run -c Release

❌ Wrong: setup inside benchmark

[Benchmark]
public int M()
{
    var data = ExpensiveLoad();   // measured
    return Process(data);
}

✅ Correct: setup attribute

[GlobalSetup] public void Setup() => _data = ExpensiveLoad();
[Benchmark] public int M() => Process(_data);

❌ Wrong: discarded return

[Benchmark] public void M() { var _ = Compute(); }

JIT may dead-code eliminate.

✅ Correct: return

[Benchmark] public int M() => Compute();

Design patterns for this topic

Pattern 1 — "Baseline + candidates"

• Intent: ratio shows actual gain.

Pattern 2 — "GlobalSetup for expensive prep"

• Intent: isolate measurement to the operation.

Pattern 3 — "MemoryDiagnoser always"

• Intent: allocations matter as much as time.

Pattern 4 — "Params sweep for scaling"

• Intent: performance vs input size curve.

Pattern 5 — "Disassembly for tight loops"

• Intent: JIT analysis at the asm level.

Pros & cons / trade-offs

When to use / when to avoid

• Use for algorithm comparison, allocation tuning, library impact.
• Avoid for HTTP service load testing.
• Avoid for pure-curiosity benchmarks unrelated to actual hot paths.

Interview Q&A

Q1. Why BenchmarkDotNet over Stopwatch? Handles JIT warmup, GC stabilization, statistical analysis, outlier detection. Stopwatch alone gives unreliable numbers.

Q2. What's [MemoryDiagnoser]? Tracks GC allocations per benchmark. Critical complement to time.

Q3. Why must benchmarks return values? Otherwise JIT can dead-code eliminate the operation.

Q4. [GlobalSetup] vs [IterationSetup]? Global: once per benchmark. Iteration: every iteration. Use Global for expensive prep; Iteration for state reset.

Q5. What's the Ratio column? Time relative to [Baseline]. 0.5 = 2x faster; 2.0 = 2x slower.

Q6. How handle async benchmarks? Return Task<T>; BDN awaits. Measures total async completion.

Q7. Multi-runtime comparison? [SimpleJob(RuntimeMoniker.Net80)] etc. Same code; multiple runtimes.

Q8. What's [Params]? Generates one benchmark per value. Shows scaling.

Q9. When NOT to use BDN? HTTP/load tests, end-to-end perf, real-world scenarios. Use bombardier/k6 there.

Q10. Common micro-benchmark trap? Constant folding by the JIT — must use varying inputs and return the result.

Q11. Why Release config required? Debug disables optimizations. Numbers don't represent production behavior.

Q12. Disassembly diagnoser? Dumps JIT-produced assembly. Lets you see actual codegen for tight loops.

Gotchas / common mistakes

• ⚠️ Debug build — invalid numbers.
• ⚠️ Setup inside benchmark — measures setup.
• ⚠️ Discarded result — DCE.
• ⚠️ Tiny operations — loop overhead dominates.
• ⚠️ Reading mean only — check StdDev.
• ⚠️ Comparing across machines without same hardware.

Further reading

• BenchmarkDotNet docs
• Stephen Toub: .NET perf series
• Adam Sitnik: BDN talks
• MS Learn: Performance