Mutation Testing — Deep

💡 This deep-dive complements Snapshot & Mutation Testing (topic 4). Read that first for the basics. Here we drill into operators, scoring, equivalent mutants, performance, CI integration, and where mutation testing pays off vs where it's overkill.

Key Points

• Mutation testing rewrites your production code (e.g., flips < to <=, removes a statement) and re-runs the test suite. Each mutation that survives = a test gap.
• Stryker.NET is the canonical .NET tool. Free, open-source, mature, integrates with the Stryker dashboard.
• Mutation operators fall into families: arithmetic (+/-), conditional/boundary (</<=), logical (&&/||), statement deletion, return value, literal, increment.
• Mutation score is not 100%-or-bust. Aim 80%+ on critical modules; tolerate 60% on UI/glue. The signal is which mutants survived, not the headline number.
• Equivalent mutants are mutations that produce identical behavior — false positives. Real ones exist; document or ignore.
• Performance: N× slower than a single test run. Use incremental mode (mutations only on changed code) for PRs and full nightly runs.
• CI pattern: per-PR incremental Stryker (5-minute budget) + nightly full run + dashboard tracking trend.
• PBT vs mutation: different angles. PBT explores input space; mutation explores test-suite quality. Complementary.

Concepts (deep dive)

Mutation operators in detail

Stryker.NET applies a curated set of operators by default. Categories:

Arithmetic operators

total = price + tax;     // mutated → total = price - tax;
total = price * qty;     // mutated → total = price / qty;
total = a % b;           // mutated → total = a * b;

If your test only asserts total > 0 (instead of == expected), arithmetic mutations survive. Strict equality assertions kill them.

Conditional / boundary operators

if (x > 0)               → if (x >= 0)
if (x >= 0)              → if (x > 0)
if (x < n)               → if (x <= n)
if (x == y)              → if (x != y)

These find off-by-one bugs in your tests. If count >= 5 is correct and tests only verify count == 6, the boundary mutation count > 5 survives — your test never tried count == 5.

Logical and statement-level

public bool IsValid() => true;     // mutated → => false;
public int Count() => _items.Count; // mutated → => 0;
public string Name() => "Alice";   // mutated → => "";

These catch tests that don't actually inspect the return value (just call the method).

Literal mutation

Constants flip toward 0/1/-1/off-by-one neighbors; strings flip to "". Catches tests that don't pin specific values.

Stryker.NET in practice

dotnet tool install -g dotnet-stryker
cd src/MyApp.Tests
dotnet stryker

Stryker discovers your test project, builds the mutated copies, runs dotnet test per mutation, and writes an HTML report:

mutation-report.html
StrykerOutput/
  reports/
    mutation-report.html
    mutation-report.json

Open the HTML report — every mutation is a clickable line; survived mutations highlighted red.

Configuration: stryker-config.json

{
  "stryker-config": {
    "test-projects": ["MyApp.Tests/MyApp.Tests.csproj"],
    "project": "MyApp/MyApp.csproj",
    "mutation-level": "Standard",
    "thresholds": { "high": 80, "low": 60, "break": 50 },
    "since": { "target": "main", "ignore-changes-in": ["**/*.md", "**/*.png"] },
    "reporters": ["html", "dashboard", "progress"],
    "dashboard": {
      "project": "github.com/myorg/myapp",
      "version": "main"
    },
    "mutate": ["src/**/*.cs", "!src/MyApp/Migrations/**", "!src/MyApp/Program.cs"]
  }
}

Key settings: - mutation-level: Basic / Standard / Advanced / Complete — more mutations = slower run. - thresholds.break: build fails if score < this (e.g., 50%). - since.target: enables incremental — only mutate code changed vs that branch. - mutate: glob for what to mutate; exclude generated code, migrations, Program.cs. - reporters.dashboard: posts trend to Stryker dashboard for tracking.

Mutation score interpretation

Mutation score:
  Killed:    430 / 500   (86%)
  Survived:   55         (11%)
  Timeout:     5         (1%)
  No coverage: 10        (2%)

• Killed: at least one test failed under the mutation. Good.
• Survived: all tests passed under the mutation. Bad — gap.
• Timeout: mutation caused infinite loop / deadlock. Stryker times out. Counts as killed (test would have failed in real life).
• No coverage: no test ever executed the mutated line. Either dead code or missing tests.

The headline % is a starting point. The real value is opening the report and asking: "Why does this mutation survive?"

src/Domain/OrderValidator.cs
  Line 42: replaced `if (qty > 0)` with `if (qty >= 0)` — SURVIVED

This says: no test ever passed qty = 0. Either the validator should accept zero (tests are right; mutation is meaningful but acceptable) or it shouldn't (write a test for qty = 0).

"100% kill rate" is the wrong goal

Common misunderstanding. 100% kill rate doesn't mean perfect tests: - Equivalent mutants can never be killed. - Some mutants are in legitimate dead code (logging, defensive checks). - Some are in code that's costly to test (third-party SDK glue).

Senior take: target 80%+ for critical modules (domain logic, calculations, security checks); 60-70% acceptable for less critical code; investigate every survivor in core modules; tolerate or annotate survivors elsewhere.

Equivalent mutants — the false-positive problem

public int CountActive()
{
    var count = 0;
    foreach (var u in users)
        if (u.IsActive) count++;
    return count;
}

Mutation: count++ → ++count. Behaviorally identical in this context. Tests can never distinguish. The mutation survives forever.

Real example:

public bool IsExpired(DateTime now) => now > _expiry;   // mutated to >=

Strategies: accept (annotate why), exclude (Stryker filter), or refactor to remove ambiguity. Literature estimates equivalent-mutant rates of 5–20% — a few survivors is normal.

Performance — the elephant

Mutation testing is N × test-suite-runtime where N is the number of mutations. A 1000-test suite that runs in 10 seconds × 500 mutations = ~85 minutes per run.

Strategy 1 — Incremental mode

dotnet stryker --since main

Only mutate code that changed vs main. PR with 50 changed lines → maybe 5–10 mutations → 1–3 minute Stryker run. Fits in the PR pipeline.

{
  "since": { "target": "main", "ignore-changes-in": ["**/*.md", "tests/**"] }
}

Strategy 2 — Test filtering

For each mutation, run only tests covering the mutated line. Stryker does this automatically (it tracks coverage at mutate-time). Without this, mutation testing would be impractical.

Strategy 3 — Parallelism

dotnet stryker --concurrency 8

Mutations are independent. Spread across cores; near-linear speedup.

Strategy 4 — Mutation level

Set mutation-level to Basic for PR runs (fewer operators, faster), Complete for nightly.

CI integration patterns

Pattern A — Incremental on PR + full nightly

# pull-request.yml
- name: Stryker (incremental)
  run: |
    dotnet stryker --since origin/main \
      --reporter "html" --reporter "dashboard" \
      --break-at 70

# nightly.yml
- name: Stryker (full)
  run: |
    dotnet stryker \
      --reporter "html" --reporter "dashboard" \
      --break-at 75

PR run: 1–5 min, blocks merge if score on changed code drops below 70. Nightly run: 60+ min, posts to dashboard; alerts the team channel on regression.

Pattern B — Critical-modules-only

Restrict mutate glob to high-stakes modules (pricing, billing, auth, crypto). Run on every PR — small, fast, focused. For codebases where full PR runs are infeasible, fall back to nightly-only with weekly trend review.

Stryker dashboard

Free SaaS at dashboard.stryker-mutator.io. POST your mutation-report.json from CI; dashboard tracks score per branch over time.

{ "reporters": ["dashboard"],
  "dashboard": { "project": "github.com/org/repo", "version": "main", "module": "Domain" } }

Trend graph reveals slow regressions invisible in single-PR diffs.

When mutation testing pays off — and when it's overkill

Mutation testing vs property-based testing

Often confused; they attack different problems.

They compose: a property-based test on a fuzz-rich input space + mutation testing on the production code = strong safety net. PBT extends coverage; mutation extends assertion strength.

Reading the report well

[ Mutation report ]

OrderValidator.cs    [Score: 60%]   ⚠️
  ↳ 8 mutants
     • Line 42: > → >=    SURVIVED      (qty=0 not tested)
     • Line 43: ! → noop  SURVIVED      (negation never tested)
     • Line 51: + → -     KILLED
     • ...

EmailParser.cs       [Score: 95%]   ✅
PricingEngine.cs     [Score: 78%]   ⚠️
ProgramExtensions.cs [Score: 100%]  ✅ (trivial code)

Sort by score ascending. The bottom 5 files with mostly survivors are where the next sprint of test improvement lives.

How it works under the hood

Stryker.NET pipeline:

1. Parse the production project with Roslyn — build the syntax tree.
2. Identify mutation points — visit nodes matching operator patterns (BinaryExpressionSyntax for arithmetic, IfStatementSyntax for conditional, etc.).
3. Coverage scan — run the test suite once with instrumentation; record which tests cover which lines.
4. For each mutation:
5. Rewrite the syntax node (e.g., replace > with >=).
6. Compile the mutated assembly.
7. Run only the tests that cover the mutated line (per coverage scan).
8. Record Killed / Survived / Timeout.
9. Aggregate results, write HTML/JSON, post to dashboard.

Performance tricks: - In-memory compilation of mutated assemblies (no disk I/O per mutation). - Test filtering by coverage map. - Concurrency across mutations. - Incremental — skip mutations on unchanged code (using git diff).

The mutated assembly is loaded into a separate AppDomain / load context so test framework state doesn't leak between mutations.

Code: correct vs wrong

❌ Wrong: weak assertion lets mutations survive

[Fact]
public void Total_works()
{
    var t = OrderTotal.Of(new[] { new Line(2, 5m) });
    Assert.True(t > 0);   // weak: any positive value passes
}

Mutation * → + produces 2 + 5 = 7 instead of 2 * 5 = 10. Both > 0. Survives.

✅ Correct: exact assertion

Assert.Equal(10m, t);

❌ Wrong: testing only one boundary

[Fact]
public void Allows_qty_above_zero() =>
    Assert.True(_validator.Validate(new Order { Qty = 1 }));

Mutation > 0 → >= 0 survives — qty = 0 untested.

✅ Correct: test the boundary explicitly

[Theory]
[InlineData(-1, false)]
[InlineData(0, false)]
[InlineData(1, true)]
[InlineData(100, true)]
public void Validate_qty(int qty, bool ok) =>
    Assert.Equal(ok, _validator.Validate(new Order { Qty = qty }));

❌ Wrong: ignoring the survivors

Stryker report: 65% mutation score, 50 survivors.
> Senior dev: "We're at 65%, ship it."

✅ Correct: open the report, fix the top 5

Investigate survivors in critical modules; tolerate or annotate elsewhere.

❌ Wrong: full mutation on every PR

- run: dotnet stryker        # 60 minutes per PR

✅ Correct: incremental on PR, full nightly

- run: dotnet stryker --since origin/main --break-at 70

Design patterns for this topic

Pattern 1 — "Incremental on PR, full nightly"

• Intent: fast feedback per PR; trend tracking via dashboard.

Pattern 2 — "Module-tiered thresholds"

• Intent: strict bar on critical modules; relaxed bar on glue.

Pattern 3 — "Investigate top-5 survivors per sprint"

• Intent: continuous test-quality improvement without a 100% goal.

Pattern 4 — "Exclude generated code"

• Intent: migrations, DTOs, Program.cs stay out of mutation scope.

Pattern 5 — "Annotate equivalent mutants"

• Intent: known-equivalent survivors are tracked, not investigated repeatedly.

Pattern 6 — "Boundary-driven test data"

• Intent: every Theory includes the boundary (zero, max, off-by-one).

Pros & cons / trade-offs

When to use / when to avoid

• ✅ Use for high-stakes business logic (pricing, billing, security, planning).
• ✅ Use for libraries / SDKs where downstream depends on your behavior.
• ✅ Use in nightly CI for trend tracking.
• ✅ Use when integrating mutation as a cultural ratchet — improving over time.
• ❌ Avoid as a hard PR gate at high thresholds — friction kills adoption.
• ❌ Avoid for boilerplate / glue code — low ROI.
• ❌ Avoid without dedicated time to investigate survivors — generates noise.
• ❌ Avoid mutating generated code (migrations, DTOs).

Interview Q&A

Q1. What's the difference between line coverage and mutation score? Line coverage measures whether a line executed. Mutation score measures whether changing the line's behavior is detected by a test. A line can be 100% covered with no real assertion — mutation testing exposes that.

Q2. Name five Stryker mutation operators. Arithmetic (+/-), boundary (</<=), logical (&&/||), statement deletion, return value replacement.

Q3. Why isn't 100% mutation score the goal? Equivalent mutants exist; some lines are dead-code-ish (defensive checks, logging); pursuing 100% drives tests toward implementation details. Aim 80%+ on critical, accept lower elsewhere.

Q4. What's an equivalent mutant? A mutation that produces behaviorally identical code — no test can distinguish. False-positive survivor. Annotate or accept.

Q5. How do you make mutation testing fast enough for CI? Incremental mode (--since main), test filtering by coverage, parallelism, lower mutation level on PR, full level nightly.

Q6. Per-PR run pattern? Incremental Stryker on changed code, 1–5 min budget, threshold gate. Full run on a nightly schedule with dashboard trend.

Q7. How does mutation differ from property-based testing? PBT explores the input space — finds bugs by varying inputs. Mutation explores the test-quality space — finds gaps by varying production code. Complementary.

Q8. Why exclude generated code? Migrations, DTOs, Program.cs are mostly mechanical or imperative bootstrap. Mutating them generates noise without surfacing real test gaps.

Q9. What does "Timeout" mean in a Stryker report? The mutation caused an infinite loop or deadlock; Stryker timed out. Counts as killed — a real test would have hung too.

Q10. How to handle an exploding survivor list on first run? Sort by file score; investigate top 5 critical modules; tolerate the rest. Set incremental thresholds and ratchet over sprints.

Q11. What's the role of the Stryker dashboard? Free SaaS that tracks mutation score per branch / module over time. Reveals slow regressions invisible in single-run reports.

Q12. How do snapshot tests interact with mutation testing? Snapshots cover lots of code but may not distinguish behaviors. Mutation testing reveals if the snapshot data discriminates — survivors mean the snapshot was generated under conditions that mask the mutation.

Gotchas / common mistakes

• ⚠️ Hard PR gate at 90% → developers game it with weak tests on simple paths.
• ⚠️ Mutating generated code → noise floods report.
• ⚠️ Ignoring "Timeout" mutants → they often signal real performance problems.
• ⚠️ Treating equivalent mutants as bugs → wasted investigation time.
• ⚠️ No incremental mode → CI run blows up to hours.
• ⚠️ Excluding too aggressively → score becomes meaningless.
• ⚠️ Single global threshold → underprotects critical modules / overburdens glue.
• ⚠️ Reading only the headline % → real value is the survivor list.
• ⚠️ Running full mutation on every commit → developer flow death.
• ⚠️ Forgetting the dashboard → no trend visibility; regressions normalize.

Further reading

• Stryker.NET docs
• Stryker dashboard
• [Mutation Testing chapter — Working Effectively with Unit Tests, Jay Fields]
• Andrew Lock: Stryker.NET in CI
• Pitest (Java mutation tool — historical context)
• Snapshot & Mutation Testing (basics)
• Property-Based Testing with FsCheck (complementary)