Skip to content

Snapshot & Mutation Testing

Key Points

  • Snapshot testing captures the output of a function and compares against a stored "golden" file. New behavior diff = manual review/approval. Library: Verify (Simon Cropp).
  • Mutation testing intentionally changes (mutates) production code, runs your tests, and checks if any test catches the mutation. Reveals weak tests. Library: Stryker.NET.
  • Snapshot is for complex output assertions (large DTOs, JSON shapes, generated SQL, OpenAPI specs). Mutation is for measuring test quality beyond line coverage.
  • Both tend to be high-value, low-frequency tools — run snapshot tests in CI, run mutation testing periodically (slow).

Concepts (deep dive)

Verify (snapshot testing)

[UsesVerify]   // attribute on class
public class OrderTests
{
    [Fact]
    public Task Place_creates_order()
    {
        var order = OrderFactory.Place(/* args */);
        return Verify(order);
    }
}

First run: writes OrderTests.Place_creates_order.received.txt with the serialized output. Verify expects you to rename to .verified.txt (manual approval). Subsequent runs compare; differences cause failure.

Verify's strengths: - Diff-friendly snapshots (deterministic JSON serialization). - IDE integration: Visual Studio / Rider plugins offer "approve" buttons on diff. - Many input types: objects (auto-serialized), HTTP responses, EF models, generated SQL, OpenAPI specs, even images.

return Verify(order)
    .ScrubMembers("Id", "Timestamp")    // strip non-deterministic fields
    .UseDirectory("Snapshots")
    .UseFileName($"order-{order.Status}");

Snapshot testing patterns

Good fits: - Complex DTO comparisons (avoid 30 Assert.Equal calls). - Generated code (compile output, OpenAPI, EF SQL). - HTTP response shape regression. - Event payloads. - Razor / Blazor markup.

Bad fits: - Trivial assertions (Assert.Equal(5, sum)). - Highly dynamic output (timestamps, IDs). - Tests where the snapshot becomes the spec — the test no longer expresses intent; reviewers can't tell what should be true.

Determinism

Tests must be deterministic. Mock TimeProvider, control random seeds, scrub volatile fields:

return Verify(response)
    .ScrubMembers("CorrelationId", "ServerTime")
    .ScrubInlineGuids()
    .ScrubLinesContaining("server-name");

Verify ecosystem

  • Verify.Xunit / Verify.NUnit — test framework integration.
  • Verify.EntityFramework — verify generated SQL.
  • Verify.AspNetCore — verify HTTP responses.
  • Verify.NewtonsoftJson, Verify.SystemTextJson — JSON serialization options.
  • Verify.SourceGenerators — verify generator output.

Mutation testing with Stryker

dotnet tool install -g dotnet-stryker
cd src/MyApp.Tests
dotnet stryker

Stryker mutates production code (changes + to -, < to <=, removes statements, replaces literals) and runs your tests. Mutation score = percent of mutations killed by tests.

Mutation operators (examples): - + → - - >= → > - true → false - Method call removed. - Boolean negation.

If a mutation survives (tests pass with the mutated code), your tests don't cover that behavior — even if line coverage says they do.

Sample report

Mutation testing report
- Killed:    280 / 310    (90.3%)
- Survived:   25
- Timeout:     5
- No coverage: 0

Survived mutants are clickable in the HTML report — points to the exact line + the mutation that wasn't caught.

Cost

Mutation testing is slow — runs the full test suite once per mutation. A 1000-test suite × 500 mutations = 500K test executions. Strategies:

  • Filter to changed files in CI (--since=main).
  • Run nightly, not per-PR.
  • Use the dashboard for trends rather than gating.

Reading mutation results

[MutationScore: 65%]   src/Domain/Order.cs
  Line 42: replaced `if (qty > 0)` with `if (qty >= 0)` — SURVIVED
  Line 51: replaced `total += line.Amount` with `total -= line.Amount` — KILLED

Each survived mutation is a missing test. Add a test that would have failed under the mutation.

Equivalent mutants

Some mutations don't change behavior (e.g., if (x > 0) → if (x >= 0) when x is always strictly positive). These survive but aren't real gaps. Stryker's --ignore- flags help.

Combining

Snapshot tests + mutation testing = strong coverage measure. Snapshots catch behavioral changes; mutations measure how strict the tests really are.

[Property]
public bool Reverse_twice_is_identity(string s) =>
    Reverse(Reverse(s)) == s;

FsCheck / Hedgehog generate random inputs and shrink failures. Catches edge cases human-written tests miss. Common in functional/domain heavy code.

Code: correct vs wrong

❌ Wrong: snapshot full of timestamps

return Verify(response);   // includes RequestId, Timestamp, NowDate

Every run regenerates → no real signal.

✅ Correct: scrub volatile fields

return Verify(response).ScrubMembers("RequestId", "Timestamp");

❌ Wrong: no manual approval for snapshot changes

// Auto-approving snapshots in CI defeats the purpose

✅ Correct: snapshots committed only after human review

# Locally:
mv MyTest.received.json MyTest.verified.json    # after eyeballing diff
git add MyTest.verified.json

Design patterns for this topic

Pattern 1 — "Snapshot for complex outputs only"

  • Intent: simple assertions stay simple.

Pattern 2 — "Scrub volatile fields"

  • Intent: deterministic snapshots.

Pattern 3 — "Mutation testing as quality gauge, not gate"

  • Intent: trend score; investigate dips.

Pattern 4 — "Property-based for invariants"

  • Intent: roundtrip, idempotence, ordering.

Pattern 5 — "Snapshot generated artifacts"

  • Intent: OpenAPI, EF SQL, source generators.

Pros & cons / trade-offs

Tool Pros Cons
Snapshot Captures complex shapes Can drift; needs review
Mutation Reveals weak tests Very slow
Property-based Edge case discovery Learning curve

When to use / when to avoid

  • Snapshot: large DTOs, generated code, HTTP responses.
  • Mutation: nightly quality dashboard.
  • Property-based: domain logic with invariants.
  • Avoid: snapshot for trivial values; mutation as PR gate.

Interview Q&A

Q1. What's snapshot testing? Compare current output to a stored "golden" file. Diff = manual review.

Q2. When is snapshot a good fit? Complex shapes — DTOs, HTTP responses, generated code. Avoid for trivial values.

Q3. How handle non-deterministic fields? Scrub them: Verify(x).ScrubMembers("Timestamp"). Or mock TimeProvider.

Q4. What's mutation testing? Modifies production code; runs tests; if no test fails, mutation "survived" — your tests didn't cover that behavior.

Q5. Mutation score vs line coverage? Coverage = lines executed. Mutation score = behaviors actually verified. A line covered with weak assertions has low mutation score.

Q6. Why mutation testing is slow? Full test run per mutation. 500 mutations × 1000 tests = half a million executions.

Q7. Run mutation testing in CI? Usually nightly or weekly, not per-PR. Run on changed files in PRs.

Q8. What's an equivalent mutant? Mutation that doesn't change behavior (always-equivalent code). Survives but isn't a real test gap.

Q9. Property-based testing? Random inputs + invariants. Library shrinks failing inputs to minimal repro. FsCheck / Hedgehog.

Q10. Verify vs ApprovalTests? Verify is the modern .NET-focused successor; better serialization, scrubbing, IDE integration.

Gotchas / common mistakes

  • ⚠️ Auto-approving snapshots — defeats purpose.
  • ⚠️ Massive snapshot files — review fatigue; reduce scope.
  • ⚠️ Non-deterministic snapshots — flaky tests.
  • ⚠️ Mutation testing in PR gate — slow, frustrating.
  • ⚠️ Ignoring survived mutants — accumulating tech debt.

Further reading