Regex & SearchValues

Key Points

• [GeneratedRegex] (.NET 7+) — source-generated regex. Compile-time cost; zero runtime regex compilation; AOT-safe.
• Regex with RegexOptions.Compiled — runtime IL emit; faster than interpreted but doesn't work in AOT.
• SearchValues<T> (.NET 8+) — pre-built SIMD-optimized character/byte set for IndexOfAny / ContainsAny. Massively faster than passing a char[].
• CompositeFormat (.NET 8+) — pre-parses a format string for repeated string.Format calls, avoiding per-call parse.
• For performance-critical paths, source-generated regex + SearchValues are usually orders-of-magnitude faster than naive equivalents.
• Don't use regex when string-method equivalents exist (StartsWith, IndexOf, Split) — they're faster.

Concepts (deep dive)

[GeneratedRegex] — the modern way

public partial class IpHelper
{
    [GeneratedRegex(@"^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$",
                    RegexOptions.IgnoreCase | RegexOptions.CultureInvariant)]
    private static partial Regex IpRegex();

    public static bool IsValidIp(string s) => IpRegex().IsMatch(s);
}

The source generator produces a fully optimized Regex subclass at compile time. Comparison:

For any regex used more than once, prefer source-gen.

C# 13 added support for [GeneratedRegex] on partial properties (not just methods).

RegexOptions worth knowing

💡 Always use RegexOptions.CultureInvariant unless you specifically want culture-aware case folding. The default behavior depends on the current culture and can match unexpectedly.

SearchValues<T> — the SIMD search primitive

private static readonly SearchValues<char> _vowels = SearchValues.Create("aeiouAEIOU");

public static int FirstVowel(ReadOnlySpan<char> input)
    => input.IndexOfAny(_vowels);

For "find any of these characters in this span", SearchValues<char> is dramatically faster than IndexOfAny(char[]). The implementation uses SIMD + bitmap lookups specialized to your character set.

Sample timings (rough orders of magnitude):

Cache SearchValues in a static field. Construction is expensive (precomputes the bitmap).

SearchValues<byte> works for byte spans too — useful for binary protocols.

CompositeFormat

private static readonly CompositeFormat _greeting =
    CompositeFormat.Parse("Hello, {0}! You have {1} messages.");

public static string Format(string name, int count)
    => string.Format(CultureInfo.InvariantCulture, _greeting, name, count);

Pre-parses the format string. Subsequent string.Format(_greeting, ...) calls skip the parse step. Useful for "the same format used in a hot loop".

When to use regex vs string methods

// ❌ regex overkill
bool startsWithApi = Regex.IsMatch(url, @"^/api");

// ✅ direct
bool startsWithApi = url.StartsWith("/api", StringComparison.Ordinal);

// ❌ regex overkill
string[] parts = Regex.Split(csv, ",");

// ✅ direct
string[] parts = csv.Split(',');

// ✅ regex appropriate (true pattern)
Regex emailRx = new(@"^[^@\s]+@[^@\s]+\.[^@\s]+$");

Rule of thumb: if the operation can be expressed with StartsWith, Contains, IndexOf, Split on a literal — use those. They're faster than regex even for simple patterns.

Regex catastrophic backtracking

// ❌ pathologically slow on inputs like "aaaaaaaa..."b
new Regex(@"^(a+)+b$").IsMatch(input);

Nested repetition ((a+)+) over the same character class can backtrack exponentially. Symptom: regex appears to hang on certain inputs.

Mitigations: - Atomic groups (?>a+) — disable backtracking for that group. - Possessive quantifiers a++ — same idea, terser (regex engines that support it). - Avoid nested repetition with overlapping character classes. - Set a Regex.MatchTimeout to bound the worst case:

new Regex(pattern, RegexOptions.Compiled, TimeSpan.FromMilliseconds(100));

⚠️ User-supplied regex patterns are an attack surface: ReDoS (regex denial of service). Always set MatchTimeout or run in a sandboxed worker.

Compiled-vs-interpreted-vs-source-gen choice tree

   Use regex repeatedly in hot paths?
   │
   ├─ Yes ─► Pattern known at compile time?
   │        │
   │        ├─ Yes ─► [GeneratedRegex]   (best)
   │        └─ No  ─► RegexOptions.Compiled (if not AOT)
   │
   └─ No, used once or ad hoc ──► new Regex(pattern) (interpreted)

Modern split / replace / matches

ReadOnlySpan<char> input = "hello, world";

// .NET 7+: enumerator-based, allocation-light
foreach (var range in input.Split(','))
    Process(input[range]);

// Regex on ReadOnlySpan<char> (modern)
foreach (var match in Regex.EnumerateMatches(input, "\\w+"))
    Process(input.Slice(match.Index, match.Length));

Regex.EnumerateMatches is allocation-free (returns ValueMatch enumerator). Use over Regex.Matches in hot paths.

How it works under the hood

[GeneratedRegex] source generator parses the regex pattern at compile time, builds an optimized FA (finite automaton), and emits C# implementing it. The generated class is a Regex subclass that overrides the matching machinery — no runtime parsing, no IL emit.

SearchValues<T> uses different strategies based on the set size: - 1-3 values: fast direct comparison. - Larger sets of ASCII: bitmap lookup with SIMD. - Larger sets of arbitrary chars: vectorized search with hashing.

The implementation lives in System.Buffers. Source: runtime/src/libraries/System.Private.CoreLib/src/System/SearchValues.

Regex.Compiled emits IL via Reflection.Emit — generates a JIT-compiled subclass at runtime. Faster than interpreted but won't work in AOT (no Reflection.Emit).

CompositeFormat parses the format string into a list of "format hole" structs at construction. string.Format(compositeFormat, args) walks the list directly.

Code: correct vs wrong

❌ Wrong: regex constructed in a hot path

public bool IsEmail(string s)
    => Regex.IsMatch(s, @"^[^@]+@[^@]+\.[^@]+$");   // parses pattern every call

✅ Correct: source-gen

public partial class Validator
{
    [GeneratedRegex(@"^[^@]+@[^@]+\.[^@]+$")]
    private static partial Regex EmailRx();

    public bool IsEmail(string s) => EmailRx().IsMatch(s);
}

❌ Wrong: IndexOfAny(new char[]{...}) per call

public int FindSpecial(ReadOnlySpan<char> s)
    => s.IndexOfAny(new char[]{ '<', '>', '&', '"' });   // allocates

✅ Correct: cached SearchValues

private static readonly SearchValues<char> _special = SearchValues.Create("<>&\"");
public int FindSpecial(ReadOnlySpan<char> s) => s.IndexOfAny(_special);

❌ Wrong: regex without timeout, user-supplied pattern

public bool Match(string pattern, string input)
    => Regex.IsMatch(input, pattern);   // ReDoS possible

✅ Correct: timeout

public bool Match(string pattern, string input)
{
    var rx = new Regex(pattern, RegexOptions.Compiled, TimeSpan.FromMilliseconds(50));
    try { return rx.IsMatch(input); }
    catch (RegexMatchTimeoutException) { return false; }
}

❌ Wrong: regex for simple operations

if (Regex.IsMatch(url, @"^/api/")) ...   // unnecessary

✅ Correct

if (url.StartsWith("/api/", StringComparison.Ordinal)) ...

Design patterns for this topic

Pattern 1 — "[GeneratedRegex] for every fixed pattern"

• Intent: AOT-safe + fastest.
• Code sketch: see "correct vs wrong" #1.

Pattern 2 — "SearchValues<char> for character set lookups"

• Intent: SIMD-fast IndexOfAny.
• Code sketch: see "correct vs wrong" #2.

Pattern 3 — "CompositeFormat for hot-path formatting"

• Intent: pre-parse format strings.

Pattern 4 — "MatchTimeout for user-supplied patterns"

• Intent: ReDoS defense.

Pattern 5 — "Prefer string methods to regex when possible"

• Intent: less code, faster execution.

Pros & cons / trade-offs

When to use / when to avoid

• Use [GeneratedRegex] for any pattern used more than once.
• Use SearchValues<T> for any IndexOfAny/ContainsAny over a fixed character set.
• Use CompositeFormat for hot-path string.Format with fixed format string.
• Avoid regex for operations that are simple string matches.
• Avoid Regex.Compiled in AOT — silently fails.
• Avoid user-supplied patterns without timeout.

Interview Q&A

Q1. What's [GeneratedRegex]? Source generator that produces an optimized Regex subclass at compile time. No runtime parsing or IL emit. AOT-safe. Best perf for any pattern used more than once.

Q2. Why is RegexOptions.Compiled not AOT-safe? It uses Reflection.Emit to generate IL at runtime. AOT has no JIT and forbids Reflection.Emit.

Q3. What's SearchValues<T> and why is it fast? A pre-built character/byte set for IndexOfAny/ContainsAny. Implementation uses SIMD + bitmap lookups specialized to the set size. Construction is one-time cost; lookup is much faster than array-based IndexOfAny.

Q4. What's catastrophic backtracking? Pathological regex performance: nested repetition over overlapping character classes can backtrack exponentially. ^(a+)+b$ on "aaaaaaaa..." (no b) hangs. Mitigation: avoid nested repetition, use atomic groups, set MatchTimeout.

Q5. When should you NOT use regex? When string methods suffice: StartsWith, EndsWith, Contains, IndexOf, Split on literals. They're faster and clearer.

Q6. What's CompositeFormat? A pre-parsed format string. Avoids re-parsing on every string.Format call. Worth it for hot-path formatting with a fixed format string.

Q7. Why is RegexOptions.CultureInvariant important? Default regex case-insensitive matching uses the current culture, which can match unexpectedly across cultures. Always pair IgnoreCase with CultureInvariant for predictable behavior.

Q8. What's Regex.EnumerateMatches? Allocation-free regex enumerator over ReadOnlySpan<char>. Returns ValueMatch (struct) per match. Use in hot paths instead of Regex.Matches (which allocates).

Q9. How does the [GeneratedRegex] source generator handle case folding? Computes the case table at compile time. The generator emits direct character comparisons. Note: case tables can differ slightly between .NET versions; pin language version if you need exact reproducibility.

Q10. Can you use SearchValues<T> for byte-level binary protocols? Yes — SearchValues<byte> searches in ReadOnlySpan<byte>. Useful for protocol parsers (HTTP delimiters, JSON tokens, etc.).

Q11. What's a ReDoS attack? Regex denial of service: malicious user-supplied input crafted to trigger catastrophic backtracking. Sets a single CPU core to 100% on what looks like an innocent string. Defense: MatchTimeout + careful pattern design.

Q12. How would you regex-match in a streaming way over a Stream? Read into a buffer (ReadOnlySpan<char> or ReadOnlySpan<byte>), match Regex.EnumerateMatches, advance the buffer past the consumed content, refill. The trick is handling matches spanning buffer boundaries — usually overlap by a fixed amount equal to your max-match-length.

Gotchas / common mistakes

• ⚠️ Constructing Regex per call — pattern parsing every time.
• ⚠️ SearchValues not cached — defeats the purpose.
• ⚠️ No MatchTimeout on user patterns — ReDoS risk.
• ⚠️ Forgetting CultureInvariant — case-fold surprises.
• ⚠️ Regex for simple StartsWith — slower than the direct method.
• ⚠️ Regex.Compiled in AOT — runtime exception.
• ⚠️ Catastrophic backtracking patterns — (a+)+, (a|a)+, etc.

Further reading

• Regex source generators
• SearchValues<T>
• CompositeFormat
• Regex performance
• Regex backtracking & ReDoS