Hardware Intrinsics & SIMD

Key Points

• Vector<T> — width-agnostic SIMD (auto-picks best ISA at runtime). Vector128/256/512<T> — explicit width.
• Intrinsics in System.Runtime.Intrinsics.X86 (Sse, Sse2, Avx, Avx2, Avx512F) and System.Runtime.Intrinsics.Arm (AdvSimd).
• Auto-vectorization in modern JIT (.NET 8+) handles many simple loops over Span<T>. Try that first; reach for intrinsics when profiling shows gain.
• Common wins: Sum, Min/Max, IndexOf, byte search, format/parse, base64, hashing — anywhere a tight loop processes blittable data.
• SearchValues<T> (.NET 8+) packages SIMD-accelerated search behind a safe, zero-ceremony API — usually the right entry point.

Concepts (deep dive)

What SIMD does

Scalar:  a[0]+b[0]   (1 op per cycle)
SIMD256: [a[0]…a[7]] + [b[0]…b[7]]   (8 int adds in 1 op)
SIMD512: [a[0]…a[15]] + [b[0]…b[15]]  (16 int adds)

Speedup is roughly the vector width / element size, minus loop overhead and tail handling.

Width-agnostic Vector<T>

using System.Numerics;

public static int Sum(ReadOnlySpan<int> data)
{
    int total = 0;
    int i = 0;
    int stride = Vector<int>.Count;     // 4, 8, 16 depending on hardware
    var acc = Vector<int>.Zero;
    for (; i <= data.Length - stride; i += stride)
        acc += new Vector<int>(data.Slice(i, stride));
    total = Vector.Sum(acc);
    for (; i < data.Length; i++) total += data[i];
    return total;
}

The JIT picks the widest available vector at codegen — no recompile needed for new CPUs.

Explicit Vector128/256/512<T>

When you need a specific width:

using System.Runtime.Intrinsics;
using System.Runtime.Intrinsics.X86;

public static unsafe int Sum256(ReadOnlySpan<int> data)
{
    if (!Avx2.IsSupported) return SumFallback(data);
    fixed (int* p = data)
    {
        var acc = Vector256<int>.Zero;
        int i = 0;
        for (; i + 8 <= data.Length; i += 8)
            acc = Avx2.Add(acc, Avx.LoadVector256(p + i));
        // Horizontal sum
        int total = 0;
        for (int k = 0; k < 8; k++) total += acc.GetElement(k);
        for (; i < data.Length; i++) total += p[i];
        return total;
    }
}

Always check IsSupported and provide a fallback.

IsHardwareAccelerated

if (Vector.IsHardwareAccelerated) { /* use Vector<T> */ }

False on platforms without SIMD instructions — fall back to scalar.

AVX-512 (.NET 8+)

if (Avx512F.IsSupported)
{
    var v = Vector512.Create(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15);
    // ...
}

512-bit vectors — 16 ints, 8 longs, 8 doubles per op. Available on Sapphire Rapids and Zen 4+.

SearchValues<T> (.NET 8+)

The simplest SIMD entry point:

private static readonly SearchValues<char> Vowels =
    SearchValues.Create("aeiouAEIOU");

string Strip(string s) => string.Concat(s.AsSpan().Trim(Vowels));
int firstVowel = s.AsSpan().IndexOfAny(Vowels);

Internally uses SIMD when beneficial, falls back automatically. Use this first before hand-rolling intrinsics.

Tensor primitives (.NET 9+)

System.Numerics.Tensors.TensorPrimitives has SIMD-accelerated math over ReadOnlySpan<T>:

TensorPrimitives.Add(a, b, dest);
TensorPrimitives.CosineSimilarity(query, doc);
TensorPrimitives.Sum(values);

For ML embedding workloads (dot product, normalize, cosine similarity), preferred over hand-rolled SIMD.

Auto-vectorization

JIT auto-vectorizes simple loops over Span<T>:

public static int Count(ReadOnlySpan<int> data, int target)
{
    int c = 0;
    for (int i = 0; i < data.Length; i++)
        if (data[i] == target) c++;
    return c;
}

In .NET 8+ this generates SIMD code automatically. Profile before manually vectorizing.

Loading / storing

var v = Vector256.LoadUnsafe(ref data[0]);   // ref-based
var v2 = Vector256.Create(data);              // copies span
Vector256.StoreUnsafe(v, ref result[0]);

Misaligned loads (LoadVector256 on unaligned address) work but slower on older CPUs; modern (Skylake+) handles unaligned at full speed.

Tail handling

After the SIMD loop, process the remaining length % stride elements scalar. Forgetting this causes off-by-one bugs.

Code: correct vs wrong

❌ Wrong: ignoring IsSupported

public int Sum(int[] data)
{
    var v = Vector256<int>.Zero;
    // crashes on hardware without AVX
}

✅ Correct: gated with fallback

public int Sum(ReadOnlySpan<int> data)
{
    if (Avx2.IsSupported) return SumAvx2(data);
    if (Vector.IsHardwareAccelerated) return SumVector(data);
    return SumScalar(data);
}

❌ Wrong: hand-rolled when auto-vectorization works

Profile first. Often the simple for loop generates identical SIMD code.

✅ Correct: SearchValues for character search

private static readonly SearchValues<byte> Whitespace = SearchValues.Create(" \t\r\n"u8);
int next = data.IndexOfAny(Whitespace);

Design patterns for this topic

Pattern 1 — Fallback ladder

return Avx512F.IsSupported ? Process512(data)
     : Avx2.IsSupported    ? Process256(data)
     : Sse2.IsSupported    ? Process128(data)
     : ProcessScalar(data);

Order from widest to scalar; IsSupported checks elide on platforms that don't have the ISA.

Pattern 2 — SearchValues for byte/char filtering

Pre-compute a SearchValues<T> at startup, reuse forever — internally chooses the optimal SIMD strategy for that set.

Pattern 3 — TensorPrimitives for vector math

Span<float> norm = stackalloc float[embedding.Length];
TensorPrimitives.Divide(embedding, TensorPrimitives.Norm(embedding), norm);

For embedding pipelines and ML data prep.

Pattern 4 — Aligned span iteration

for (int i = 0; i + Vector256<int>.Count <= data.Length; i += Vector256<int>.Count)
{
    var v = Vector256.LoadUnsafe(ref Unsafe.Add(ref MemoryMarshal.GetReference(data), i));
    // ...
}

Modern alternative to unsafe + fixed.

Pros & cons / trade-offs

When to use / when to avoid

• Reach for: tight loops over blittable data, hot paths in parsers/codecs/serializers/ML.
• Avoid: business logic, anything with branches per element, code your team can't review.
• Always: benchmark before and after — sometimes scalar wins on small N.

Interview Q&A

Q1. Difference between Vector<T> and Vector256<T>? A. Vector<T> is width-agnostic — JIT picks the widest available SIMD on the host. Vector256<T> is fixed at 256 bits. Use Vector<T> for portable code; the typed widths when you need a specific instruction.

Q2. What's a horizontal sum? A. Reducing a SIMD vector to a single scalar — adding all lanes together. Vector.Sum(v) does it cross-platform; Avx2.HorizontalAdd does it on AVX2.

Q3. Why might intrinsics not beat scalar? A. Loop overhead, branch mispredictions on the tail loop, small N, memory bandwidth bottleneck. Always benchmark; SIMD shines on >100s of elements.

Q4. What's IsHardwareAccelerated? A. Static property indicating the runtime has detected SIMD instructions. False on ancient hardware or in some emulators.

Q5. How does SearchValues<T> choose its strategy? A. Inspects the value set: small set of bytes → bitmap SIMD; ASCII printable → vectorized lookup; arbitrary chars → SIMD popcount. Implementation detail; use the API and let it pick.

Q6. What does Vector512 add over Vector256? A. Twice the width (16 ints vs 8) on Sapphire Rapids / Zen 4+. Gain depends on memory bandwidth — often not 2× because data movement dominates.

Q7. Why LoadUnsafe over LoadVector256? A. LoadUnsafe takes ref T (managed), works without unsafe. LoadVector256(T*) takes a pointer (unsafe). Same codegen.

Q8. How does AVX-512 affect CPU clock? A. On older Intel CPUs, AVX-512 instructions throttled the clock for the whole core; modern (Sapphire Rapids+) fixed this. Test on your target hardware.

Gotchas / common mistakes

• Forgetting IsSupported checks → PlatformNotSupportedException at runtime.
• Off-by-one in the tail loop after SIMD pass.
• Misaligned loads on old CPUs — measurable perf loss.
• Hand-rolling SIMD when JIT auto-vectorizes the same loop.
• Using Vector<T> and assuming a specific width.
• Wide vectors causing thermal throttling — measure under sustained load.
• Trim/AOT pruning intrinsic types — usually fine, but verify.
• SIMD division — slow on most ISAs; multiply by reciprocal where possible.

Further reading

• SIMD-accelerated types
• System.Runtime.Intrinsics
• SearchValues
• TensorPrimitives
• Hardware intrinsics deep dive