Parallelism & Data Parallel

Key Points

• Parallelism is for CPU-bound work; async is for I/O-bound. Don't conflate.
• Parallel.ForEach for CPU-bound sync; Parallel.ForEachAsync (.NET 6+) for async I/O fan-out with MaxDegreeOfParallelism.
• PLINQ (AsParallel()) — declarative data parallelism. Auto-partitions; aggregates results. Good fit for pure functional pipelines.
• Task.WhenAll for unbounded fan-out; Channel<T> for producer-consumer.
• Watch for: shared mutable state (need Interlocked/ConcurrentDictionary), partitioning costs (small workloads slower than serial), thread oversubscription.

Concepts (deep dive)

Two axes

            CPU-bound        I/O-bound
Parallel    Parallel.For*    don't (use async)
Async       Task.Run(CPU)    await

CPU-bound parallelism uses multiple cores. Async I/O frees the thread while waiting. Different problems.

Parallel.ForEach — synchronous CPU work

Parallel.ForEach(images, img => Resize(img));   // CPU-bound, sync

Auto-partitions; uses threadpool; MaxDegreeOfParallelism caps concurrency.

Parallel.ForEach(images, new ParallelOptions { MaxDegreeOfParallelism = 4 }, Resize);

Parallel.ForEachAsync — async I/O fan-out

await Parallel.ForEachAsync(urls,
    new ParallelOptions { MaxDegreeOfParallelism = 20 },
    async (url, ct) => await _http.GetAsync(url, ct));

Bounded async parallelism — exactly what you want for HTTP fan-out, queue draining, etc.

PLINQ

var hashes = files
    .AsParallel()
    .WithDegreeOfParallelism(8)
    .Select(f => Hash(f))
    .ToArray();

Declarative; preserves order if .AsOrdered(); aggregates partition results.

Caveats: - Order can change unless .AsOrdered(). - Partitioning has overhead — small inputs may be slower. - Side effects in lambdas → race conditions.

Task.WhenAll

var results = await Task.WhenAll(items.Select(async i => await ProcessAsync(i)));

Unbounded — all tasks start at once. Fine for ~tens; problematic for thousands.

Channels — producer-consumer

var channel = Channel.CreateBounded<WorkItem>(100);

// Producer
_ = Task.Run(async () =>
{
    foreach (var item in source)
        await channel.Writer.WriteAsync(item);
    channel.Writer.Complete();
});

// Consumers
var consumers = Enumerable.Range(0, 4).Select(_ => Task.Run(async () =>
{
    await foreach (var item in channel.Reader.ReadAllAsync())
        await ProcessAsync(item);
}));
await Task.WhenAll(consumers);

Bounded buffer + multiple consumers + backpressure. The .NET-idiomatic streaming pipeline.

Shared state

Race conditions are the #1 parallel bug:

int sum = 0;
Parallel.ForEach(numbers, n => sum += n);   // ❌ race; lost updates

Fix:

int sum = 0;
Parallel.ForEach(numbers, n => Interlocked.Add(ref sum, n));   // ✅

// Or thread-local then aggregate:
Parallel.ForEach(numbers,
    () => 0,                     // thread-local init
    (n, _, local) => local + n,   // body
    local => Interlocked.Add(ref sum, local));   // combine

PLINQ's Aggregate(...) overload is the declarative version.

Partitioner

For non-uniform work:

Parallel.ForEach(Partitioner.Create(0, items.Length, chunkSize: 100),
    range =>
    {
        for (int i = range.Item1; i < range.Item2; i++)
            Process(items[i]);
    });

Reduces overhead per item.

When parallelism hurts

• Tiny workloads: partitioning overhead > work.
• Memory-bandwidth bound: more cores don't help; bandwidth is the limit.
• Lots of synchronization: contention serializes work.
• Inherently sequential (e.g., dependency chains).

Always benchmark.

Oversubscription

Parallel.ForEach(items, item =>
    Parallel.ForEach(item.SubItems, sub => Process(sub)));   // nested → far too many threads

Use MaxDegreeOfParallelism = -1 (unbounded) only carefully. Nested Parallel.ForEach rarely a good idea.

ConcurrentDictionary and friends

var counts = new ConcurrentDictionary<string, int>();
Parallel.ForEach(words, w => counts.AddOrUpdate(w, 1, (_, c) => c + 1));

Lock-free reads, fine-grained locking on writes. For high-write workloads, partition by key (sharded dictionaries).

ConcurrentBag<T>, ConcurrentQueue<T>, ConcurrentStack<T> for collections.

Vectorization (SIMD)

var v1 = new Vector<int>(input);
var v2 = new Vector<int>(input2);
var sum = v1 + v2;

System.Numerics.Vector uses SIMD instructions (4–8 ints per op). For numeric-heavy code (image processing, ML), 4–10x speedup possible.

For more control: System.Runtime.Intrinsics (Vector256<int>, AVX2/AVX512 intrinsics). Stephen Toub's blog has deep dives.

Parallel.For vs LINQ vs PLINQ

// Sequential
var sum = 0; for (int i = 0; i < n; i++) sum += f(i);

// Parallel.For
Parallel.For(0, n, i => Interlocked.Add(ref sum, f(i)));

// PLINQ
sum = Enumerable.Range(0, n).AsParallel().Sum(f);

PLINQ is most readable. Parallel.For most controllable. Choose based on style + perf needs.

CPU vs Memory bound

Profile to know which.

Async + parallelism

Parallel.ForEachAsync is the right combo. Don't Task.Run inside loop bodies — it just adds thread hops.

Cancellation in parallel

Parallel.ForEach(items, new ParallelOptions { CancellationToken = ct }, Process);

Aborts the loop on cancel. Already-started items finish.

Code: correct vs wrong

❌ Wrong: race on shared state

List<int> results = new();
Parallel.ForEach(items, i => results.Add(Process(i)));

List<T> is not thread-safe.

✅ Correct: thread-safe collection

ConcurrentBag<int> results = new();
Parallel.ForEach(items, i => results.Add(Process(i)));

Or build per-thread, aggregate.

❌ Wrong: Parallel.ForEach for I/O

Parallel.ForEach(urls, url => _http.GetAsync(url).Result);   // sync over async

✅ Correct: async parallel

await Parallel.ForEachAsync(urls, async (url, ct) => await _http.GetAsync(url, ct));

❌ Wrong: PLINQ on tiny inputs

var doubled = new[] { 1, 2, 3 }.AsParallel().Select(x => x * 2).ToArray();

Partitioning overhead > work.

✅ Correct: PLINQ on big inputs

var hashes = millionFiles.AsParallel().Select(Hash).ToArray();

Design patterns for this topic

Pattern 1 — "Parallel.ForEachAsync for I/O fan-out"

• Intent: bounded async parallelism.

Pattern 2 — "Channel for streaming pipelines"

• Intent: producer-consumer with backpressure.

Pattern 3 — "Thread-local + aggregate"

• Intent: avoid contention on shared state.

Pattern 4 — "SIMD via System.Numerics.Vector"

• Intent: numeric speedup.

Pattern 5 — "Sharded ConcurrentDictionary for high-write"

• Intent: reduce contention.

Pros & cons / trade-offs

When to use / when to avoid

• Use Parallel.ForEachAsync for I/O fan-out.
• Use PLINQ for declarative CPU parallelism on large inputs.
• Use SIMD for numeric loops.
• Avoid Parallel.ForEach for I/O.
• Avoid parallelism on small inputs.

Interview Q&A

Q1. CPU-bound vs I/O-bound parallelism? CPU: many cores executing. I/O: many requests in flight, threads await. Different patterns.

Q2. Parallel.ForEach vs Parallel.ForEachAsync? ForEach: sync work. ForEachAsync: async work; bounded; respects async.

Q3. PLINQ caveats? Order changes unless .AsOrdered(). Side effects break. Tiny inputs slower than serial.

Q4. How avoid race conditions? Thread-safe collections, Interlocked, lock, or thread-local + aggregate pattern.

Q5. Task.WhenAll vs Parallel.ForEachAsync? WhenAll: unbounded — all tasks start. ForEachAsync: bounded by MaxDegreeOfParallelism.

Q6. Channel use case? Producer-consumer with bounded buffer + backpressure. Multiple producers/consumers.

Q7. Why Task.Run inside parallel-loop body bad? Adds thread hop. Body already runs on a threadpool thread.

Q8. SIMD speedup? 4–10x for numeric loops. Vector auto-uses available width.

Q9. ConcurrentDictionary high-write contention? Striped locking; for very hot keys, shard manually.

Q10. Cancellation in Parallel? new ParallelOptions { CancellationToken = ct }. Loop aborts on cancel.

Q11. Memory-bandwidth bound — does parallelism help? No. Adding cores doesn't speed bandwidth; may even hurt via cache contention.

Q12. Partitioner use case? Tune chunk size for non-uniform work. Reduces per-item overhead.

Gotchas / common mistakes

• ⚠️ Race on shared state — use Interlocked or thread-local.
• ⚠️ Parallel for I/O — use async parallel instead.
• ⚠️ Tiny inputs in PLINQ — overhead dominates.
• ⚠️ Nested Parallel.ForEach — oversubscription.
• ⚠️ Side effects in PLINQ lambdas — order undefined.

Further reading

• MS Learn: Parallel programming
• Stephen Toub: SIMD in .NET
• System.Threading.Channels
• PLINQ docs