Skip to content

Distributed Tracing

Key Points

  • A trace is a tree of spans (operations) representing one logical request through multiple services.
  • W3C TraceContext (traceparent header) is the standard. Propagated across HTTP, gRPC, message queues. Replaced older formats (B3, Jaeger).
  • Trace ID identifies the whole request; Span ID identifies one operation. Parent Span ID links the tree.
  • Activity / ActivitySource are .NET's native primitives. OpenTelemetry uses them.
  • Common pitfalls: lost context across Task.Run, missing propagation in custom queue consumers, sampling decisions inconsistent across services.

Concepts (deep dive)

Trace anatomy

Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736
├── Span: GET /orders                  (root, web service)
│   ├── Span: SELECT * FROM orders     (db span)
│   ├── Span: GET /products            (HttpClient span to product service)
│   │   └── Span: SELECT * FROM products (db span in product service)
│   └── Span: PUBLISH order.viewed     (queue span)

Each span has: name, start/end timestamps, status, attributes, events, links.

W3C TraceContext

traceparent: 00-{trace-id}-{span-id}-{flags}
               00          version
                  16-byte hex trace-id
                                32-byte hex span-id (parent for next hop)
                                                01 = sampled

tracestate: vendor1=foo,vendor2=bar       (vendor-specific extras)

When service A calls service B, A includes its current span ID as the parent. B creates a new span as a child of A's span. The trace ID stays the same throughout.

Auto-propagation in .NET

var http = factory.CreateClient();
var r = await http.GetAsync("/api/data");
// HttpClient instrumentation auto-adds traceparent header.

// In the receiver (also instrumented):
// ASP.NET reads traceparent → creates a child Activity → all spans within are children.

When OpenTelemetry is registered with AddAspNetCoreInstrumentation and AddHttpClientInstrumentation, this is automatic.

Manual span creation

private static readonly ActivitySource _src = new("MyApp.Domain");

using var activity = _src.StartActivity("CalculateTotal", ActivityKind.Internal);
activity?.SetTag("order.id", orderId);
activity?.SetTag("line.count", lines.Count);

ActivityKind: - Server — incoming request (ASP.NET sets this). - Client — outbound call (HttpClient sets this). - Producer / Consumer — message queue. - Internal — within service.

Cross-process: HTTP

Auto. HttpClient writes; ASP.NET reads.

Cross-process: gRPC

.AddGrpcClientInstrumentation()

Auto-propagates via gRPC metadata.

Cross-process: message queues

NOT automatic for many brokers. Inject manually:

// Producer
using var activity = _src.StartActivity("publish", ActivityKind.Producer);
var props = new Dictionary<string, string>();
DistributedContextPropagator.Current.Inject(activity, props,
    static (carrier, key, value) => ((Dictionary<string,string>)carrier!)[key] = value);

await broker.PublishAsync(payload, props);

// Consumer
var parentContext = DistributedContextPropagator.Current.Extract(props,
    static (carrier, key, out value, out var values) =>
    {
        var dict = (Dictionary<string,string>)carrier!;
        value = dict.GetValueOrDefault(key);
        values = null;
    });

using var activity = _src.StartActivity("consume", ActivityKind.Consumer,
    parentContext: parentContext);

Newer brokers (RabbitMQ MassTransit v8+, Kafka via Confluent.Kafka.OpenTelemetry, Azure Service Bus SDK) handle this automatically.

Activity loss in Task.Run / threadpool

Activity.Current lives on ExecutionContext, which flows on Task.Run/async/await by default. But:

// ❌ Loses context if ExecutionContext is suppressed:
using (ExecutionContext.SuppressFlow())
    Task.Run(() => DoWork());

// ❌ Loses context across `Thread.Start`:
new Thread(() => DoWork()).Start();

For long-lived background workers, link the new activity to the originating one:

var newActivity = _src.StartActivity("background-work",
    ActivityKind.Internal,
    parentContext: default,
    links: new[] { new ActivityLink(originatingActivity.Context) });

Sampling consistency

If service A samples 10% and service B samples 50%, you get incomplete traces. Use parent-based sampling — once the root sampler decides, all downstream services honor it.

.SetSampler(new ParentBasedSampler(new TraceIdRatioBasedSampler(0.1)))

Tail sampling (in collector)

Sample after the trace completes:

# OTel collector config
processors:
  tail_sampling:
    policies:
      - { name: errors, type: status_code, status_code: { status_codes: [ERROR] } }
      - { name: slow,   type: latency,     latency: { threshold_ms: 500 } }
      - { name: random, type: probabilistic, probabilistic: { sampling_percentage: 1 } }

Keep all error/slow traces + 1% of normal. The most common production setup.

Span attributes vs events

  • Attribute (tag): static metadata about the span. db.statement, user.id.
  • Event: time-stamped occurrence within the span. "cache miss", "retry attempt".

Cross-trace relationships (e.g., a fan-out where one span causes 100 child traces). Use ActivityLink.

Performance and overhead

Each span: ~1µs creation, ~1KB serialized. With sampling at 10%, overhead is sub-1% in CPU and bandwidth. Without sampling, hot endpoints can hit 5–10%.

Visualization

  • Jaeger: open-source. Span tree views.
  • Tempo: Grafana's trace store. Pairs with Loki/Prometheus.
  • Datadog APM, New Relic, App Insights: commercial.

All consume OTLP.

Real-world example

A slow /orders request:

GET /orders                       │█████████████████████ 1200ms
  ├─ SELECT FROM orders           │██ 30ms
  ├─ HTTP GET /products           │█████████████████ 1100ms ← culprit
  │   └─ db.query                 │███ 120ms
  │   └─ slow.thirdparty.api      │████████████ 950ms ← real culprit
  └─ kafka.publish                │█ 20ms

Tracing pinpointed the third-party API call inside the products service — invisible from logs alone.

Correlating logs with traces

// Logs auto-attached with TraceId/SpanId when using OTel logging:
_log.LogInformation("Processing order {OrderId}", id);
// Output:
// { "OrderId": 123, "TraceId": "4bf...", "SpanId": "00f..." }

In Datadog/Tempo, click a span → linked logs filter to those IDs.

Code: correct vs wrong

❌ Wrong: starting Activity outside ActivitySource

var activity = new Activity("MySpan").Start();   // not collected

✅ Correct: via ActivitySource (which has a listener)

using var activity = _src.StartActivity("MySpan");

❌ Wrong: throwing without recording

catch (Exception ex) { throw; }   // span has no error info

✅ Correct: status + record

catch (Exception ex)
{
    activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
    activity?.RecordException(ex);
    throw;
}

❌ Wrong: missing propagation in queue

await producer.PublishAsync(payload);
// consumer creates a new trace; not linked

✅ Correct: inject context

DistributedContextPropagator.Current.Inject(activity, props, ...);

Design patterns for this topic

Pattern 1 — "Parent-based sampling"

  • Intent: consistent sampling across services.

Pattern 2 — "Tail sampling at collector"

  • Intent: keep slow/error traces; drop noise.
  • Intent: relate spans that aren't direct parents.

Pattern 4 — "Manual propagation for custom queues"

  • Intent: stitch async/queue traces.

Pattern 5 — "Logs joined to traces by TraceId"

  • Intent: click-through investigation.

Pros & cons / trade-offs

Aspect Pros Cons
Auto-instrumentation Free coverage Sometimes too verbose
Manual spans Domain visibility Code clutter if overdone
Tail sampling Best traces kept Collector buffer cost
Propagation Cross-service correlation Extra header bytes

When to use / when to avoid

  • Always for distributed systems.
  • Add manual spans for important domain operations only.
  • Avoid spans for micro-operations in hot loops.
  • Avoid inconsistent sampling across services.

Interview Q&A

Q1. What's a span? One operation in a trace — name, timing, status, attributes, parent link.

Q2. W3C TraceContext format? traceparent: 00-{traceId}-{spanId}-{flags}. Standard across vendors.

Q3. ActivityKind values? Server (incoming), Client (outgoing), Producer/Consumer (queue), Internal.

Q4. Why does Activity sometimes get lost across threads? ExecutionContext flow can be suppressed or not propagated (Thread.Start, SuppressFlow). Use links to relate.

Q5. Auto-instrumentation in .NET — what's covered? ASP.NET, HttpClient, EF Core, gRPC, SqlClient, Npgsql, Redis. ~80% of telemetry for free.

Q6. Why parent-based sampling? Ensures all services in a trace make the same decision — full traces, not partial.

Q7. Tail sampling? Decision after trace completes (in collector). Keep slow/error; drop random.

Q8. How propagate across queues? Inject traceparent into message headers. Receiver extracts when starting consumer span.

Q9. Logs and traces — how correlated? OTel logging attaches TraceId/SpanId from Activity.Current. Backend joins by ID.

Q10. What's a span link? Reference to another span that's related but not parent (fan-out, batch processing).

Q11. Can sampling lose error traces? Yes with head sampling. Use tail sampling at collector to keep all errors.

Q12. Cost of distributed tracing? ~1µs per span; ~1KB serialized. With sampling, sub-1% overhead.

Gotchas / common mistakes

  • ⚠️ No propagation in custom queues — broken traces.
  • ⚠️ Sampling differs across services — partial traces.
  • ⚠️ Excessive manual spans in tight loops.
  • ⚠️ Activity created without ActivitySource listener — not collected.
  • ⚠️ Forgetting RecordException.
  • ⚠️ Lost context in Thread.Start without ExecutionContext flow.

Further reading