Sampling Strategies

Key Points

• You can't trace everything. At 100K req/s, full-fidelity tracing is millions of dollars/year. Sampling is mandatory.
• Head-based sampling decides at the start of a trace (in the SDK). Cheap, simple, but biased toward fast happy-path traces — slow/error traces are statistically rare and likely dropped.
• Tail-based sampling decides after the trace completes (in the OTel Collector). Lets you keep 100% of errors and 100% of slow requests. Costs collector memory + a buffering window.
• Parent-based sampling propagates the upstream decision. Mandatory for trace consistency: either all spans of a trace are kept or none are. Otherwise you get orphan spans — useless.
• Adaptive sampling auto-tunes the rate to hit a target events-per-second budget.
• Sampling for logs is different: usually rule-based ("drop healthy 200s, keep 5xx"), not probabilistic. Driven by cost more than fidelity.
• Worst outcome of bad sampling: lose the slow tail and the error traces — exactly the ones you needed.

Concepts (deep dive)

Why sample at all

A typical trace has 5–50 spans. At 100K req/s, that's 500K–5M spans/s. Even at $0.50/M spans, full sampling costs $20–200K/day. You also can't store, index, or search that volume usefully — most traces are 200 OK in 50 ms and tell you nothing.

The goal: keep the interesting traces, drop the boring ones.

Head-based vs tail-based

HEAD-BASED (decision at trace start, in the SDK)
[Service A]          [Service B]          [Service C]
   |                      |                     |
 sampled?-----traceparent flags-------------> respect
 yes/no
   |
 emit or drop ---- decision is final ---->

Pros: cheap, no coordinator, low latency.
Cons: random — favors common paths. Errors usually slip through unsampled.

TAIL-BASED (decision after trace completes, in the Collector)
[A]----all spans----+
[B]----all spans----+----> [Collector buffer (e.g., 30s)]
[C]----all spans----+              |
                                   v
                         { policy evaluation }
                          - error? keep
                          - slow > p99? keep
                          - else: 1% sample
                                   |
                                   v
                              [Backend]

Pros: keep 100% of errors / slow requests.
Cons: collector RAM, completion-window latency, sticky routing required.

Parent-based sampling — the consistency rule

A trace spans many services. If service A samples in but service B samples out, you get a broken trace with missing spans. Useless.

ParentBasedSampler enforces: if upstream sampled, we sample; if upstream didn't, we don't. The decision is made once, at the trace root, and rides on the traceparent header's sampled flag.

.SetSampler(new ParentBasedSampler(
    rootSampler: new TraceIdRatioBasedSampler(0.05)))   // 5% of new traces

If your service is a root (entry point), the root sampler decides. If it's downstream, parent decision wins.

Probabilistic / TraceIdRatioBased

.SetSampler(new TraceIdRatioBasedSampler(0.10))         // 10%

Hashes the traceId to decide. Two crucial properties:

1. Deterministic: same traceId → same decision. Great for joining traces with logs/metrics.
2. Consistent across services: every service hashing the same traceId at the same ratio reaches the same answer. Even without parent-based, you'd get coherent traces (in theory). In practice, mixed sample rates exist; use parent-based for safety.

Rule-based / per-service

# OTel Collector tail_sampling processor
policies:
  - name: errors-policy
    type: status_code
    status_code: { status_codes: [ERROR] }
  - name: slow-policy
    type: latency
    latency: { threshold_ms: 500 }
  - name: probabilistic
    type: probabilistic
    probabilistic: { sampling_percentage: 1 }

Combine: keep all errors, all slow requests, 1% of the rest. This gives you a usable signal at a fraction of the cost.

Adaptive sampling

Adjust the rate dynamically to hit a target spans-per-second budget. If traffic doubles, halve the rate automatically. Many vendors (Datadog, New Relic) ship this; in OTel, you can build it in the Collector with transform + dynamic ratio, or use vendor extensions.

Rule of thumb: target a fixed traces/second budget (e.g., 1000/s), not a fixed ratio. Ratios punish you when traffic spikes.

Stratified sampling

Different rules per service / per route:

• Public API: 1% probabilistic + 100% errors.
• Background worker: 100% (low volume; high diagnostic value).
• Health-check probe: 0%.
• Payment service: 100% always (compliance).

The Collector's tail_sampling processor with named policies + services filter is how you express this.

Sampling and traceId determinism

OpenTelemetry uses the lowest 14 hex digits of traceId for ratio sampling. As long as all services use compatible algorithms (W3C Trace Context defines a recommended approach), they reach the same decision independently — useful when parent-based is impractical.

.NET SDK configuration

builder.Services.AddOpenTelemetry()
    .WithTracing(t => t
        .AddSource("MyApp.*")
        .SetSampler(new ParentBasedSampler(
            new TraceIdRatioBasedSampler(0.10)))
        .AddOtlpExporter());

For environment-driven config:

OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.10

Collector tail-sampling

processors:
  tail_sampling:
    decision_wait: 30s            # buffer window
    num_traces: 100000            # in-flight traces cap
    expected_new_traces_per_sec: 5000
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 1000 }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 1.0 }

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlp/tempo]

⚠️ Tail-sampling collectors are stateful for the buffer window. Scale them carefully: - All spans of a trace must reach the same collector instance. - Use loadbalancing exporter with routing_key: traceID to hash-route trace spans to a fixed collector. - Memory sized for num_traces × avg_spans × span_size.

Sampling for logs

Tracing sampling is statistical; log sampling is editorial:

The last is powerful: trace_sampled = true → keep logs. Implement in the Collector or in your logging filter.

Cost vs fidelity

Cost  ←————————— sampling rate —————————→  Fidelity
high                                          high
                  sweet spot
                  (tail-sampled + 100% errors)

The single worst sampling outcome is uniform low rate (e.g., 1% probabilistic). Errors are <1% of traffic; you'll lose most of them. Always combine with rule-based "keep errors / slow."

Anti-patterns

• Different rates per service without parent-based → orphan spans.
• Pure random with no error keep-rule → invisible incidents.
• Tail-sampling without sticky routing → spans of one trace land in different collectors → tail-sampling sees partial trace → wrong decision.
• Over-aggressive sampling (0.1%) → can't reproduce customer issue from traces.
• Sampling decisions made post-collection → already paid the bandwidth.

Sampling and metrics

Metrics are aggregated; you do not sample metrics. A 5% sample rate would corrupt counters, histograms, and rates. Metrics are cheap because they're aggregated upstream — keep them at 100%. Cardinality control is the metrics analogue of sampling.

How it works under the hood

[Trace lifecycle with parent-based + ratio sampler]

Service A (entry)
  - new traceId T
  - root sampler: hash(T) < 0.10? → SAMPLED flag = 1
  - emits spans tagged sampled=1
  - sets traceparent: 00-T-S1-01
       |
       v
Service B
  - reads traceparent flags=01 → sampled
  - parent-based sampler: respect parent → SAMPLED
  - emits spans tagged sampled=1
       |
       v
Service C
  - same: respect parent → SAMPLED
  - emits spans

All sampled or none sampled — never partial.

[Tail-sampling collector]
spans arrive (all sampled at SDK = 100% to collector)
       |
       v
buffered by traceId for `decision_wait`
       |
       v
on completion timeout OR span signaling end:
  - evaluate policies
  - keep | drop
       |
       v
exported to backend

Code: correct vs wrong

❌ Wrong: AlwaysOn in production

.SetSampler(new AlwaysOnSampler())

Cost balloons; backend chokes.

✅ Correct: parent-based ratio

.SetSampler(new ParentBasedSampler(new TraceIdRatioBasedSampler(0.10)))

❌ Wrong: independent ratio per service

// Service A: 0.10
// Service B: 0.05    <-- mismatched

Half of A's sampled traces lose B's spans.

✅ Correct: parent-based downstream

// Service B
.SetSampler(new ParentBasedSampler(new TraceIdRatioBasedSampler(0.10)))
// Will defer to parent flag from A.

❌ Wrong: probabilistic only

- name: baseline
  type: probabilistic
  probabilistic: { sampling_percentage: 1.0 }

Errors lost.

✅ Correct: errors + slow + baseline

- name: errors
  type: status_code
  status_code: { status_codes: [ERROR] }
- name: slow
  type: latency
  latency: { threshold_ms: 1000 }
- name: baseline
  type: probabilistic
  probabilistic: { sampling_percentage: 1.0 }

❌ Wrong: tail-sampling without sticky routing

Round-robin LB to the tail-sampling collector pool.

✅ Correct: traceId-hash routing

exporters:
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp: { tls: { insecure: true } }
    resolver:
      static: { hostnames: [collector-tail-1, collector-tail-2] }

Design patterns for this topic

Pattern 1 — "Parent-based ratio at the SDK"

• Intent: consistent traces, low overhead.

Pattern 2 — "Tail-sampling for errors and slow tail"

• Intent: keep what matters; drop noise.

Pattern 3 — "Stratified per-service rules"

• Intent: payment 100%, public 1%, health 0%.

Pattern 4 — "Adaptive rate by budget"

• Intent: stable cost despite traffic spikes.

Pattern 5 — "Trace-aware log keep"

• Intent: keep logs of sampled traces; drop the rest.

Pros & cons / trade-offs

When to use / when to avoid

• Always use parent-based at the SDK.
• Use tail-sampling at the gateway collector for production.
• Use "keep errors / slow / baseline %" combo as the default policy.
• Avoid different head ratios across services without parent-based.
• Avoid sampling metrics — cardinality control instead.

Interview Q&A

Q1. Why sample traces? Cost. At scale full tracing is unaffordable and unsearchable. Sampling keeps signal, drops noise.

Q2. Head vs tail sampling? Head: decide at start, fast, biased to common. Tail: decide after completion in the collector, captures errors/slow tail, requires buffering and sticky routing.

Q3. Why parent-based? Trace consistency. Without it, services make independent decisions and you get orphan spans.

Q4. TraceIdRatioBased — how does it work? Hashes the traceId; compares to ratio. Deterministic per traceId; consistent across services with the same algorithm.

Q5. What's the worst sampling outcome? Losing slow/error traces. A pure 1% probabilistic policy will drop most errors because errors are rare.

Q6. Sampling vs cardinality control? Sampling for traces and logs. Cardinality control for metrics — never sample metrics; you'd corrupt aggregates.

Q7. Tail-sampling memory? num_traces × avg_spans × span_size. With decision_wait=30s and 5K traces/s, expect GBs.

Q8. Sticky routing? LB hash by traceId so all spans of a trace land on the same tail-sampling collector instance.

Q9. Adaptive sampling? Auto-adjust rate to hit a target events/sec budget. Stable cost under traffic spikes.

Q10. Sampling logs? Rule-based usually: drop healthy 2xx, keep all errors, rate-limit chatty loggers, or keep logs whose trace was sampled.

Q11. Sampling and metrics? Don't. Aggregates would be wrong. Use cardinality limits and meter pre-aggregation.

Q12. Multiple ratios across a service mesh? Use parent-based with the strictest root rate; don't ask each service to roll its own.

Gotchas / common mistakes

• ⚠️ No parent-based → orphan spans across services.
• ⚠️ Pure probabilistic → errors invisible.
• ⚠️ Tail-sampling without sticky routing → wrong decisions.
• ⚠️ Sampling metrics → broken aggregates.
• ⚠️ Decision_wait too short → late-arriving spans dropped.
• ⚠️ Decision_wait too long → collector OOM.
• ⚠️ Forgetting health-check exclusions → noise dominates the budget.

Further reading

• OTel sampling docs
• OTel Collector tail_sampling
• W3C Trace Context
• .NET TraceIdRatioBasedSampler