OpenTelemetry: Traces, Metrics, Logs

Key Points

• OpenTelemetry (OTel) is the vendor-neutral standard for emitting observability data: traces (causal request flow), metrics (aggregated numbers), logs. CNCF graduated.
• .NET integration: OpenTelemetry.Extensions.Hosting + auto-instrumentation packages for ASP.NET, HttpClient, EF, gRPC, etc.
• OTLP (OpenTelemetry Protocol) ships data to a collector (vendor-neutral) → vendor backend (Datadog, Jaeger, Tempo, Prometheus, App Insights).
• Activity (System.Diagnostics) is the .NET trace primitive. ActivitySource emits spans. Same API ML, EF, ASP.NET use under the hood.
• Meter / Counter<T> / Histogram<T> for metrics. Modern API — replaces EventCounter/PerfCounter.

Concepts (deep dive)

The three pillars

Traces ─ How a request flowed: A → B → C with timings
Metrics ─ How many, how fast, how big (aggregated, fast)
Logs ─ Discrete events with rich context

OpenTelemetry unifies all three on one wire format (OTLP) and one SDK.

Setup

builder.Services.AddOpenTelemetry()
    .ConfigureResource(r => r.AddService(serviceName: "MyApp", serviceVersion: "1.0.0"))
    .WithTracing(t => t
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddEntityFrameworkCoreInstrumentation()
        .AddSource("MyApp.*")           // your custom ActivitySources
        .AddOtlpExporter(o => o.Endpoint = new Uri("http://otel-collector:4317")))
    .WithMetrics(m => m
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddRuntimeInstrumentation()    // GC, threadpool, etc.
        .AddProcessInstrumentation()
        .AddMeter("MyApp.*")
        .AddOtlpExporter())
    .WithLogging(l => l.AddOtlpExporter());

Auto-instrumentation packages

A modern .NET app gets ~80% useful telemetry from auto-instrumentation alone.

Custom traces (Activity)

public class OrderService
{
    private static readonly ActivitySource _src = new("MyApp.Orders");

    public async Task PlaceAsync(PlaceOrder cmd, CancellationToken ct)
    {
        using var activity = _src.StartActivity("OrderService.Place");
        activity?.SetTag("order.customerId", cmd.CustomerId);
        activity?.SetTag("order.lineCount", cmd.Lines.Count);

        try
        {
            // ... work
            activity?.SetStatus(ActivityStatusCode.Ok);
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.RecordException(ex);
            throw;
        }
    }
}

ActivitySource.StartActivity returns null if no listener is configured — ?. is fine. Tags are searchable.

Custom metrics (Meter)

public class OrderMetrics
{
    private static readonly Meter _meter = new("MyApp.Orders", "1.0.0");
    public static readonly Counter<long> Placed = _meter.CreateCounter<long>("orders.placed", "orders");
    public static readonly Histogram<double> Total = _meter.CreateHistogram<double>("orders.total", "USD");
    public static readonly UpDownCounter<long> InFlight = _meter.CreateUpDownCounter<long>("orders.in_flight");

    public static void Record(decimal totalUsd, string customerType)
    {
        Placed.Add(1, new TagList { { "customer_type", customerType } });
        Total.Record((double)totalUsd, new TagList { { "customer_type", customerType } });
    }
}

Trace context propagation

A single user request flowing through 3 services should be one trace. OTel propagates the trace via the traceparent header (W3C TraceContext):

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
              ^   ^                                ^                ^
              v   trace-id                         span-id          flags

HttpClient instrumentation auto-adds it. ASP.NET auto-reads it and links spans. Cross-cutting magic.

Sampling

.WithTracing(t => t.SetSampler(new TraceIdRatioBasedSampler(0.1)))   // 10% of traces

Or parent-based (sample if parent sampled):

.SetSampler(new ParentBasedSampler(new TraceIdRatioBasedSampler(0.05)))

Strategies: - Head sampling: decided at start. Fast; loses interesting tail latency. - Tail sampling: decided after trace completes (in collector). Captures slow/error traces. Requires collector buffer.

Exporters

.AddOtlpExporter(o => { o.Endpoint = new Uri("http://collector:4317"); o.Protocol = OtlpExportProtocol.Grpc; })
.AddConsoleExporter()                  // dev
.AddJaegerExporter()                   // legacy; OTLP preferred
.AddPrometheusExporter()               // metrics scrape endpoint
.AddAzureMonitorTraceExporter(...)

For 2026, OTLP to a collector is the standard. Vendor-specific exporters mostly deprecated in favor of OTLP at the collector.

The OpenTelemetry Collector

[App] → OTLP → [Collector] → [Datadog | Tempo | App Insights | ...]
                              [Prometheus]
                              [Loki]

A separate process (or sidecar) that receives OTLP, processes (filter, sample, batch), and exports to one or more backends. Decouples your app from the vendor.

Resource attributes

.ConfigureResource(r => r
    .AddService("MyApp", "1.0.0", serviceInstanceId: Environment.MachineName)
    .AddAttributes(new Dictionary<string, object>
    {
        ["deployment.environment"] = "production",
        ["cloud.provider"] = "azure",
        ["cloud.region"] = "eastus"
    }))

Standard semantic conventions exist (service.name, host.name, deployment.environment). Use them — vendor dashboards rely on standard attributes.

Logs via OTel

builder.Logging.AddOpenTelemetry(o =>
{
    o.IncludeFormattedMessage = true;
    o.IncludeScopes = true;
    o.AddOtlpExporter();
});

Logs auto-correlate with the active Activity (TraceId, SpanId attached). One ID stitches logs + traces.

Semantic conventions

OTel publishes standard names for spans/metrics:

http.request.method = "GET"
http.response.status_code = 200
db.system = "postgresql"
db.name = "orders"
db.statement = "SELECT * FROM orders WHERE..."
messaging.system = "kafka"
messaging.destination.name = "orders.placed"

Use them — every backend has built-in dashboards keyed on these names.

GenAI semantic conventions

For AI/LLM apps, OTel has GenAI conventions (covered in the AI/LLM Integration section):

gen_ai.system = "openai"
gen_ai.request.model = "gpt-4o"
gen_ai.usage.input_tokens = 350
gen_ai.usage.output_tokens = 180

Microsoft.Extensions.AI emits these automatically.

Performance

OTel SDK is highly optimized — spans cost ~microseconds. The dominant cost is export (network). Use batching exporters (default).

Common pitfalls

• Activity loss in Task.Run — Activity.Current is on ExecutionContext. Task.Run captures, but be careful with manual thread pool work.
• High-cardinality metric tags (per-user-ID) — explodes time-series; cost spikes.
• Missing sampling in production — full-traffic tracing is expensive.

Code: correct vs wrong

❌ Wrong: high-cardinality metric tags

counter.Add(1, new TagList { { "user_id", userId } });   // millions of series

✅ Correct: bounded tags

counter.Add(1, new TagList { { "tier", user.Tier } });

❌ Wrong: forgetting RecordException

catch { activity?.SetStatus(ActivityStatusCode.Error); }   // no error detail in trace

✅ Correct: full record

catch (Exception ex)
{
    activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
    activity?.RecordException(ex);
    throw;
}

❌ Wrong: 100% sampling in prod

.SetSampler(new AlwaysOnSampler())

Cost explodes at scale.

✅ Correct: sample

.SetSampler(new ParentBasedSampler(new TraceIdRatioBasedSampler(0.1)))

Design patterns for this topic

Pattern 1 — "Auto-instrumentation first"

• Intent: ASP.NET + HttpClient + EF cover most.

Pattern 2 — "OTLP to a collector"

• Intent: vendor-neutral pipeline.

Pattern 3 — "Tail sampling for slow/error traces"

• Intent: keep interesting traces; drop noise.

Pattern 4 — "Resource attributes via env"

• Intent: environment-aware dashboards.

Pattern 5 — "Semantic conventions"

• Intent: vendor-portable dashboards/alerts.

Pros & cons / trade-offs

When to use / when to avoid

• Always OpenTelemetry for new services.
• Use OTLP + collector pattern.
• Always sampling in production.
• Avoid high-cardinality tags.

Interview Q&A

Q1. Three pillars of observability? Logs, metrics, traces.

Q2. What's ActivitySource? .NET primitive for emitting spans. OpenTelemetry hooks into it.

Q3. How traces propagate across services? W3C traceparent header. HttpClient + ASP.NET handle it automatically when OTel is configured.

Q4. Counter vs UpDownCounter? Counter: monotonic up. UpDownCounter: can go up or down (in-flight, queue depth).

Q5. Histogram vs Gauge? Histogram: records distribution (latency). Gauge: current value sampled.

Q6. Head vs tail sampling? Head: decided at trace start. Tail: decided after trace completes (in collector). Tail captures slow/error tail.

Q7. What's the OTel collector? Standalone process receiving OTLP, processing, exporting to vendor backends. Decouples app from vendor.

Q8. Semantic conventions? OTel-defined attribute names (http.request.method, db.system). Vendors build dashboards keyed on them.

Q9. Why OTel over vendor-specific SDKs? Vendor lock-in. Switch backends by reconfiguring collector, not rewriting code.

Q10. Cost of cardinality? Each unique tag combo = a new time series. Per-user-ID = millions of series. Expensive.

Q11. Where do logs fit? With OTel logs API, logs include active TraceId/SpanId. One ID stitches logs + traces in the backend.

Q12. Trace via Task.Run — what happens? ExecutionContext flows by default. Activity.Current propagates. Manual thread pool work can lose it.

Gotchas / common mistakes

• ⚠️ High-cardinality tags — series explosion.
• ⚠️ No sampling — cost spike.
• ⚠️ Missing RecordException — no stack in trace.
• ⚠️ Auto-instrumentation packages mismatched with .NET version.
• ⚠️ Forgetting AddSource for custom traces.
• ⚠️ No resource attributes — can't filter by environment.

Further reading

• MS Learn: OpenTelemetry in .NET
• OpenTelemetry .NET
• OTel Semantic Conventions
• Activity API