OTel Exporters

Key Points

• Exporter = the SDK component that ships telemetry off the process. Multiple may run in parallel (console + OTLP, etc.).
• OTLP (OpenTelemetry Protocol) is the modern standard — gRPC or HTTP/Protobuf. Use it; avoid vendor-specific exporters.
• The collector pattern: app → OTLP → collector → (vendor backend, Prometheus, Loki, Tempo, ...). Decouples app from vendor.
• Prometheus scrapes metrics — pull model. Different from OTLP (push). AddPrometheusExporter exposes /metrics.
• Batching: default. Tune batch size, queue, timeout for throughput vs latency.

Concepts (deep dive)

OTLP exporter

.AddOtlpExporter(o =>
{
    o.Endpoint = new Uri("http://otel-collector:4317");
    o.Protocol = OtlpExportProtocol.Grpc;       // or HttpProtobuf
    o.Headers = "Authorization=Bearer ...";
    o.TimeoutMilliseconds = 10_000;
})

gRPC is more efficient; HTTP is firewall-friendly.

Console exporter (dev)

.AddConsoleExporter()

Dumps spans/metrics to stdout. Dev only; expensive at volume.

Prometheus exporter

.WithMetrics(m => m
    .AddPrometheusExporter()
    /* ... */);

app.MapPrometheusScrapingEndpoint();   // exposes /metrics

Prometheus scrapes the endpoint. Pull model — Prometheus polls every N seconds.

For traces, Prometheus doesn't apply — use Tempo/Jaeger via OTLP.

Vendor exporters

.AddAzureMonitorTraceExporter(o => o.ConnectionString = "...")
.AddDatadogExporter(o => o.AgentEndpoint = ...)
.AddNewRelicTraceExporter(o => o.ApiKey = ...)

These are mostly legacy. The modern pattern: emit OTLP, let the collector route to the vendor.

Batch vs simple

.AddOtlpExporter()                                                   // BatchExportProcessor (default)
.AddOtlpExporter(o => /* ... */).AddProcessor<SimpleExportProcessor<...>>()   // not recommended

Simple exports per-span (latency-sensitive but expensive). Batch buffers (efficient, default).

Tuning:

.AddOtlpExporter((o, batch) =>
{
    batch.MaxQueueSize = 2048;
    batch.MaxExportBatchSize = 512;
    batch.ScheduledDelayMilliseconds = 5000;
    batch.ExporterTimeoutMilliseconds = 30_000;
})

The collector

A standalone process that receives OTLP, processes (filter, sample, attribute manipulation), and exports to multiple backends.

# otel-collector.yaml
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  batch:
  memory_limiter: { check_interval: 1s, limit_mib: 400 }
  tail_sampling:
    policies:
      - { name: errors, type: status_code, status_code: { status_codes: [ERROR] } }

exporters:
  otlphttp/datadog: { endpoint: https://api.datadoghq.com, headers: { DD-API-KEY: "..." } }
  prometheus: { endpoint: 0.0.0.0:8889 }
  loki: { endpoint: http://loki:3100/loki/api/v1/push }

service:
  pipelines:
    traces: { receivers: [otlp], processors: [batch, memory_limiter, tail_sampling], exporters: [otlphttp/datadog] }
    metrics: { receivers: [otlp], processors: [batch], exporters: [prometheus] }
    logs:    { receivers: [otlp], processors: [batch], exporters: [loki] }

Deploy as a sidecar (per pod), DaemonSet (per node), or central deployment.

Where to run the collector

• Sidecar: low latency, scales with app, more resource cost.
• DaemonSet: shared per node — most common.
• Centralized: easy to manage; a network hop.

Resource attribution at the collector

processors:
  attributes:
    actions:
      - { key: deployment.environment, value: production, action: insert }

Add or modify resource attributes centrally, e.g., to standardize across services.

Metrics: cumulative vs delta

OTLP can transmit: - Cumulative — total since process start. - Delta — change since last export.

Prometheus prefers cumulative. Datadog/New Relic accept both. Set via SDK or collector.

Pull vs push

For Prometheus: app exposes /metrics endpoint; Prometheus scrapes. For everything else: app pushes.

You can mix — use OTLP for traces, Prometheus for metrics, both via collector.

Authentication

.AddOtlpExporter(o =>
{
    o.Headers = $"Authorization=Bearer {token}";
})

Or mTLS for gRPC. Most cloud vendors document the right header.

Failure handling

o.ExportProcessorType = ExportProcessorType.Batch;
// On export failure: spans are dropped (after retry within timeout)

OTel SDK retries within the export timeout, then drops to avoid memory pressure. Persistent retry queues are a collector feature.

Compression

OTLP supports gzip:

o.HttpClientFactory = () => /* with handler that adds Accept-Encoding: gzip */;

For high-volume telemetry, compression saves significant bandwidth.

Multiple exporters

.AddOtlpExporter(o => o.Endpoint = new Uri(prodUri))   // primary
.AddConsoleExporter()                                    // also dump locally

All exporters get all spans. Useful for dev (console + OTLP).

Cost levers

1. Sample at source.
2. Tail-sample at collector to keep interesting traces.
3. Drop high-cardinality dimensions.
4. Compress OTLP payloads.
5. Reduce auto-instrumentation noise (e.g., health-check requests).

Code: correct vs wrong

❌ Wrong: vendor-specific exporter from app

.AddDatadogExporter(o => o.ApiKey = ...)

Couples app to vendor. Switching means redeploying.

✅ Correct: OTLP + collector

.AddOtlpExporter(o => o.Endpoint = new Uri("http://collector:4317"))
// Collector configured to export to whichever backend

❌ Wrong: simple exporter in production

.AddProcessor<SimpleExportProcessor<...>>()   // export per span; high latency

✅ Correct: batch (default)

.AddOtlpExporter()

❌ Wrong: no compression at high volume

o.Protocol = OtlpExportProtocol.HttpProtobuf;   // raw, uncompressed

✅ Correct: gzip via HTTP exporter

Configure compression in the HTTP client.

Design patterns for this topic

Pattern 1 — "App → OTLP → collector → vendors"

• Intent: vendor-neutral pipeline.

Pattern 2 — "Sidecar/DaemonSet collector"

• Intent: low latency; easy upgrades.

Pattern 3 — "Tail sampling at collector"

• Intent: keep value, cap cost.

Pattern 4 — "Mix push + Prometheus pull"

• Intent: some metrics scraped, traces pushed.

Pattern 5 — "Multiple exporters in dev"

• Intent: OTLP to dev collector + console for visibility.

Pros & cons / trade-offs

When to use / when to avoid

• Use OTLP everywhere.
• Use collector pattern in production.
• Use Prometheus exporter for metrics if your stack is Prometheus-native.
• Avoid vendor-specific exporters in app code.
• Avoid simple exporter in production.

Interview Q&A

Q1. OTLP gRPC vs HTTP? gRPC: efficient, port 4317. HTTP/Protobuf: port 4318, firewall-friendly.

Q2. Why the collector pattern? Decouples app from vendor; centralizes processing (sampling, attribute mutation); routes to multiple backends.

Q3. Pull (Prometheus) vs push (OTLP)? Prometheus scrapes. OTLP pushes. Different models. Prometheus best for metrics in scrape-friendly environments.

Q4. Where to deploy the collector? Sidecar (per pod), DaemonSet (per node), or central. DaemonSet is most common.

Q5. Cumulative vs delta metrics? Cumulative: since process start. Delta: change since last export. Prometheus prefers cumulative.

Q6. Batch vs simple processor? Batch (default) buffers exports; efficient. Simple exports each span; expensive.

Q7. Multiple exporters? Yes — register multiple. Each gets all data.

Q8. Auth on OTLP? Headers (Bearer token) for gRPC/HTTP. mTLS for gRPC.

Q9. What if collector is down? SDK retries within timeout, then drops. Use a collector with persistent queue for critical telemetry.

Q10. Compression? gzip on HTTP OTLP. Significant bandwidth savings at scale.

Q11. Cost reduction levers? Sample at source; tail-sample at collector; drop high-cardinality; compress; reduce auto-instr noise.

Q12. Vendor-specific exporters — when? Only if no OTLP support in vendor. Modern Datadog/New Relic/App Insights all accept OTLP — don't use legacy SDKs.

Gotchas / common mistakes

• ⚠️ Vendor exporter in app — coupling.
• ⚠️ Simple processor in prod — latency.
• ⚠️ No collector in production — fragile telemetry.
• ⚠️ No compression at high volume.
• ⚠️ Mixing cumulative/delta carelessly.

Further reading

• OpenTelemetry Collector docs
• MS Learn: OTel exporters
• Prometheus pull model
• OTLP spec