OpenTelemetry Collector

Key Points

• The OTel Collector is a vendor-neutral telemetry pipeline. Apps emit OTLP; the Collector receives, processes, exports — fan-out to one or many backends.
• Pipeline shape: receiver → processor → exporter. Configured in YAML. Three signals (traces / metrics / logs) flow through independent or shared pipelines.
• Two deployment patterns: agent (sidecar / per-host, low overhead) and gateway (centralized, does heavy work like tail-sampling, redaction, fanout). Most prod stacks use both.
• Mandatory processors in prod: memory_limiter (always first) and batch (always before exporters).
• OTTL (OpenTelemetry Transformation Language) is the in-pipeline mini-language for filtering/transforming attributes — replaces hand-coded processors.
• Tail-sampling needs sticky routing: spans of one trace must reach the same Collector. Use the loadbalancing exporter with routing_key: traceID.
• Vendor-neutral pipeline = no lock-in: rewire YAML to switch (or dual-ship) backends without touching app code. Single most valuable property of the Collector.
• Connectors (newer) bridge pipelines (e.g., spans → metrics). Distinct from processors.
• .NET integration: app uses AddOtlpExporter(); Collector listens on :4317 (gRPC) or :4318 (HTTP).

Concepts (deep dive)

What it is, what it isn't

The Collector is a stateless data pipeline for telemetry. It does not store, query, or visualize. It receives, transforms, routes. Backends (Tempo, Loki, Datadog, Application Insights, etc.) handle storage and search.

Pipeline shape

[Receivers]            [Processors]              [Exporters]
  OTLP/gRPC                memory_limiter          OTLP -> Tempo
  OTLP/HTTP        ->      batch              ->   Prometheus remote_write
  Prometheus               filter                  Datadog
  Filelog                  attributes              Loki
  Jaeger                   resource                Azure Monitor
                           tail_sampling
                           transform (OTTL)
                           redaction
                           k8sattributes

YAML defines components and wires them into named pipelines.

Minimal example

receivers:
  otlp:
    protocols:
      grpc:  { endpoint: 0.0.0.0:4317 }
      http:  { endpoint: 0.0.0.0:4318 }

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1500
    spike_limit_mib: 500
  batch:
    send_batch_size: 8192
    timeout: 5s

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls: { insecure: true }

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/tempo]

Receivers

Processors

OTTL — the transformation language

processors:
  transform:
    trace_statements:
      - context: span
        statements:
          - replace_pattern(attributes["http.url"], "secret=\\w+", "secret=REDACTED")
          - set(attributes["env"], resource.attributes["deployment.environment"])
          - delete_key(attributes, "http.user_agent") where attributes["http.route"] == "/health"

Mini-language for filtering and mutating telemetry. Replaces the attributes/filter/custom-processor pile.

Exporters

Connectors

A 2024-era addition. A connector is both an exporter (of one pipeline) and a receiver (of another). Use case: derive metrics from spans without going outside the Collector.

connectors:
  spanmetrics:
    histogram: { explicit: { buckets: [10ms, 100ms, 1s] } }

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo, spanmetrics]    # also feeds the connector
    metrics:
      receivers: [otlp, spanmetrics]          # consumes spanmetrics output
      processors: [batch]
      exporters: [prometheusremotewrite]

spanmetrics produces RED metrics (Rate / Errors / Duration) from spans automatically.

Deployment patterns

Agent (per-host / sidecar)

• One Collector per node or per pod.
• Receives from local app over loopback (zero network hop).
• Light processing: batching, basic enrichment.
• Forwards to gateway.
• Memory: ~100–300 MB.

Gateway (central pool)

• Centralized fleet behind a load balancer.
• Heavy processing: tail-sampling, redaction, fanout.
• Stateful for tail-sampling (buffered traces).
• Memory: 2–8 GB per instance.

[App] -> [Local agent] -> [LB] -> [Gateway pool] -> [Backends]
                                      |
                                      v
                              [Multiple backends]

The agent isolates the app from backend outages (local buffer survives backend down). The gateway centralizes expensive policy.

Resource detection

processors:
  resourcedetection:
    detectors: [env, system, docker, ec2, gcp, azure, k8s]
    timeout: 2s

Auto-tags every signal with host / container / cloud / pod metadata. Vendor dashboards rely on these standard attributes.

k8sattributes processor

processors:
  k8sattributes:
    auth_type: serviceAccount
    extract:
      metadata: [k8s.namespace.name, k8s.pod.name, k8s.deployment.name]
      labels:
        - { tag_name: app, key: app, from: pod }

Watches the K8s API and enriches spans with pod metadata based on source IP / pod UID. Mandatory for K8s-deployed observability.

High availability

• The Collector is stateless for non-tail-sampling pipelines. Run N replicas behind any load balancer.
• For tail-sampling: stateful (buffered traces). Use the loadbalancing exporter to hash-route by traceID:

exporters:
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp: { tls: { insecure: true } }
    resolver:
      dns:
        hostname: collector-tail.svc.cluster.local

A "front" Collector pool fans out to a "tail-sampling" pool with consistent hashing. Trace's spans always land on the same instance.

Sizing rules of thumb

Always set memory_limiter. Without it, OOMs on traffic spikes.

Vendor-neutrality — the big payoff

One pipeline, many exporters:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/tempo, datadog, azuremonitor]

Migrating from Datadog → Honeycomb? Add honeycomb exporter, run dual-ship, validate, remove Datadog. Zero app changes. This is the single best argument for OTel + Collector over vendor SDKs.

Configuration sources & secrets

exporters:
  datadog:
    api:
      key: ${env:DD_API_KEY}

Use env-var substitution. Vault/secret-manager integration via init containers or projected secrets. Never hard-code.

.NET app configuration

builder.Services.AddOpenTelemetry()
    .WithTracing(t => t
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddSource("MyApp.*")
        .AddOtlpExporter(o =>
        {
            o.Endpoint = new Uri("http://otel-collector:4317");
            o.Protocol = OtlpExportProtocol.Grpc;
        }))
    .WithMetrics(m => m.AddOtlpExporter())
    .WithLogging(l => l.AddOtlpExporter());

App is now backend-agnostic. The Collector decides where the data goes.

Observability of the Collector itself

The Collector emits its own metrics on :8888:

otelcol_receiver_accepted_spans
otelcol_processor_batch_batch_send_size
otelcol_exporter_send_failed_spans
otelcol_processor_dropped_spans

Scrape with Prometheus. Alert on dropped_spans > 0 and exporter_send_failed_spans > threshold.

Common pitfalls

• ⚠️ Forgetting memory_limiter → OOM under spike.
• ⚠️ No batch processor → many tiny exports; backend rate-limits.
• ⚠️ Round-robin LB to tail-sampling collectors → broken decisions.
• ⚠️ Treating Collector as a backend — it's a pipeline. Backends are separate.
• ⚠️ Auto-discovering K8s without RBAC — k8sattributes silently no-ops.

How it works under the hood

[App]
  | OTLP/gRPC :4317
  v
[Receiver: otlp]
  - decode protobuf
  - emit to pipeline channel
  v
[Processor chain]
  memory_limiter: check RSS; refuse if hot
        |
        v
  k8sattributes: enrich
        |
        v
  transform (OTTL): mutate
        |
        v
  tail_sampling: buffer by traceID, decide
        |
        v
  batch: coalesce up to send_batch_size or timeout
  v
[Exporter chain]
  otlp/tempo: gRPC -> Tempo
  datadog:    HTTP -> Datadog intake
  v
[Backends]

Each processor is a stage; each pipeline is independent for traces / metrics / logs (though connectors can bridge). Memory limiter and batch are special: they're effectively required.

Code: correct vs wrong

❌ Wrong: no memory_limiter

processors: [batch]

Spike → OOM → restart loop.

✅ Correct: memory_limiter first

processors: [memory_limiter, batch]

❌ Wrong: tail-sampling behind round-robin LB

[Front] -- random LB --> [Tail-1] [Tail-2] [Tail-3]

Same trace's spans split across instances → wrong decisions.

✅ Correct: hash routing

exporters:
  loadbalancing:
    routing_key: traceID
    resolver: { dns: { hostname: tail-pool } }

❌ Wrong: secrets in YAML

datadog:
  api: { key: dd-abc-123 }

✅ Correct: env substitution

datadog:
  api: { key: ${env:DD_API_KEY} }

❌ Wrong: app exports directly to vendor SDK

.AddDatadogExporter(...);   // locked in

✅ Correct: OTLP to Collector; Collector exports to vendor

.AddOtlpExporter(o => o.Endpoint = new Uri("http://otel-collector:4317"));

❌ Wrong: scraping Prometheus with no relabel

Pulling 100K series unfiltered into your trace pipeline.

✅ Correct: receiver scopes + filters

prometheus:
  config:
    scrape_configs:
      - job_name: 'app'
        metric_relabel_configs:
          - source_labels: [__name__]
            regex: 'go_.*'
            action: drop

Design patterns for this topic

Pattern 1 — "Agent + gateway"

• Intent: local resilience + central policy.

Pattern 2 — "Tail-sampling at the gateway with hash routing"

• Intent: keep errors/slow traces; consistent decisions.

Pattern 3 — "OTTL transforms, not bespoke processors"

• Intent: in-pipeline transformation as config, not code.

Pattern 4 — "Dual-ship for migration"

• Intent: add a new exporter; validate; remove the old. Zero app code change.

Pattern 5 — "Resource detection + k8sattributes"

• Intent: auto-tag every signal with cloud/k8s metadata.

Pros & cons / trade-offs

When to use / when to avoid

• Always put a Collector in front of the backend. Even one local sidecar.
• Always use memory_limiter and batch.
• Use gateway pool for tail-sampling and central redaction.
• Use OTLP from app to Collector — never vendor-native exporter.
• Avoid exporting from app directly to multiple vendors.

Interview Q&A

Q1. What's the OTel Collector? Vendor-neutral telemetry pipeline. Receivers → processors → exporters. Decouples app from backend.

Q2. Required processors? memory_limiter (first) and batch (last before exporter).

Q3. Agent vs gateway? Agent: per-host, low overhead, local buffering. Gateway: central pool, heavy processing (tail-sampling, redaction, fanout).

Q4. Tail-sampling routing? Hash by traceID via loadbalancing exporter so all spans of a trace land on one instance.

Q5. OTTL? OpenTelemetry Transformation Language — config-language for filtering/mutating telemetry inside processors.

Q6. Connectors vs processors? Processor mutates within a pipeline. Connector bridges two pipelines (e.g., span → metric).

Q7. Why OTLP from app, not vendor SDK? Vendor neutrality. Rewire YAML to switch backends; no code change.

Q8. K8s enrichment? k8sattributes processor watches K8s API and enriches signals with pod/namespace/labels.

Q9. Resource detection? Auto-tags signals with host / container / cloud metadata. Powers vendor-built dashboards.

Q10. HA strategy? Stateless replicas for non-tail pipelines. Hash-routed pool for tail-sampling.

Q11. How do you observe the Collector? It emits Prom metrics on :8888. Alert on dropped_spans and exporter_send_failed.

Q12. Migrating from Datadog to Tempo? Add otlp/tempo exporter; dual-ship; validate; remove Datadog. App untouched.

Gotchas / common mistakes

• ⚠️ No memory_limiter → OOM on spike.
• ⚠️ No batch → tiny exports; rate limits.
• ⚠️ Round-robin LB for tail-sampling → broken decisions.
• ⚠️ Hard-coded secrets in YAML — use env vars.
• ⚠️ Treating Collector as a backend — it's a pipeline.
• ⚠️ k8sattributes without RBAC — silently no-ops.
• ⚠️ Mismatched OTLP versions between SDK and Collector.

Further reading

• OTel Collector docs
• Collector Contrib repo (processors, exporters)
• OTTL spec
• .NET OTLP exporter
• K8s deployment patterns