Logging at Scale

Key Points

• Logs are the most expensive pillar at scale: ingestion volume × retention × indexing tier. A mid-size service can spend more on logs than on compute.
• Cost math: 100K req/s × 5 lines × 500 B = 250 MB/s = ~21 TB/day. At $2–5/GB ingest = $40–100K/day. Logs dominate budgets if you don't tier and sample.
• Tiering (hot / warm / cold) is the single biggest lever. Hot is expensive and searchable; cold is cheap and slow.
• Cardinality kills: high-cardinality fields like userId indexed as queryable explode storage costs. Index fewer fields; keep the rest as raw payload.
• Sampling logs is different from sampling traces: rule-based, not statistical. Drop healthy 200s; keep all errors.
• Pipeline reliability: in-memory buffering loses data on crash. Disk-backed buffers (Fluent Bit, OTel Collector with file storage) survive restarts.
• PII redaction belongs in the pipeline, not in app code. Devs forget; pipelines don't.
• .NET specifics: ILogger<T> is the abstraction; LoggerMessage source-gen for hot paths; OTLP log exporter for vendor-neutral shipping.

Concepts (deep dive)

The cost surface

Cost = ingest_rate × bytes_per_event × retention_days × tier_multiplier
       + index_size × indexed_fields_factor
       + query_volume × scan_size

Three knobs: how much you send, how long you keep it, what you index. Most teams discover logging cost only after the bill spikes.

Worked example

A single chatty service can spend more on its logs than its EC2 bill. Senior decisions live here.

Tiering

HOT  (0–7 days)    SSD-backed, fully indexed, sub-second search.   $$$
WARM (8–30 days)   Indexed but slower; common cloud blob.            $$
COLD (>30 days)    Archive (S3 Glacier / ADLS cool); restore-on-need. $

Strategies: - Hot: live operational use. - Warm: post-incident forensic search. - Cold: compliance / audit. Often >90% of total volume but <5% of cost when archived.

Most platforms (Splunk SmartStore, Datadog Flex Logs, Elasticsearch ILM, Loki) support automatic tiering by index age.

Structured vs unstructured

Unstructured ("User 42 placed order 7 for $99"): - Cheap to ingest (1 string). - Expensive to query (regex/parse on read). - Bad for aggregation.

Structured JSON ({"user":42,"order":7,"total":99}): - More to ingest (field tags repeat). - Cheap to query (typed fields, indexed). - Aggregations are first-class.

Recommendation: structured for everything. Modern ingest pipelines compress field names; the cost delta is small. The query advantage is enormous.

Indexing tiers

A typical mistake: indexing every field "just in case." Each indexed field multiplies storage 1.5–3×.

Cardinality kills

A field's cardinality = distinct values. level has ~5 (Trace/Debug/Info/Warn/Error). userId has millions.

Indexed userId: - Index = inverted-index entry per distinct value. - 10M users × 30 days × per-bucket overhead = explosion.

Rule of thumb: don't index any field with cardinality > ~10K unless you specifically need it for queries. Keep userId as a payload field — searchable via full-text or a secondary lookup.

Sampling vs filtering

Anti-pattern: "we sample 5% of all logs." You'll lose most errors (errors are rare) and keep most noise.

The shipping pipeline

[App]
   |   (ILogger -> sink)
   v
[Local agent: Fluent Bit / OTel Collector / Vector]
   |   - parse (if needed)
   |   - enrich (add k8s/cloud metadata)
   |   - filter / drop
   |   - redact (PII)
   |   - sample
   |   - buffer (memory + disk)
   v
[Backend: Elastic / Loki / Datadog / Splunk / New Relic]

Reliability

For high-stakes services, write logs to a local file and ship via agent. The file is the truth source; the agent is the delivery mechanism.

Back-pressure

When the backend slows, the agent must: - Buffer to disk up to a cap. - Drop the oldest when cap exceeded (or block — but blocking your app is worse than losing logs). - Surface metrics: agent_buffer_size, agent_dropped_records. Alert on them.

Log levels at scale

A common smell: 50 INFO lines per request. At 100K req/s that's 5M lines/s. Either lower default level to Warn for chatty namespaces or drop them in the pipeline.

builder.Logging.AddFilter("Microsoft.AspNetCore.Hosting.Diagnostics", LogLevel.Warning);
builder.Logging.AddFilter("Microsoft.EntityFrameworkCore.Database.Command", LogLevel.Warning);

Sensitive data — pipeline redaction

# OTel Collector
processors:
  redaction:
    allow_all_keys: true
    blocked_values:
      - '\b\d{16}\b'                  # credit card
      - '\b\d{3}-\d{2}-\d{4}\b'       # SSN
    summary: silent

Or attributes processor with delete actions. Don't trust developers to remember [Redacted] in every log line — they'll forget once and ship a regulator-fine.

ILogger and structured logging in .NET

public class OrderHandler(ILogger<OrderHandler> log)
{
    public async Task HandleAsync(PlaceOrder cmd)
    {
        log.LogInformation(
            "Order {OrderId} placed by {CustomerId} for {Total:C}",
            cmd.OrderId, cmd.CustomerId, cmd.Total);
    }
}

Properties become structured fields when sink is JSON-aware. Serilog, OTel logs, NLog all support this.

LoggerMessage source-gen (hot paths)

public partial class OrderHandler
{
    [LoggerMessage(Level = LogLevel.Information,
        Message = "Order {OrderId} placed by {CustomerId} for {Total:C}")]
    static partial void LogOrderPlaced(ILogger l, Guid orderId, Guid customerId, decimal total);
}

Compile-time-generated method — no allocations, no boxing of value types, no format-string parsing per call. ~10× faster than log.LogInformation(...) on hot paths.

OTLP log exporter

Send logs through the OTel Collector for the same vendor-neutral pipeline as traces and metrics:

builder.Logging.AddOpenTelemetry(o =>
{
    o.IncludeScopes = true;
    o.IncludeFormattedMessage = true;
    o.AddOtlpExporter(opts => opts.Endpoint = new Uri("http://otel-collector:4317"));
});

The Collector then redacts, samples, fans out to multiple backends, and survives backend outages with disk buffering. Strongly recommended in 2026.

Cost dashboards

Operational hygiene every senior owns: - $ per service per day — who's spending what. - GB ingested per service per hour — spike detection. - % indexed fields — drift over time. - Top 20 chatty loggers — namespace ranking. - Drop rate — pipeline back-pressure.

Datadog has Usage tab; Splunk has license usage; Elastic has ILM stats; build your own panel from Collector metrics.

Comparing backends (one paragraph each)

• Elasticsearch / OpenSearch: heavy indexing; powerful full-text; ops cost is real. Self-host = cheap volume / expensive ops; hosted = inverse.
• Grafana Loki: indexes only labels; payload stored cheap. Cardinality of labels is the cost lever, not volume. Excellent for K8s log labels.
• Datadog Logs: rich UI, expensive. "Flex Logs" for warm tier helps. Easy to overspend.
• Splunk: enterprise standard; license per GB ingested historically; SmartStore tiers to S3.
• New Relic Logs: lower per-GB; integrated with their APM.
• Azure Monitor Logs / Log Analytics: KQL queries; bundled with App Insights; reasonable for Azure-heavy shops; commitment tiers.
• AWS CloudWatch Logs + Insights: default for AWS shops; expensive at scale; use Firehose to S3 for cold.

For .NET on Azure: Application Insights (Logs workspace) + a cold-tier ADLS archive is the pragmatic default.

How it works under the hood

[App]
  |
  | ILogger.LogInformation(template, args)
  |
  v
[ILoggerProvider chain]   (Console, OTel, Serilog sink, ...)
  |
  v
[Sink]
  - serialize (JSON / OTLP / line)
  - enrich (scopes, traceId, machine)
  - emit
  |
  v
[Local agent]
  - tail file or receive OTLP
  - parse / enrich (k8s pod, container, region)
  - filter / sample
  - redact
  - batch
  - buffer (mem / disk)
  - export
  |
  v
[Backend]
  - tier hot/warm/cold (ILM)
  - index configured fields
  - retain per policy
  - serve queries

Performance bottlenecks usually live in: (1) synchronous Console sink in prod (slow!), (2) string.Format in hot paths (use source-gen), (3) excessive indexed fields (vendor-side cost).

Code: correct vs wrong

❌ Wrong: string-interpolated message

log.LogInformation($"Order {orderId} for {total}");

Loses structured fields. Searches become regex. Source-gen can't help.

✅ Correct: template + args

log.LogInformation("Order {OrderId} for {Total}", orderId, total);

❌ Wrong: high-cardinality indexed field

// shipped to backend with indexing on "userId"
log.LogInformation("Login {UserId}", userId);   // 10M users => index explosion

✅ Correct: keep as payload, not index

# pipeline config: don't promote userId to indexed metadata
attributes:
  actions:
    - key: user.id
      action: hash               # or move to non-indexed payload

❌ Wrong: PII in logs unredacted

log.LogInformation("User {Email} {Ssn}", email, ssn);

✅ Correct: redact in pipeline (defense in depth)

processors:
  redaction:
    blocked_values:
      - '\b\d{3}-\d{2}-\d{4}\b'   # SSN

And avoid logging the field at all.

❌ Wrong: hot-path LogInformation with allocations

// 1M req/s; each call boxes value types and parses template
log.LogInformation("Trade {Id} {Px} {Qty}", id, px, qty);

✅ Correct: LoggerMessage source-gen

[LoggerMessage(Level = LogLevel.Information,
    Message = "Trade {Id} {Px} {Qty}")]
static partial void LogTrade(ILogger l, Guid id, decimal px, int qty);

❌ Wrong: in-memory buffer in agent for critical logs

Crash → loss.

✅ Correct: disk-backed buffer

extensions:
  file_storage:
    directory: /var/lib/otelcol
exporters:
  otlp:
    sending_queue: { storage: file_storage }

Design patterns for this topic

Pattern 1 — "Tier hot / warm / cold"

• Intent: keep recent searchable; archive the rest cheaply.

Pattern 2 — "Index few, store many"

• Intent: searchable on bounded fields; payload cheap.

Pattern 3 — "Pipeline-side redaction"

• Intent: defense-in-depth for PII.

Pattern 4 — "Disk-backed shipping buffer"

• Intent: survive restarts and backend outages.

Pattern 5 — "Source-gen on hot paths"

• Intent: zero-allocation, fast logging.

Pros & cons / trade-offs

When to use / when to avoid

• Always ship via an agent + disk buffer.
• Always redact PII in pipeline.
• Always tier; never keep 90 days hot.
• Use source-gen LoggerMessage for hot paths.
• Avoid indexing high-cardinality fields.
• Avoid Debug/Info chatter at full volume in prod.

Interview Q&A

Q1. Why is logging the most expensive pillar? Volume × retention × indexing. Easily $10–100K/day at scale. Compute is often cheaper.

Q2. Hot/warm/cold tiers? Hot: indexed, fast, expensive. Warm: indexed slower, cheaper. Cold: archive, cheap, slow restore.

Q3. Cardinality kills — meaning? High-cardinality fields (per-user) indexed → index size explodes. Keep them in payload, not indexed metadata.

Q4. Sampling logs vs traces? Logs: rule-based (drop healthy 2xx, keep errors). Traces: probabilistic + tail-sampling. Different shapes.

Q5. Why disk-backed buffering? Crash safety. In-memory loses on restart; disk buffer replays.

Q6. Where to redact PII? Pipeline. Devs forget; pipelines don't. Defense in depth — also avoid logging the field.

Q7. LoggerMessage source-gen? Compile-time method, no boxing, no template parsing per call. ~10× faster on hot paths.

Q8. ELK vs Loki vs Datadog? ELK: heavy index, full-text. Loki: cheap volume, label-based. Datadog: rich UI, expensive. Pick by query patterns and budget.

Q9. How to log structured in .NET? log.LogInformation("{Field} {Field2}", a, b). Sinks emit fields as structured. Don't interpolate.

Q10. OTLP for logs? Yes — OTel logs API + OTLP exporter. Same pipeline as traces and metrics. Vendor-neutral.

Q11. What's a chatty logger? A namespace emitting orders of magnitude more than business value. Filter via AddFilter("Namespace", LogLevel.Warning).

Q12. Should I sample 5% of all logs uniformly? No — you'll keep mostly noise and lose errors. Keep all errors; sample/drop healthy noise.

Gotchas / common mistakes

• ⚠️ Indexing every field — cost explosion.
• ⚠️ High-cardinality indexed fields (userId, requestId) — index blows up.
• ⚠️ In-memory buffer only — crash = data loss.
• ⚠️ PII in logs unredacted — regulatory risk.
• ⚠️ Console sink in prod — synchronous, slow.
• ⚠️ No log level tuning — Microsoft.* INFO drowns signal.
• ⚠️ String-interpolated messages — kills structured fields.

Further reading

• .NET High-performance logging with LoggerMessage source-gen
• OpenTelemetry Logs spec
• Grafana Loki: Logs as labels
• Datadog Flex Logs
• Fluent Bit