eBPF Observability

Key Points

• eBPF = extended Berkeley Packet Filter. Tiny verified programs that run inside the Linux kernel, attached to syscalls, network events, scheduler events, function entries — without recompiling the kernel and without restarting the workload.
• The big win: zero application instrumentation. Observe traffic, latency, syscalls, file I/O, network policy across every process on the box without redeploying or adding SDKs.
• Read-only by default for observability. eBPF programs are verified for safety (bounded loops, no arbitrary memory) before kernel load.
• Tools: Pixie (auto-instrumented HTTP/gRPC for K8s; Apache 2 since New Relic acquired it), Cilium / Tetragon (network + security), Parca (continuous profiling), bpftrace / bcc (the original toolkit), Inspektor Gadget (K8s gadget framework).
• Use cases: "what's slow in this pod" without redeploy; network policy enforcement; security observability (process/file/network events); kernel-level CPU profiling (flamegraphs).
• Trade-offs: kernel version requirements (>= 5.x for the good stuff), privileged DaemonSet posture, blind spots for in-process TLS, small but non-zero overhead.
• Complement, not replace, OTel: eBPF for the syscall/network layer; OTel for the app/business layer. Together they answer "is it the kernel, the network, or my code?"
• .NET specifics: weaker uprobe support than Go/Rust because CoreCLR JIT obfuscates symbols. Pixie has partial .NET support; per-app instrumentation is still recommended for .NET deep-dive.

Concepts (deep dive)

What eBPF actually is

A small, verified VM inside the Linux kernel. You write a program (in C, Rust, or via a higher-level DSL like bpftrace), compile to eBPF bytecode, and attach it to a kernel hook:

The verifier guarantees: bounded execution, no out-of-bounds memory, no infinite loops. If it loads, it can't crash the kernel.

Why it changed observability

Before eBPF, instrumenting a syscall meant either: - Source-modifying the kernel (unrealistic). - LD_PRELOAD shims (fragile). - Or instrumenting every app individually (sprawl).

With eBPF, you load one program that watches all syscalls system-wide. No app changes. No redeploy. Read-only by default.

For a senior: eBPF is the equivalent of OTel auto-instrumentation, but at the kernel level — observing the things OTel can't see (TCP retransmits, slow disk, scheduler delays, DNS hiccups, packet drops).

What you can see (concretely)

This is the kernel's view of your workload, surfaced.

Tooling landscape

Pixie (Apache 2)

• Originally built for K8s; New Relic open-sourced it.
• Deploys a DaemonSet on every node.
• Auto-instruments: HTTP, HTTP/2, gRPC, MySQL, Postgres, Redis, DNS, Kafka — by parsing socket bytes and protocols.
• Provides PxL (Python-like) DSL for queries. Live UI for cluster traffic.
• Best at: "what's slow in this pod right now" without instrumentation.

Cilium + Tetragon

• Cilium = eBPF-native CNI for K8s. Replaces kube-proxy, enforces NetworkPolicy at the kernel.
• Tetragon = security observability — process/file/network event tracing, with rule-based alerts and (optional) inline kill.
• Used by every major hyperscaler at K8s scale.

Parca / Pyroscope

• Continuous profiling: CPU flamegraphs across the fleet, low overhead, always on.
• Sampling-based perf profiles, stitched together by service/host/pod.
• Replaces "fire up perf for 5 minutes when something is slow."

bcc / bpftrace

• The original toolkits. Brendan Gregg's collection.
• bcc: Python+C library; many ready-made tools (execsnoop, tcptop, biolatency, runqlat).
• bpftrace: AWK-like one-liners for kernel tracing.

# All TCP retransmits across the host
bpftrace -e 'tracepoint:tcp:tcp_retransmit_skb { printf("%s\n", comm); }'

# Histogram of disk I/O latency
bpftrace -e 'tracepoint:block:block_rq_complete { @ = hist(args->latency / 1000); }'

Inspektor Gadget

• K8s-native eBPF tool launcher. "Gadgets" for trace-network, trace-exec, top-file, profile-cpu, etc.
• Useful when you don't want to install a full observability stack but need ad-hoc kernel insight.

Use cases (concrete scenarios)

"Is my pod slow because of code or network?"

OTel says request took 800 ms. Pixie or bpftrace shows: 50 ms in app code, 700 ms in TCP retransmits to upstream. The kernel layer answer.

"Which syscall is dragging us?"

bcc/syscount-bpfcc shows top syscalls by latency / count system-wide. Reveals slow getaddrinfo (DNS), fsync (disk), connect (network).

Continuous profiling

Always-on flamegraphs across 1000 nodes. Hot function on canary spike → known instantly without re-deploying with a profiler.

Network policy

Cilium enforces "service-A can talk to service-B on port 5432, nothing else" via XDP/TC programs. Sub-microsecond enforcement; no iptables.

Security observability

Tetragon: alert when a pod runs curl | sh or opens /etc/shadow. Optional inline blocking (LSM hooks).

Trade-offs and limits

eBPF + OTel: the layered stack

[Business logic]      <-- OTel custom spans (orders.placed)
[Framework]           <-- OTel auto-instrumentation (ASP.NET, EF)
[Process]             <-- /proc, runtime metrics
[Syscalls / Kernel]   <-- eBPF (this)
[Network]             <-- eBPF + Cilium
[Hardware]            <-- node_exporter / hostmetrics

eBPF fills the layer between "process metrics" and "what the app told us." For a 500ms request that's 50ms code + 450ms upstream RTT, OTel knows the upstream call exists; eBPF tells you it was retransmits.

.NET specifics

• Pixie: parses HTTP/HTTPS (where ktls or pre-TLS visibility exists), works for ASP.NET Core; gRPC works partially (HTTP/2 framing parsed).
• Profiling .NET with Parca/Pyroscope: needs frame pointers (build with -p:UseFramePointers=true or runtime config) and PerfMap files (COMPlus_PerfMapEnabled=1 / DOTNET_PerfMapEnabled=1). Stack frames otherwise show as ??.
• uprobes into CoreCLR: hard. JIT-generated code has no stable symbol table; helpers like dotnet-monitor and EventPipe are usually a better choice for .NET-internal tracing.
• Practical .NET stance: use eBPF for infra/network/syscall observability across all workloads, and use dotnet-trace / dotnet-counters / OTel for in-process detail. They complement.

Production posture

• Run eBPF observability tools as DaemonSet with privileged + hostNetwork + hostPID.
• Mount /sys/kernel/debug, /proc, /sys/fs/bpf.
• Resource caps: most eBPF agents use < 200 MB and < 0.2 cores per node.
• Image signing + admission policy: this is privileged code; treat the image like a kernel module.

Status (2026 caveat)

The eBPF tooling space is fast-moving. New runtimes (eunomia, ebpf-go, libbpf), new attach types (kfunc), and consolidation among vendors are ongoing. The fundamentals (verified in-kernel programs, kprobes/uprobes, perf events) are stable; the user-space frameworks shift quarterly. Pin versions; track releases.

What eBPF is not

• Not a replacement for OTel. eBPF doesn't know your business semantics.
• Not a silver bullet for TLS payload visibility (encryption inside the process is opaque).
• Not "free": small overhead, real privilege footprint.
• Not a Windows technology — Linux-only at scale (Windows has eBPF-for-Windows, early days).

How it works under the hood

[Userland: Pixie / bpftrace / Tetragon agent]
       |
       | (1) compile C/DSL to eBPF bytecode
       | (2) bpf() syscall: BPF_PROG_LOAD
       v
[Kernel: verifier]
       - bounded loops?
       - memory accesses safe?
       - allowed helpers only?
       |
       | OK
       v
[Kernel: program loaded]
       - JIT'd to native (x86_64, arm64)
       - attached to hook (kprobe / tracepoint / XDP / uprobe / ...)
       v
[Event fires]
       - hook invokes program
       - program reads context, writes to BPF map (hash, ringbuffer, perf buffer)
       v
[Userland: agent]
       - reads from map / ringbuffer
       - aggregates / ships to backend (Prom / OTLP / vendor)

The verifier is the magic — it enforces safety at load time, so failures are at deploy, not at runtime in the kernel hot path.

Maps

eBPF programs talk to userland through maps: hash, array, ringbuffer, perf buffer, LRU, percpu. Userland reads; the kernel program writes. This separation lets you accumulate billions of events as histograms in-kernel without crossing the user/kernel boundary per event.

Code: correct vs wrong

❌ Wrong: deploying eBPF tooling without kernel version check

# DaemonSet on a 4.4 kernel

Most modern features fail to load; tool degrades silently.

✅ Correct: pin minimum kernel; use BTF when available

nodeSelector:
  kernel.version: ">=5.4"

Or rely on CO-RE (Compile Once, Run Everywhere) builds via libbpf+BTF.

❌ Wrong: expecting eBPF to read TLS payload

# Tries to capture HTTPS body

Sees ciphertext.

✅ Correct: uprobe into the TLS library or terminate TLS at a sidecar

// Uprobe on SSL_read in libssl
SEC("uprobe/SSL_read")
int BPF_KPROBE(uprobe_ssl_read, void *ssl, void *buf, int num) { ... }

Pixie does this for OpenSSL/BoringSSL automatically.

❌ Wrong: profiling .NET without PerfMap

Stack: 0x7fff... ??  ??  ??

✅ Correct: enable PerfMap

DOTNET_PerfMapEnabled=1 dotnet MyApp.dll

Plus frame pointers; stacks resolve to method names.

❌ Wrong: privileged DaemonSet without admission policy

Anyone with kubectl apply can ship a kernel rootkit.

✅ Correct: signed images + Kyverno/Gatekeeper policy

# Only allow privileged DaemonSets from signed registry

Design patterns for this topic

Pattern 1 — "eBPF for the kernel layer; OTel for the app layer"

• Intent: complementary stacks; each answers what the other can't.

Pattern 2 — "Continuous profiling with Parca/Pyroscope"

• Intent: always-on flamegraphs; instant hot-function diagnosis.

Pattern 3 — "Pixie for K8s zero-instrument observability"

• Intent: see HTTP/gRPC/SQL traffic without redeploying.

Pattern 4 — "Cilium for kernel-level network policy"

• Intent: sub-microsecond enforcement; no iptables sprawl.

Pattern 5 — "Tetragon for security observability"

• Intent: rule-based process/file/network alerts; optional inline block.

Pros & cons / trade-offs

When to use / when to avoid

• Use for "is it the kernel/network or my code" questions.
• Use for continuous profiling across the fleet.
• Use Cilium/Tetragon for K8s network and security observability.
• Avoid as the only observability stack — pair with OTel.
• Avoid expecting TLS-payload visibility without uprobes.
• Avoid on legacy kernels (< 4.18) — most useful features missing.

Interview Q&A

Q1. What is eBPF? Verified programs that run in the Linux kernel, attached to hooks (kprobes, tracepoints, XDP). Observe and (with care) modify behavior without recompiling the kernel.

Q2. Why is the verifier important? It guarantees safety: bounded execution, safe memory access. eBPF can't crash the kernel.

Q3. eBPF vs OTel? eBPF: kernel/syscall/network layer, zero app changes. OTel: app/business layer with semantics. Use both.

Q4. Pixie's value prop? Auto-instrument HTTP/gRPC/SQL/Redis in K8s without app changes. Apache 2.

Q5. Cilium vs kube-proxy? Cilium replaces kube-proxy with eBPF-native programs. Faster, scales better, network policy enforcement at the kernel.

Q6. Tetragon use case? Security observability — alert on process/file/network events with rule policies. Inline blocking via LSM hooks.

Q7. eBPF and TLS? TLS-in-process is opaque (ciphertext at the socket). Workarounds: uprobes into OpenSSL/BoringSSL, ktls, sidecar TLS termination.

Q8. .NET specifics? Profiling works with PerfMap + frame pointers. uprobes into CoreCLR are hard because of JIT. Pixie has partial coverage.

Q9. Continuous profiling overhead? Sub-1% typical (sampling at 99 Hz). Always-on flamegraphs.

Q10. Production posture for eBPF agents? Privileged DaemonSet; signed images; admission policy. Treat it like a kernel module.

Q11. Kernel version requirements? 4.18 minimum; 5.4+ for BTF/CO-RE/ring buffer. Modern managed K8s usually fine.

Q12. Map types? Hash, array, ring buffer, perf event array, LRU, percpu. The kernel/user-space data interface.

Gotchas / common mistakes

• ⚠️ Old kernels (< 4.18) — most features missing.
• ⚠️ TLS-in-process blind spot — see ciphertext only.
• ⚠️ Privileged DaemonSet without signing/admission policy.
• ⚠️ JIT runtimes (.NET, Java) need PerfMap / frame pointers for usable stacks.
• ⚠️ Treating eBPF as a replacement for OTel — it's complementary, not a substitute.
• ⚠️ Tracepoint overuse — non-trivial overhead at high rates.
• ⚠️ Windows assumption — eBPF-for-Windows is early; production stack is Linux.

Further reading

• eBPF.io
• Cilium docs
• Tetragon
• Pixie
• Parca and Pyroscope
• Brendan Gregg: BPF Performance Tools
• .NET PerfMap docs