Kafka — Deep

This file deepens Brokers Comparison. Refer to that for Kafka's place among other brokers; this file covers the senior-level operational and design concerns specific to Kafka.

Key Points

• Consumer groups + partitions are the core scaling unit. Rule: at most one consumer per partition per group. Partitions ≥ desired parallelism.
• Partition assignment matters: Range, RoundRobin, Sticky, Cooperative-Sticky (default in modern clients). Cooperative incremental rebalances avoid stop-the-world.
• Exactly-once in Kafka = idempotent producer + transactional API + read-committed consumer. Real, but pays ~10-20% throughput.
• KRaft replaced ZooKeeper. Operationally simpler: one binary, one quorum, faster metadata. New deployments are KRaft-only.
• Retention: by time (retention.ms), by size (retention.bytes), or log compaction (keep latest per key). Compaction = "current state in a log".
• Schema Registry (Confluent / Apicurio) + Avro/Protobuf with evolution rules (backward / forward / full) is mandatory for any non-trivial event platform.
• Tiered storage keeps hot segments on broker disks and offloads warm to S3/Blob; long retention without growing brokers.
• Performance levers: batch.size, linger.ms, compression.type (lz4/zstd preferred), acks=all, min.insync.replicas=2, partition count.
• .NET client: Confluent.Kafka (librdkafka under the hood). Mature, performant, well-maintained.
• Operational signals: consumer lag, ISR shrink, under-replicated partitions, log-end-offset growth, request latency p99.

Concepts (deep dive)

Topics, partitions, consumer groups

Topic "orders" (3 partitions, replication-factor 3)

       P0 ────────────────────────────────► (offsets 0..N)
       P1 ────────────────────────────────►
       P2 ────────────────────────────────►

Consumer Group "billing" (3 instances)
   instance-A → P0
   instance-B → P1
   instance-C → P2

Consumer Group "shipping" (2 instances)
   instance-X → P0, P1
   instance-Y → P2

Each consumer group tracks its own offsets independently. Two groups consume the same topic at their own pace.

The "one consumer per partition" rule

Within a group, a partition is assigned to exactly one consumer. Adding more consumers than partitions = the extras sit idle. Therefore:

max parallelism within a group = partition count

If your topic has 4 partitions, you cannot scale a single consumer group beyond 4 instances. Plan partition counts for peak load, not current. You can always add partitions, but it disrupts key-based ordering for the affected keys.

Partition assignment strategies

The group coordinator (a broker) assigns partitions to consumers using a strategy negotiated by the clients.

Eager vs incremental cooperative rebalances

Eager (legacy):
   1. All consumers stop, revoke ALL partitions.
   2. Coordinator assigns.
   3. All consumers resume.
   Stop-the-world for the whole group.

Incremental Cooperative:
   1. Only partitions that need to move are revoked.
   2. Untouched partitions keep flowing.
   3. New owners pick up the moved ones.
   Significantly less disruption; lower lag spikes.

Use CooperativeStickyAssignor everywhere unless you have a legacy reason not to.

Exactly-once semantics (EOS)

Kafka EOS combines:

1. Idempotent producer (enable.idempotence=true) — broker dedupes producer retries on (PID, sequence number) per partition.
2. Transactional API — a producer can write to multiple partitions and commit consumer offsets atomically.
3. isolation.level=read_committed — consumers skip records from aborted transactions.

Read from input topic ─┐
                       ├── transaction begin
Process               │
Write to output topic  │
Commit consumer offset │
                       └── transaction commit (atomic)

This gives "read-process-write" exactly-once within Kafka. External side effects (HTTP calls, DB writes) are NOT included unless you also do idempotency keys or use Kafka Connect with EOS-aware sinks.

Cost: ~10-20% throughput vs at-least-once with idempotent producer. Use only when needed (financial, deduplication-sensitive).

KRaft (no more ZooKeeper)

Pre-KRaft:                After KRaft:
─────────                 ───────────
ZooKeeper ensemble        Kafka controllers (Raft)
+ Kafka brokers           + Kafka brokers
Two systems to operate    One system
Slow metadata at scale    Fast metadata; snapshots; no external dep

KRaft is mandatory in new clusters. Operationally: separate controller nodes (or combined with brokers in dev), metadata.log.dir as a Raft log. Big win for cluster size, recovery time, and ops overhead.

Retention vs log compaction

Two retention modes:

1. Time/size retention (default for most topics)
   retention.ms = 7d
   retention.bytes = 1GB
   Old segments deleted; old data gone.

2. Log compaction (per-key "current state" log)
   cleanup.policy = compact
   Keep the latest record per key forever (or until tombstoned).
   Used for: changelog topics, materialized state, "table" topics.

You can combine: cleanup.policy=compact,delete to keep the latest per key but also expire old keys.

Compaction example

Records (key, value):
  (k1, v1)   (k2, v2)   (k1, v3)   (k3, v4)   (k2, v5)

After compaction (latest per key):
  (k1, v3)   (k3, v4)   (k2, v5)

Tombstones (null value) eventually delete the key. Used to model entity-state-as-log (Kafka Streams KTable).

Retention sizing math

events/sec × avg-bytes × retention-seconds × replication-factor
= storage required

Example: 50K events/sec × 1KB × 7 days × 3 RF
       = 50_000 × 1024 × 7 × 86400 × 3
       ≈ 90 TB

Plus headroom (30%) and indexes (~5%).

Forgetting to multiply by replication factor is the #1 sizing mistake.

Schema Registry

Schemas live in a registry; messages carry only the schema ID + payload. The registry enforces compatibility on schema upload.

[Producer] ── Avro/Protobuf payload + schema ID ──→ [Kafka]
                                                       │
[Consumer] ◄─────────── pulls schema by ID ─────── [Registry]

Compatibility modes:

Backward is the most common — you upgrade consumers first, then producers.

Avro vs JSON Schema vs Protobuf

For Kafka-only pipelines, Avro is the historical default. For polyglot ecosystems with gRPC, Protobuf often wins.

Tiered storage

Hot tier:   broker local disks (recent segments, fast)
Warm tier:  object storage (S3, Azure Blob) (older segments, cheap)

Brokers serve from hot directly; warm fetches transparent to consumers.

Effect: long retention (months / years) without scaling broker disks. Available in Confluent Platform / Cloud and recent Apache Kafka (KIP-405).

Kafka Connect, Streams, ksqlDB, Flink

For .NET, Kafka Streams isn't native; use Flink (with Java) or .NET-side processing with Confluent.Kafka + your own state store.

.NET client: Confluent.Kafka

var config = new ProducerConfig
{
    BootstrapServers = "kafka:9092",
    EnableIdempotence = true,
    Acks = Acks.All,
    CompressionType = CompressionType.Lz4,
    LingerMs = 10,
    BatchSize = 64 * 1024
};

using var producer = new ProducerBuilder<string, string>(config).Build();
await producer.ProduceAsync("orders",
    new Message<string, string> { Key = orderId, Value = json });

var cConfig = new ConsumerConfig
{
    BootstrapServers = "kafka:9092",
    GroupId = "billing",
    EnableAutoCommit = false,
    AutoOffsetReset = AutoOffsetReset.Earliest,
    PartitionAssignmentStrategy = PartitionAssignmentStrategy.CooperativeSticky,
    IsolationLevel = IsolationLevel.ReadCommitted
};

using var consumer = new ConsumerBuilder<string, string>(cConfig).Build();
consumer.Subscribe("orders");

while (!ct.IsCancellationRequested)
{
    var r = consumer.Consume(ct);
    await ProcessAsync(r.Message);
    consumer.Commit(r);   // commit only after successful processing
}

Wraps librdkafka in C. High performance; well-tested. Don't use it from a single-threaded context if you need parallel partitions — give each consumer its own thread, or use consumer.Poll from a dedicated worker.

Performance levers

Operational signals to watch

consumer lag           = log-end-offset - committed-offset
ISR shrink             = a replica fell behind/dead
under-replicated parts = partitions where ISR < replication factor
request latency p99    = broker health
log size growth rate   = retention sizing sanity check

Monitor with Kafka Exporter + Prometheus, Confluent Control Center, or Datadog Kafka integration.

How it works under the hood

A produce request flow with acks=all and EOS:

1. Producer batches records by partition.
2. (If transactional) producer writes BEGIN to transaction log.
3. Producer sends batch to partition leader.
4. Leader appends to its local log; assigns offset.
5. Followers fetch and replicate; leader waits for min.insync.replicas.
6. Leader ACKs producer.
7. Producer commits transaction (atomic across partitions).
8. Consumers with read_committed see records only after commit marker.

Rebalance flow (cooperative):

1. New consumer joins group → sends JoinGroup.
2. Coordinator (broker) selects leader; computes new assignment.
3. Coordinator returns "revoke these specific partitions" to losers.
4. Losers commit offsets, revoke; rest of group unaffected.
5. Coordinator returns "you also own these now" to gainers.
6. Gainers fetch from last committed offset.

Code: correct vs wrong

❌ Wrong: producer with no key + need for ordering

await producer.ProduceAsync("orders",
    new Message<string, string> { Value = json });   // null key → random partition

Order events for the same customer scatter; downstream sees them out of order.

✅ Correct: key by ordering domain

await producer.ProduceAsync("orders",
    new Message<string, string> { Key = customerId, Value = json });

Same key → same partition → ordered.

❌ Wrong: auto-commit + processing failure = silent data loss

new ConsumerConfig { EnableAutoCommit = true };
// auto-commits offsets even if Process() throws

✅ Correct: manual commit after success

new ConsumerConfig { EnableAutoCommit = false };
var r = consumer.Consume(ct);
await ProcessAsync(r.Message);
consumer.Commit(r);

❌ Wrong: too many partitions "for safety"

Topic with 5000 partitions for 10 events/sec.
→ Broker overhead per partition (memory, file handles) explodes.
→ Rebalances slow.
→ Cluster instability.

✅ Correct: partition for max-needed parallelism

Start at 2 × peak-consumers-expected, scale by adding partitions when measured.

❌ Wrong: hot-key partitioning

// "VIP" tenant has 80% of traffic; key = tenantId → one partition cooks

✅ Correct: composite key or per-tenant topic

Key = $"{tenantId}:{userId}"   // spreads VIP across partitions if order-per-user is fine

Or shard the VIP into a dedicated topic.

Design patterns for this topic

Pattern 1 — "Key by aggregate"

• Intent: keep a single entity's events ordered on one partition.

Pattern 2 — "Compacted topic = current state"

• Intent: materialize the latest value per key as a log.

Pattern 3 — "Schema Registry + backward compatibility"

• Intent: evolve schemas without breaking consumers.

Pattern 4 — "EOS for read-process-write within Kafka"

• Intent: atomic offset commit + output write.

Pattern 5 — "Tiered storage for long retention"

• Intent: keep months of events without scaling broker disks.

Pattern 6 — "Cooperative-sticky everywhere"

• Intent: smaller lag spikes during scale events.

Pros & cons / trade-offs

When to use / when to avoid

• ✅ Use Kafka for streaming, replay, event sourcing, change-data-capture.
• ✅ Use compacted topics for materialized current-state.
• ✅ Use EOS only where dedup matters; default to at-least-once + downstream idempotency.
• ❌ Avoid Kafka for small workloads — Service Bus / SQS / RabbitMQ are far simpler.
• ❌ Avoid Kafka as a database — it isn't one.
• ❌ Avoid running Kafka without Schema Registry on multi-team platforms.
• ❌ Avoid picking partition count by superstition; measure throughput per partition.

Interview Q&A

Q1. Consumer-group rule? At most one consumer per partition per group. Max parallelism within a group = partition count.

Q2. Cooperative-sticky vs eager rebalance? Cooperative: only moving partitions are revoked; rest keeps flowing. Eager: stop-the-world. Use cooperative.

Q3. What does EOS need? Idempotent producer + transactional API + read_committed consumer. Atomic across partitions and offset commit.

Q4. KRaft — what changed? ZooKeeper gone. Brokers/controllers run a Raft quorum themselves. Simpler ops, faster metadata, no external dependency.

Q5. Retention vs compaction? Retention deletes by time/size. Compaction keeps the latest record per key — a "table as log".

Q6. Schema evolution modes? Backward (new schema reads old data; upgrade consumers first), forward (old reads new), full (both). Backward is most common.

Q7. Why partition by key? Keys hash to partitions. Same key → same partition → ordered. No key → random distribution → no per-key ordering.

Q8. Hot partition fix? Better key (composite), separate topic for the hot tenant, or accept loss of ordering for the hot key by random keying.

Q9. acks settings? 0: fire-and-forget. 1: leader-only. all + min.insync.replicas≥2: durable across failures.

Q10. Why too many partitions is bad? Per-partition broker overhead (memory, file handles, replication threads). Slow rebalances. Cluster instability.

Q11. .NET Kafka client? Confluent.Kafka (librdkafka). High-performance; supports EOS, transactions, schema registry plug-ins.

Q12. Tiered storage? Hot segments on broker disks; warm segments offloaded to object storage. Long retention without growing brokers.

Q13. Streams vs ksqlDB vs Flink? Streams = JVM library. ksqlDB = SQL over Streams. Flink = independent stream processor; richer state and windows.

Q14. Kafka Connect — when? For source/sink integrations without writing custom producers/consumers — DBs (Debezium), object storage, search indices.

Q15. Monitoring signals? Consumer lag, ISR shrink, under-replicated partitions, request p99, log-end-offset growth.

Gotchas / common mistakes

• ⚠️ Auto-commit on with side-effecting handlers — silent data loss on errors.
• ⚠️ Hot-key partitioning — one partition pegs at 100%, others idle.
• ⚠️ Adding partitions later — breaks per-key ordering for keys that re-hash.
• ⚠️ EOS without measuring — paying the throughput tax for at-least-once-equivalent semantics.
• ⚠️ Retention sized without RF — undersized by 3×.
• ⚠️ No Schema Registry — runtime breakage on shape changes.
• ⚠️ Long-running message handler — exceeds max.poll.interval.ms, consumer kicked, partition reassigned, message reprocessed.
• ⚠️ Default eager assignor in legacy clients — unnecessary lag spikes.
• ⚠️ gzip compression by default — slow vs lz4/zstd; use lz4 unless you need ratio.
• ⚠️ Treating Kafka as a queue — it's a log; deletion comes from retention/compaction, not "ack and remove".

Further reading

• Apache Kafka documentation
• Confluent Platform docs
• Confluent.Kafka .NET client
• KIP-500: KRaft
• KIP-405: Tiered Storage
• Kafka transactions design
• Schema Registry
• Designing Event-Driven Systems — Ben Stopford