A single misconfigured setting in Kafka doesn't just slow a page down. It can silently vaporize financial transactions. In this deep dive we walk through real production failure scenarios and the exact reasoning interviewers look for in staff-level system design interviews.
You'll learn:
- Why acks=1 loses data, and the four configuration layers for zero data loss
- Consistency vs availability: why a healthy cluster refuses writes on purpose
- Pushing from 400,000 to 2 million events/s with batching, linger.ms and compression
- How retries reorder a debit and a credit, and how idempotent producers fix it
- Rebalance storms, cooperative sticky assignment and static membership
- Hot partitions and the irreducible trade-off between ordering and scale
- How deleting and recreating a topic silently skips hours of data
- A 3 AM role-play: 85 under-replicated partitions, a disk at 96%, and why you never restart the broker
- Exactly-once with Kafka transactions, and exactly-once into PostgreSQL
- Tiered retry topics and dead letter queues
- Split brain, KRaft quorums and vanishing tombstones
- System design at scale: multi-tenant topics, quotas, event-driven sagas and change data capture with Debezium
Ideal for backend and platform engineers preparing for senior and staff-level interviews.
Chapters
00:00 Distributed systems have no X-ray
01:08 Why Kafka interviews matter
02:19 acks=1 and the 200 missing payments
03:27 Four layers of zero data loss
05:21 The CAP theorem trade-off
06:31 Throughput: taxis vs buses
07:42 batch.size, linger.ms and compression
09:38 When retries reorder messages
11:38 Idempotent producers
12:48 Message keys and partition ordering
13:12 Flash sale: buffer exhausted
14:22 Decoupling with a local buffer and circuit breaker
16:18 Rebalance storms
18:13 Eager vs cooperative sticky rebalancing
19:25 Static group membership
20:12 Hot partitions and the hot key problem
23:18 Silent data loss after recreating a topic
25:20 auto.offset.reset and topic versioning
26:08 3 AM: 85 under-replicated partitions
28:54 Why you never restart the broker
29:42 Disk at 96%: the 15-minute response
31:36 Throttled reassignment, Cruise Control, tiered storage
33:36 Duplicates in consume-transform-produce
34:47 Kafka transactions
37:11 Exactly-once into PostgreSQL
39:56 Head-of-line blocking from retries
41:08 Tiered retry topics and DLQs
42:42 Split brain
44:38 KRaft and quorum math
45:27 Log compaction tombstones
48:33 Multi-tenant topic design
50:07 Tenant tiers and client quotas
51:40 Event-driven sagas
53:15 Compensating transactions
54:51 Change data capture with Debezium
57:13 Recap
58:46 Will offsets still matter in five years?
#Kafka #SystemDesign #DistributedSystems #SoftwareEngineering #InterviewPrep #BackendEngineering
Interview Prep Podcast
