Kafka Interview and System Design Guide
The capstone module for the Apache Kafka track: full worked system-design interview questions synthesizing partitioning, delivery semantics, schema design, consumer groups, and monitoring, plus a complete vocabulary cheat sheet, common interview traps, and rapid-fire conceptual Q&A.
Quick answer
Kafka design interviews check whether partitioning, delivery semantics, schemas, consumer-group layout and monitoring follow from the requirements. Recurring tools: a keyed topic for per-entity ordering, a compacted topic for current state, separate groups for independent consumers, and a dead letter topic for bad messages.
- Pick keys from the ordering the business needs.
- Choose acks and idempotence from the cost of losing data.
- Say how you would detect lag and failures.
orders key=order_id, 12 partitions, RF=3, acks=all, idempotent producer
order-state compacted, key=order_id (current status)
consumers billing, email, analytics → one group each
failures orders.dlq after 3 retries
alerts under-replicated > 0, billing lag > 10k for 5 minYou Have Learned the Previous 25 Modules. Here Is How It Comes Together.
This lesson is different from the previous 25. It does not introduce new Kafka mechanics. It is a synthesis — a place where partitioning strategy, delivery semantics, schema design, consumer group topology, and monitoring all show up together, the way they actually do in a real system-design interview or a real production architecture review, instead of one concept at a time.
A Kafka system-design interview almost never asks "what does acks=all do." It asks "design a real-time analytics pipeline for a ride-sharing app's driver locations," and expects you to arrive at acks=all — and a partitioning key, and a retention policy, and a consumer group topology — as the natural consequence of reasoning through requirements, not as a memorized fact recited on cue. That is the skill this lesson builds: taking everything from Lessons 01 through 25 and using it to reason through a system from a cold start, out loud, the way an interviewer actually wants to see it happen.
Work through the worked examples below the way you would in a real interview — read the prompt, pause, sketch your own answer before reading the walkthrough, then compare. The rapid-fire section and vocabulary table near the end are for cramming; the worked system-design parts are for genuinely practicing the reasoning.
A Reusable Structure for Any Kafka System-Design Prompt
Every worked example in this lesson, from Part 01 through Part 09, follows the same five-step structure deliberately, because that structure is itself the thing interviewers are evaluating — more than any single correct answer. Internalizing this shape means a genuinely unfamiliar prompt on interview day is still approachable, because the process for attacking it is the same one practiced repeatedly below.
| Step | What to actually do | Why interviewers weight this step heavily |
|---|---|---|
| 1. Requirements gathering | Ask about scale, latency needs, acceptable staleness or loss, and ordering requirements before proposing anything. | A design built on assumed requirements that turn out wrong is a design for the wrong problem — interviewers are testing whether you ask, not whether you guess correctly. |
| 2. Capacity estimation | State assumed numbers explicitly, then apply the formulas from this lesson's capacity-planning reference to get partition count, storage, and throughput figures. | A design with no numbers behind it cannot be sanity-checked by anyone, including you — stating rough math is what makes a design falsifiable and therefore credible. |
| 3. Topic and partition design | Decide topic boundaries, partition keys, and partition counts, and justify the partition key choice specifically in terms of ordering and skew. | The partition key decision is the single choice with the most downstream consequences in almost any Kafka design — interviewers listen closely for how it is justified. |
| 4. Delivery semantics and durability trade-offs | Choose acks, replication factor, and idempotence settings as a deliberate consequence of what the data actually is, not a default. | This is where "acks=all is always right" gets tested directly — see this lesson's interview-traps section. |
| 5. Trade-offs stated out loud | Name what the design gives up, not just what it achieves — a staleness window, an idle-consumer ceiling, an operational cost. | A design presented with no acknowledged weaknesses reads as either inexperienced or evasive; naming trade-offs unprompted is a strong, deliberately practiced signal. |
Notice this structure never starts with "here's the Kafka topology" — it starts with questions. Interviewers consistently report that the single most common mistake in a system-design interview, Kafka-specific or otherwise, is a candidate leaping straight to an architecture diagram before establishing what problem that architecture is actually meant to solve. Every worked example in this lesson opens with requirements gathering for exactly this reason, not as a formality.
Design a Real-Time Analytics Pipeline for a Ride-Sharing App's Driver Location Updates
Requirements gathering
Before touching Kafka at all, the first thing a strong candidate does is ask clarifying questions, because the right architecture depends entirely on the answers. For this prompt: how many active drivers send updates, and how often? What does "real-time analytics" actually mean — a live map for riders, aggregate metrics for a dashboard, or both? Does the system need the single latest location per driver, a full history, or both? What is the acceptable staleness — one second, five seconds, thirty? Is exact ordering per driver required, or just eventual correctness?
Reasonable answers for this exercise: 2 million active drivers during peak, each sending a location ping every 4 seconds, so roughly 500,000 events per second at peak. The system needs both a live map (latest position per driver, low latency) and rolling aggregate metrics (active drivers per city, average speed per zone) for an ops dashboard. Per-driver ordering matters — a rider should never see a driver's location jump backward — but cross-driver ordering does not matter at all.
Capacity estimation
Peak event rate: 500,000 events/sec
Average event size: ~200 bytes (driver_id, lat, lng, timestamp, speed, heading)
Peak throughput: 500,000 * 200 bytes = ~100 MB/sec
Retention needed: short for raw pings (this is a firehose, not an archive) — 24 hours is plenty,
since the live map only needs recent state and aggregates are computed and stored elsewhere.
Storage for 24h retention (before replication):
100 MB/sec * 86,400 sec = ~8.6 TB/day
With replication factor 3: ~26 TB of disk across the cluster for this one topic
Partition count: one partition can sustain roughly 10-20 MB/sec of writes comfortably on
typical hardware. To handle 100 MB/sec with headroom for peak spikes and future growth:
100 MB/sec / 15 MB/sec per partition ≈ 7, round up generously to 24 partitions
(partition count should also comfortably exceed the largest expected consumer group size,
covered in Part 09 of this lesson's vocabulary and in the earlier consumer-groups module)Topic and partition design
One topic, driver.locations.raw, partitioned by driver_id. Keying by driver ID is the central decision here: it guarantees every update for a single driver lands on the same partition in send order, which is exactly what per-driver ordering requires, per the partitioning module's per-key ordering guarantee — without needing a single global partition, which would never sustain 500,000 events/second. Partition count of 24 (from the estimation above) spreads load across enough brokers while staying comfortably above any realistic consumer group size, avoiding the idle-consumer problem covered in the producers-consumers-brokers module.
A second, separate compacted topic, driver.locations.current, keyed bydriver_id, holds only the latest position per driver — exactly the log-compaction pattern from the message-brokers-queues module. A stream-processing job (Part 02 below reuses this exact pattern) consumes the raw topic and continuously writes the latest position per driver into this compacted topic, giving any new consumer (a freshly deployed map-rendering service, for example) an instant, complete current-state view on startup by reading the compacted topic from the beginning, rather than replaying the entire raw firehose.
Delivery semantics and durability trade-off
This is a case where the correct answer is deliberately not the strongest possible durability. A single missed location ping from one driver, four seconds before the next one arrives, has essentially no business consequence — the map is stale for four seconds either way.acks=1 with replication.factor=3 is the right trade-off: strong enough that a routine broker restart does not lose data, without paying the latency and throughput cost ofacks=all at 500,000 events/second, where that cost compounds significantly. This is exactly the "tolerable loss" row from the durability-vs-availability table in the message-brokers-queues module, applied deliberately rather than by default.
Consumer group topology and processing
| Consumer | Reads | Purpose |
|---|---|---|
| map-service consumer group | driver.locations.current (compacted) | Serves the live rider-facing map — always has the latest position per driver, instant startup from compacted state. |
| aggregation stream job | driver.locations.raw | Kafka Streams windowed aggregation — active drivers per city, average speed per zone, emitted to a driver.metrics.5min topic on a rolling window. |
| cold-storage sink connector | driver.locations.raw | Kafka Connect sink to object storage for historical/ML training data, independent consumer group, does not affect real-time paths. |
Three independent consumer groups reading the same raw topic — exactly the pub-sub fan-out pattern from the queues-vs-topics module — means the cold-storage pipeline can lag or even go down entirely without affecting the live map, and the aggregation job's throughput has zero coupling to how fast the map service renders updates.
Trade-offs worth stating out loud in an interview
Keying by driver_id assumes reasonably even distribution across drivers — true here since no single driver dominates traffic, unlike the hot-partition risk called out for heavily-skewed keys in the producers-consumers-brokers module. Choosing 24 partitions up front means a future repartitioning (an operationally disruptive event, since it changes which partition a given key hashes to) is deferred but not eliminated — a strong answer acknowledges this rather than treating the partition count as permanent.
Design an Order Processing System for an E-Commerce Platform Using Kafka
Requirements gathering
Clarifying questions here matter even more than in Part 01, because order processing touches money and inventory — the cost of getting delivery semantics wrong is direct financial loss or a double-charged customer, not a four-second-stale map. What must happen when an order is placed — payment, inventory deduction, shipping notification, confirmation email? Must these happen exactly once, or is at-least-once with idempotent downstream handling acceptable? Can order events for different customers be processed out of order relative to each other? Must a single customer's orders be processed in the order they were placed?
Reasonable answers: payment, inventory, and notification are separate downstream systems, each consuming the same order event independently. A double-charge or double-shipment is unacceptable — each downstream effect must happen effectively once. Cross-customer ordering does not matter; a single customer's own orders should process in the order placed.
Topic design and the fan-out pattern
One topic, orders.placed, keyed by customer_id — guaranteeing per-customer ordering exactly as in Part 01, without the cost of a single global partition. Three independent consumer groups subscribe to it: payment-service, inventory-service, andnotification-service. This is the queues-vs-topics module's pub-sub pattern applied directly: one event, several independent services, each maintaining its own offset, none blocking or affecting the others.
Topic: orders.placed (key = customer_id, 12 partitions, RF=3)
payment-service group -> charges the customer, writes to payments.processed
inventory-service group -> deducts stock, writes to inventory.adjustments
notification-service group -> sends confirmation email, writes to notifications.sent
Each group has its own committed offset in __consumer_offsets.
payment-service falling behind does not slow down notification-service at all.Exactly-once handling for the payment path specifically
Payment is the one path in this system where "effectively once" needs to be engineered deliberately rather than assumed. The producer side uses enable.idempotence=true to eliminate broker-level duplicate writes from retries, per the delivery-semantics module. But the more important protection is downstream: the payment-service consumer processes each order using an idempotency key — a unique, stable order_id — checked against the payment processor's own idempotency-key support (a pattern common to Stripe and similar payment APIs) before issuing a charge. This closes the gap that idempotent producers alone do not cover: a payment service that crashes after successfully charging a card but before committing its Kafka offset will be redelivered the same order on restart, and the idempotency key is what prevents that redelivery from charging the customer twice.
Handling a downstream failure without stopping the whole pipeline
If the payment processor is degraded and returning errors, payment-service should not block the entire partition indefinitely on retries — that would also block every order behind the failing one on the same partition, per the ordering guarantees from the ordering-guarantees module. The dead letter queue pattern from the message-brokers-queues module applies directly: after a bounded number of retries, the order is written to orders.placed.dlq with full context (original topic, partition, offset, and the error), the offset is committed, and processing continues for subsequent orders. A separate on-call process monitors the DLQ and replays once the payment processor recovers.
Monitoring this system
| Metric | What it tells the on-call engineer |
|---|---|
| Consumer lag per group, per partition | Whether payment-service specifically is falling behind — the single most important signal for a money-touching consumer. |
| orders.placed.dlq message rate | A rising DLQ rate almost always means the payment processor or another downstream dependency is degraded, not a Kafka problem. |
| Under-replicated partitions on orders.placed | A shrinking ISR here threatens the acks=all durability guarantee this topic depends on for correctness. |
| End-to-end latency, order placed to confirmation email sent | A single number the business actually cares about, cutting across all three consumer groups at once. |
Design a Notification Fan-Out System
Requirements gathering
What triggers a notification, and how many channels does a single event fan out to — push, email, SMS, in-app? Do different channels have wildly different throughput and latency needs (push is typically near-real-time and high volume; SMS is expensive per-message and often rate-limited by a third-party provider)? Should a user's notification preferences (opted out of SMS, only wants digest email) be enforced before or after the fan-out?
Reasonable answers: a single upstream event (an order shipped, a price drop on a wishlisted item) should be able to trigger zero to several notification channels depending on the user's stored preferences. Push needs to be near-instant. SMS is rate-limited by an external provider and must never block push or email. Preference filtering should happen once, centrally, not duplicated in every channel-specific consumer.
Architecture — a central fan-out topic, then per-channel topics
A single upstream topic, notification.triggers, receives the raw triggering event. One consumer group, preference-filter-service, reads it, looks up the user's stored preferences, and produces zero or more filtered, channel-specific events downstream — tonotification.push, notification.email, andnotification.sms, three separate topics. Each channel then has its own dedicated consumer service with its own scaling and rate-limiting characteristics, completely decoupled from the others by the fact that they are separate topics with separate consumer groups.
notification.triggers (key = user_id)
|
v
preference-filter-service (one consumer group)
|
+--> notification.push (key = user_id) --> push-service (scales independently, near-real-time)
+--> notification.email (key = user_id) --> email-service (scales independently, batched sends OK)
+--> notification.sms (key = user_id) --> sms-service (rate-limited by external provider)This is the same decoupling principle as the message-brokers-queues module's rate-decoupling guarantee, applied deliberately across three channels with genuinely different throughput profiles: the SMS provider's rate limit can never cause push notifications to back up, becausenotification.sms is a physically separate topic and consumer group fromnotification.push — a slow or throttled SMS consumer only grows lag on its own topic.
Why filter once centrally instead of in each channel consumer
Putting preference logic in preference-filter-service rather than duplicating it in push-service, email-service, and sms-service independently means a change to preference rules ships once, and the three channel services stay focused purely on channel-specific delivery mechanics. It also means a user who opted out of all channels for a given trigger type simply never produces a record to any of the three downstream topics — the channel services never even see the suppressed event, rather than each independently deciding to drop it.
Delivery semantics per channel
Not every channel needs the same guarantee. An occasional duplicate push notification is a mild annoyance; an occasional duplicate SMS costs real money per message sent to a third-party provider.enable.idempotence=true on the shared producer in preference-filter-serviceis the baseline for all three topics, but sms-service additionally deduplicates at the application level using a stable notification ID before calling the SMS provider, exactly the consumer-side-idempotency pattern from the delivery-semantics module — because the cost asymmetry between channels justifies the extra engineering effort for SMS specifically and not for push.
How Would You Migrate a Legacy Batch ETL Pipeline to a Streaming Kafka Pipeline?
Requirements gathering — this prompt is about migration risk, not a clean-slate design
The critical difference from Parts 01-03: this system already exists and already works, just on a nightly batch schedule. The interview is testing whether a candidate defaults to "rip it out and replace it" versus a lower-risk incremental migration. Key questions: what does the batch job actually do (extract from a source database, transform, load into a warehouse)? Who depends on its output, and on what schedule? Is the goal lower latency, or is it actually about reducing load on the source database from a nightly full-table scan?
Reasonable answers: a nightly job extracts all orders from a production Postgres database, joins them with customer data, and loads aggregated tables into a warehouse for BI dashboards. The nightly full-table extract is now measurably slowing down the production database during the extract window, and stakeholders separately want same-day visibility rather than next-day.
The migration approach — Change Data Capture, not a rewritten application
The right first move is not rewriting the source application to produce Kafka events directly — that is a large, risky change to a system that works today. The right first move is Kafka Connect with a CDC (Change Data Capture) source connector (Debezium, reading the database's write-ahead log) that streams every insert, update, and delete on the orders table into a Kafka topic automatically, with zero changes to the application code that writes to Postgres. This is exactly the Kafka Connect pattern from the connect-kafka-connect module — using a battle-tested connector instead of hand-rolling a custom producer that polls the database.
Phase 1 (parallel run, zero risk to the existing pipeline):
Postgres orders table --[Debezium CDC connector]--> orders.changes topic
Nightly batch job keeps running exactly as before, untouched.
A new Kafka Streams job consumes orders.changes and populates the SAME
warehouse tables the batch job populates, into a separate schema for validation.
Phase 2 (validation):
Compare streaming-populated tables against batch-populated tables nightly.
Fix discrepancies in the streaming transform logic until they match consistently.
Phase 3 (cutover):
BI dashboards repointed to the streaming-populated tables.
Nightly batch job disabled, but its code kept for a rollback window.
Phase 4 (cleanup, weeks later, once confidence is high):
Batch job and its scheduling infrastructure decommissioned entirely.Running the streaming pipeline in parallel with the existing batch job for a validation period is the single highest-leverage risk-reduction move in this entire migration — it means a bug in the new streaming transform logic is caught by comparing against known-good batch output, rather than discovered by a stakeholder noticing a wrong number on a dashboard weeks after cutover.
Where log compaction fits in this specific migration
The CDC topic itself should not be compacted — the batch-to-streaming validation in Phase 2 needs the full change history, not just latest state, to reconstruct exactly what the batch job would have seen. But a derived topic, orders.current, compacted and keyed byorder_id, is exactly the right pattern (from the message-brokers-queues module) for any downstream consumer that only needs current order state rather than full history — giving new consumers an instant materialized view without replaying the entire CDC log from the beginning.
Trade-offs to state explicitly
CDC shifts load from "a nightly full-table scan on the production database" to "continuous CDC read of the write-ahead log," which is far lighter per-unit-time but is now a continuous rather than a scheduled load — worth explicitly calling out to whoever owns the source database. It also introduces a genuinely new failure mode that did not exist with a nightly batch job: if the CDC connector falls behind or the source database's WAL retention is exceeded before the connector catches up, a full re-snapshot of the source table may be required — a cost that a nightly full extract, by definition, never has to worry about.
Design a Fraud Detection Pipeline for Real-Time Payment Transactions
Requirements gathering
What must the latency budget be from transaction to fraud verdict — is this blocking the transaction itself, or an async flag reviewed afterward? Does fraud scoring need only the current transaction, or a rolling window of a customer's recent transaction history? What is the cost of a false negative (a fraudulent transaction let through) versus a false positive (a legitimate transaction blocked or delayed)?
Reasonable answers: fraud scoring must complete within 200ms and blocks the transaction — this makes it latency-critical in a way none of the previous four examples were. Scoring needs a rolling window of the customer's transactions over the last 24 hours (velocity checks — too many transactions too fast is a strong fraud signal). A false negative is far more costly than a false positive here, so the system should be tunable toward slightly more false positives under load.
Why this is fundamentally different from Parts 01-04 — a blocking, latency-critical path
Every previous example in this lesson was async — Kafka absorbed the work and something downstream eventually caught up. A fraud check that blocks the transaction cannot use Kafka the same way, because the caller is waiting on a synchronous answer within 200ms, and Kafka's own end-to-end latency (produce, replicate, consume, process) can already eat a meaningful fraction of that budget under load. The honest answer in an interview is that Kafka is not the request/response layer here — it is the state-building layer underneath a synchronous scoring service.
Architecture — Kafka Streams builds a materialized state store, a synchronous API queries it
transaction.events topic (key = customer_id)
|
v
Kafka Streams job: windowed aggregation (24h rolling transaction velocity per customer)
|
v
materialized state store (backed by a compacted changelog topic, per the
message-brokers-queues module's compacted-topic pattern) -- queryable via
Kafka Streams Interactive Queries, keyed by customer_id, sub-millisecond reads
Synchronous path (NOT going through Kafka's produce/consume latency at all):
payment gateway --[HTTP, <200ms budget]--> fraud-scoring-service
|
v
queries the materialized state store
directly via interactive queries,
combines with the current transaction,
returns a verdict synchronouslyThe transaction event is still produced to Kafka (for the Streams job to eventually incorporate it into the rolling window, and for downstream audit/analytics consumers), but the fraud verdict itself is computed by querying an already-materialized state store directly — not by waiting on a fresh round trip through Kafka's produce-replicate-consume path. This is the key insight the interviewer is looking for: Kafka Streams' state stores can be queried like a database, decoupling "how state gets built" (streaming, asynchronous, eventually consistent by a small margin) from "how a synchronous caller reads it" (a fast local or network query against materialized state, not a Kafka consume call).
Monitoring specific to a blocking, latency-critical consumer
Beyond the standard consumer-lag and under-replicated-partition monitoring from Part 02, this system needs one more critical metric: the Streams job's own processing lag relative to real time, because that lag directly determines how stale the velocity data behind every fraud verdict is. A growing processing lag here is not just an operational inconvenience — it is a silent, gradual degradation of fraud-detection accuracy that would not show up as an obvious outage.
Design a Multi-Region Event Replication Strategy for a Global SaaS Platform
Requirements gathering
Does every region need to see every event, or only events relevant to users in that region? Is this active-active (writes accepted in multiple regions) or active-passive (one primary region, others read-only replicas for disaster recovery)? What is the acceptable data-loss window if the primary region fails entirely?
Reasonable answers: this is a SaaS platform with regional data residency requirements — EU customer data must stay associated with an EU cluster for compliance, but a small set of global events (product catalog updates, feature flags) need to reach every region. Active-passive per customer region, with cross-region replication of only the specifically-designated global topics.
Why this is not solved by simply widening replication factor within one cluster
A common wrong turn here (also flagged in the interview-traps section below) is reasoning about this as "just increase the replication factor." Replication factor, as covered in the message-brokers-queues module, replicates a partition across brokers within one cluster for durability against a single broker failure. It says nothing about geography — all replicas of a Kafka partition are typically within the same cluster, often the same data center or a small set of nearby availability zones, because Kafka's own in-sync-replica protocol assumes relatively low inter-broker latency. Multi-region replication needs a different mechanism entirely.
The mechanism — MirrorMaker 2, not a wider replication factor
MirrorMaker 2 (built on Kafka Connect) is the standard tool for cluster-to-cluster replication across regions: it runs as a Connect cluster that consumes from a source-region Kafka cluster and produces the same records into a destination-region cluster, preserving topic partitioning and offsets in a translatable way, and it can be configured for one-directional (active-passive) or bidirectional (active-active) replication between named clusters.
EU cluster (primary for EU customers) US cluster (primary for US customers)
eu.orders.placed (EU customer orders) us.orders.placed (US customer orders)
eu.catalog.updates ---[MirrorMaker 2]---------> us.catalog.updates (replicated copy)
us.catalog.updates <---[MirrorMaker 2]--------- eu.catalog.updates (replicated copy)
Customer order data stays region-local (compliance requirement) --
eu.orders.placed is NEVER mirrored to the US cluster.
Catalog updates ARE global and ARE mirrored bidirectionally, so a
product change made through either region's admin tooling reaches both.The deliberate asymmetry here — customer order topics never replicated, catalog topics always replicated bidirectionally — is the actual system-design decision an interviewer wants to see articulated: not every topic needs the same replication treatment, and conflating "replicate everything for disaster recovery" with "replicate this specific topic because multiple regions genuinely need it" leads to either a compliance violation or unnecessary cross-region bandwidth cost.
Disaster recovery for the region-local topics
For eu.orders.placed itself, which must stay EU-local for compliance, disaster recovery is handled differently — a passive MirrorMaker 2 replication into a second EU cluster in a different EU availability zone (still within the compliance boundary), not into the US cluster. This gives regional failover without ever violating the data-residency requirement that motivated keeping the topic region-local in the first place.
Design a Reliable Event-Publishing Pattern for a Monolith Migrating to Event-Driven Architecture
Requirements gathering
A common, distinct prompt from a greenfield design: an existing monolithic application writes to its own relational database as the source of truth, and now needs to reliably publish an event to Kafka every time certain rows change — without a rewrite, and without risking the two writes (the database commit and the Kafka publish) getting out of sync with each other. What happens today when the application crashes between committing to the database and publishing to Kafka? Is losing an event ever acceptable, or publishing a duplicate event, or neither?
Reasonable answers: the application currently has no event publishing at all — this is a new capability being layered on. Losing an event entirely is not acceptable (downstream systems must eventually learn about every change); an occasional duplicate event is acceptable as long as downstream consumers are idempotent, per the delivery-semantics module's general framing.
Why "just call the Kafka producer right after the database commit" is the wrong answer
The naive approach — commit to the database, then call producer.send() — has an unavoidable gap: if the process crashes, or the Kafka broker is briefly unreachable, exactly between those two calls, the database commit has already happened but the event is never published. There is no way to retry "the publish that never happened" after a crash, because the application has no durable record that a publish was even owed. This is precisely the kind of dual-write consistency problem that motivates the transactional outbox pattern.
The transactional outbox pattern
Instead of writing to the business table and calling Kafka directly, the application writes to the business table and an outbox table, in the same local database transaction — an operation the database itself guarantees is atomic, with none of Kafka's cross-system consistency problem. A separate process then reads unpublished rows from the outbox table and publishes them to Kafka, marking them published only after a confirmed send. The database transaction's atomicity is what makes this reliable: either both the business row and its corresponding outbox row are committed, or neither is — there is never a state where the business change happened but no record exists that an event is owed for it.
-- Single atomic database transaction, both writes or neither:
BEGIN;
UPDATE orders SET status = 'shipped' WHERE id = 4821;
INSERT INTO outbox (id, topic, key, payload, published)
VALUES (gen_uuid(), 'orders.status_changed', '4821',
'{"order_id": 4821, "status": "shipped"}', false);
COMMIT;
-- A separate relay process (often itself a Debezium CDC connector reading
-- the outbox table's own write-ahead log, per Part 04's CDC pattern):
-- reads unpublished outbox rows -> produces to Kafka -> marks published = true
-- If the relay crashes mid-publish, it simply resumes from the last
-- unpublished row on restart -- at worst producing a duplicate, never a lossUsing a CDC connector (as in Part 04) to read the outbox table's own change log, rather than a hand-rolled polling process, is the preferred implementation in practice — it reuses a battle-tested connector instead of building bespoke polling-and-marking logic, and it naturally picks up new outbox rows with low latency by tailing the database's write-ahead log rather than polling on an interval.
Design a Centralized Log and Metrics Aggregation Pipeline for a Microservices Platform
Requirements gathering
How many services, and what is the aggregate log volume? Do logs need to be searchable in near-real-time by on-call engineers, or is next-hour acceptable? Is sampling acceptable for very high-volume debug-level logs, or must every log line be retained? Are metrics (numeric time series) handled the same way as logs (free-text events), or do they need a fundamentally different pipeline?
Reasonable answers: roughly 400 microservices, aggregate log volume peaking around 200,000 lines per second across the fleet. On-call engineers need logs searchable within seconds during an incident. Debug-level logs can be sampled at high volume; error and warn-level logs must never be dropped. Metrics are numeric and high-cardinality and genuinely need a different downstream system (a time-series database) than logs do (a search index) — Kafka's job here is to be the reliable, decoupling ingestion layer in front of both, not to replace either specialized store.
Why this is a strong candidate for Kafka despite looking like "just logging"
A naive design has every service write logs directly to the search index (e.g. Elasticsearch). This couples every single service's log volume directly to the search index's real-time write capacity — a spike in log volume from one misbehaving service, or the search index having a slow moment during a reindex, propagates backpressure straight into application code that has nothing to do with logging. Kafka's rate-decoupling guarantee, from the message-brokers-queues module, is the exact fix: every service writes its logs to Kafka, fast and fire-and-forget, and Kafka absorbs any mismatch in rate between log production and the search index's ability to ingest.
Topic design — separated by urgency, not by service
A common mistake is one topic per microservice, which sounds organized but actually fragments partitioning decisions across 400 small topics with no meaningful traffic to justify separate partition counts for most of them. The better design groups by urgency and handling requirement instead: logs.error (never sampled, never dropped, highest priority for the search-index consumer), logs.warn, and logs.debug (aggressively sampled at the producer, before it ever reaches Kafka, to keep volume manageable). Each service still tags every log line with its own service name as a field in the payload — searchability by service is preserved entirely at the search-index layer, without needing per-service topics.
logs.error (RF=3, acks=all -- never acceptable to lose an error log during an incident)
logs.warn (RF=3, acks=1 -- durable enough, latency matters more than absolute durability)
logs.debug (RF=1, acks=0 -- sampled at 1% by the client library before it is even sent;
high volume, low individual value, loss is fully acceptable)
metrics.raw (separate topic entirely -- numeric time-series points, not free text)
Consumer groups:
search-index-sink (Kafka Connect sink connector) -> reads logs.error, logs.warn, logs.debug
-> writes into the search index, prioritizing error/warn topics' consumer threads
metrics-sink (Kafka Connect sink connector) -> reads metrics.raw
-> writes into the time-series database
cold-storage-sink -> reads all four topics -> writes to object storage for compliance retentionDifferentiating acks and replication factor by topic urgency — full durability for errors, relaxed durability for sampled debug logs — is the same deliberate trade-off principle from Part 01's driver-location example, applied to a completely different domain: the right Kafka durability settings are always a function of what the data actually is, never a single fleet-wide default.
Using Kafka Connect instead of hand-written sink consumers
Both the search-index sink and the metrics sink are exactly the kind of well-understood, generic integration the Connect module covers — "read from a Kafka topic, write to a well-known external system" — that an existing, maintained sink connector already implements reliably, including its own offset management and retry behavior. Hand-writing a custom consumer for either would mean re-implementing connector functionality that already exists and is already battle-tested at scale, exactly the trap named in this lesson's interview-traps section.
Design a Streaming ETL Pipeline That Joins Two High-Volume Topics in Real Time
Requirements gathering
A recurring shape of Kafka system-design prompt: enriching one stream with data from another in real time rather than batch. Concretely — a media-streaming platform wants to joinplayback.events (a play, pause, or seek, keyed by session ID) withcontent.catalog (metadata about each piece of content — title, genre, runtime) to produce an enriched event stream feeding a real-time recommendation model. How often does catalog metadata change relative to playback event volume? Must the join be exact, or is a brief staleness window on catalog metadata acceptable?
Reasonable answers: playback events arrive at roughly 300,000/second at peak; catalog metadata changes rarely by comparison — a few hundred updates per hour, mostly new content being added. A few seconds of staleness on catalog metadata (a just-published title not showing its genre in the very first few playback events after publish) is acceptable; playback events themselves must never be dropped.
Why this is a KStream-KTable join, not a KStream-KStream join
The stream-processing-kafka-streams module's distinction between joining two unbounded streams versus joining a stream against a continuously updated table applies directly here, and picking the wrong one is a common interview misstep. playback.events is naturally a KStream — each play/pause/seek is an independent fact, not an update to prior state for that key. content. catalog is naturally a KTable — each record is the latest metadata for a given content ID, exactly the upsert-per-key semantics a KTable represents, and it is almost certainly already backed by a compacted topic for that reason (via CDC from the catalog database, per Part 04's CDC pattern). A KStream-KTable join looks up the current value from the table for each incoming stream record — exactly the "enrich this event with current reference data" pattern this prompt describes.
KStream<String, PlaybackEvent> playbackEvents = builder.stream("playback.events");
KTable<String, ContentMetadata> catalog = builder.table("content.catalog");
// content.catalog is a compacted topic populated by CDC from the catalog database
KStream<String, EnrichedPlaybackEvent> enriched = playbackEvents
.join(
catalog,
(event, metadata) -> new EnrichedPlaybackEvent(event, metadata)
// NOTE: this requires re-keying playback.events by content_id first if
// it is currently keyed by session_id -- the join key must match the
// KTable's key, which is the single most common bug in this pattern
);
enriched.to("playback.events.enriched");The comment in the code above names the single most common real bug in this exact pattern: a KStream-KTable join requires both sides to share the same key. If playback.events is keyed by session_id (a reasonable choice for the raw topic, to keep one session's events ordered together) but the join needs to match on content_id, an explicit re-keying step (a selectKey or equivalent, which itself triggers a repartition under the hood) is required before the join — an easy detail to omit in a whiteboard design and exactly the kind of specific, correct detail that separates a strong answer from a surface-level one.
Handling the staleness trade-off explicitly
Because the KTable side is itself built from a continuously-consumed compacted topic, there is an inherent small lag between a catalog update landing in the source database and that update being reflected in the KTable a playback event gets joined against — exactly the kind of small, bounded staleness the requirements gathering established as acceptable. Stating this trade-off explicitly, rather than presenting the join as instantaneous, is the same discipline as Part 05's fraud-scoring example: acknowledging a materialized view's staleness window is itself part of a complete design answer, not a flaw to gloss over.
Kafka Vocabulary Cheat Sheet
A dense, precise reference covering every major term used across the 26-module track — built for interview cramming, not first-time learning. If a definition here is unclear, the module it maps to covers it in full depth.
| Term | Precise definition |
|---|---|
| Broker | A single Kafka server process that stores partition logs, serves client reads/writes, and participates in cluster coordination. |
| Cluster | A group of brokers working together, sharing cluster metadata and a controller. |
| Topic | A named, durable, append-only log of records, divided into one or more partitions. |
| Partition | An ordered, immutable sequence of records within a topic; the unit of parallelism and the unit of strict ordering. |
| Offset | A record's position within its partition — a monotonically increasing integer, unique per partition, never reused. |
| Leader (replica) | The one replica of a partition that handles all reads and writes for it at a given time. |
| Follower (replica) | A replica that continuously fetches and copies the leader's log, ready to become leader if the current leader fails. |
| ISR (In-Sync Replica set) | The set of replicas, including the leader, that are currently caught up within the broker's configured replication lag tolerance. |
| Replication factor | How many total copies of each partition exist across the cluster, on different brokers. |
| min.insync.replicas | The minimum ISR size required for a write with acks=all to succeed; below this, the partition refuses writes rather than risk silent data loss. |
| acks | Producer setting controlling how many replicas must confirm a write before it is acknowledged: 0 (none), 1 (leader only), all (full ISR). |
| Controller | The one broker (or, in KRaft mode, the elected leader of the controller quorum) responsible for cluster-wide decisions like leader election. |
| KRaft | Kafka's Raft-based consensus mode for cluster metadata and controller election, replacing the older ZooKeeper-based coordination entirely. |
| Consumer group | A named set of consumers that share the work of reading a topic; each partition is read by exactly one consumer within the group at a time. |
| Rebalance | The process of reassigning partitions among the members of a consumer group, triggered by a member joining, leaving, or being declared dead. |
| Cooperative (incremental) rebalancing | A rebalancing protocol that reassigns only the specific partitions that need to move, instead of revoking every partition from every consumer. |
| Static membership | Using a stable group.instance.id so a consumer reconnecting quickly (e.g. during a rolling deploy) is recognized as returning, skipping an unnecessary rebalance. |
| Consumer lag | The difference between a partition's latest offset and a consumer group's committed offset — the operational signal for backpressure. |
| Idempotence (producer) | A producer feature (enable.idempotence=true) where the broker deduplicates retried writes using a per-producer-session sequence number, eliminating retry-caused duplicates at the broker level. |
| Transaction | An atomic group of writes across one or more partitions/topics plus a consumer offset commit, coordinated by a transaction coordinator so all of it succeeds or none of it does. |
| isolation.level (read_committed) | A consumer setting that filters out records from aborted or still-in-flight transactions, showing only committed data. |
| Exactly-once semantics | The combined guarantee of idempotent producers plus transactions plus read_committed consumers, ensuring a read-process-write loop has no duplicates and no partial results. |
| Retention | The policy governing how long records are kept in a topic before being eligible for deletion — time-based, size-based, or both. |
| Log compaction | A retention alternative that keeps at least the latest record for every key forever, discarding only older, superseded versions of the same key. |
| Tombstone | A record with a null value, used to signal that a key should eventually be fully removed from a compacted topic. |
| Segment | A fixed-size chunk of a partition's log on disk; only the newest segment (the active segment) is being written to at any time. |
| Schema Registry | A separate service that stores and versions record schemas (commonly Avro, Protobuf, or JSON Schema) and enforces compatibility rules between schema versions. |
| Compatibility mode | The rule (backward, forward, or full) the Schema Registry enforces when evaluating whether a new schema version is safe relative to existing producers/consumers. |
| Kafka Connect | A framework for running reusable source connectors (external system -> Kafka) and sink connectors (Kafka -> external system) without hand-writing custom producers/consumers. |
| Change Data Capture (CDC) | A source-connector pattern that streams row-level inserts/updates/deletes from a database's own change log (e.g. a write-ahead log) into Kafka. |
| Kafka Streams | A client library for building stream-processing applications directly against Kafka, without a separate processing cluster. |
| KStream | A Kafka Streams abstraction representing an unbounded stream of individual records — every record is a new, independent event. |
| KTable | A Kafka Streams abstraction representing a continuously updated table, backed by a compacted changelog topic — each record is an upsert to the latest state for its key. |
| TopologyTestDriver | A test utility that runs a real Kafka Streams topology against simulated, in-process input/output topics with a controllable simulated clock, with no real broker involved. |
| ksqlDB | A SQL-based interface for building Kafka Streams-style stream-processing applications using declarative queries instead of client-library code. |
| MirrorMaker 2 | A Kafka Connect-based tool for replicating topics between separate Kafka clusters, commonly across regions or for disaster recovery. |
| Dead letter queue (DLQ) | A separate topic where a consumer routes a record that repeatedly fails processing, so the main partition can keep advancing instead of blocking indefinitely. |
| Testcontainers | A library that runs a real, ephemeral broker in a Docker container for the duration of an integration test, then tears it down. |
| bootstrap.servers | The producer/consumer config listing one or more brokers used only as an initial entry point to discover full cluster metadata — not a list of every broker. |
| linger.ms | How long a producer waits, hoping more records arrive to fill a batch, before sending it — the core producer latency-versus-throughput knob. |
| max.poll.interval.ms | The maximum time allowed between consecutive poll() calls before the consumer is presumed dead and its partitions are rebalanced away. |
| session.timeout.ms | The maximum time the group coordinator waits without a heartbeat before declaring a consumer dead — distinct from max.poll.interval.ms, which tracks a different failure mode. |
Capacity-Planning Formulas You Can Reuse in Any System-Design Interview
Every worked example in this lesson used the same handful of back-of-envelope calculations. Having these ready to apply immediately, rather than deriving them from scratch under interview pressure, is worth the memorization — an interviewer consistently rates a candidate who states a number and shows the arithmetic higher than one who hand-waves past sizing entirely.
1. Peak throughput:
events/sec * average event size (bytes) = bytes/sec
(always ask for or estimate BOTH numbers separately — candidates who only
estimate one and guess the other tend to be off by an order of magnitude)
2. Partition count, from throughput:
target throughput (MB/sec) / sustainable throughput per partition (~10-20 MB/sec)
round up generously — partition count is cheap to over-provision up front,
expensive to change later (changing it reshuffles every key's partition assignment)
3. Storage for a retention window, before replication:
bytes/sec * retention period (seconds) = raw bytes needed
THEN multiply by replication factor for actual disk needed across the cluster
(a common interview mistake: forgetting the replication-factor multiplier entirely)
4. Consumer group parallelism ceiling:
max useful consumers in one group = partition count
(a consumer beyond this number sits idle — this is the single most
commonly misunderstood scaling limit in a Kafka system design, per this
module's interview-traps section)Applying these to a fresh, unfamiliar prompt in an interview: state your assumed numbers explicitly ("let's say 50,000 events per second, 500 bytes average, that's about 25 MB/sec"), then walk the formulas in order rather than jumping straight to a partition count. Interviewers are usually more interested in seeing you reason through the chain than in the exact final number, which is why stating assumptions out loud before calculating is worth more than a quietly-correct final answer.
| Quantity | Typical range to assume if not given | Where it showed up in this lesson |
|---|---|---|
| Sustainable throughput per partition | 10-20 MB/sec on typical hardware | Part 01's driver-location partition-count estimate |
| Replication factor for production data | 3, with min.insync.replicas=2 | Every worked design in this lesson |
| Retention for a firehose/raw topic | 24 hours to 7 days, unless compliance dictates otherwise | Part 01's raw location-ping topic |
| Retention for a compacted current-state topic | Effectively indefinite — compaction, not time, bounds its size | Part 01 and Part 04's compacted current-state topics |
Wrong Answers Candidates Commonly Give, and Why They're Wrong
8 Quick-Recall Questions — Short, Complete Answers
🎯 Key Takeaways
- ✓A Kafka system-design interview tests whether partitioning, delivery semantics, schema design, consumer group topology, and monitoring emerge naturally from reasoning through requirements — not whether you can recite a definition on cue.
- ✓The same underlying tools recur across very different systems: a keyed topic for per-entity ordering, a compacted topic for current-state materialization, separate consumer groups for independent fan-out, and a DLQ for isolating poison messages — recognize the pattern, then adapt the specifics to the prompt.
- ✓Durability and latency settings are deliberate trade-offs tied to the actual cost of losing or delaying a specific kind of data, never a uniform default — acks=1 was the right call for driver locations and acks=all plus idempotency keys was the right call for payments, in the very same module.
- ✓Some system boundaries are outside what Kafka itself can guarantee — a transaction cannot make an external HTTP call atomic, and replication factor cannot replicate across regions — knowing exactly where Kafka's guarantees end is what separates a strong answer from a memorized one.
- ✓This 26-module track went from "what is an event" to designing multi-region, latency-critical, exactly-once systems from scratch — that arc is the actual skill being tested in a senior Kafka interview, and it is now yours to apply.
🎉 Track Complete — All 26 Modules
You've completed the full Apache Kafka track — events to system design.
From your first event, topic, and partition through producers, consumers, brokers, delivery semantics, schemas, Kafka Streams, Connect, security, operations, managed cloud platforms, testing, and now full system-design synthesis — that is the complete arc of what a working Kafka engineer actually needs, end to end. Revisit any module as a reference whenever a real project calls for it; the vocabulary table and worked examples in this lesson are built specifically to be reused before your next interview, not read once and forgotten.
Quick check
Several teams each need every order event. How do you lay out consumers?
HELP US IMPROVE
How was this lesson?
Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.
Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.
