Delivery Semantics: At-Most, At-Least, Exactly-Once
What at-most-once, at-least-once, and exactly-once actually guarantee in Kafka, how the idempotent producer and transactions work mechanically, and how to choose the right semantics for a real workload instead of copying a default.
Delivery Semantics Are Promises About Loss and Duplication, Not About Speed
Every messaging system, Kafka included, has to answer one question honestly: when something goes wrong — a network blip, a crashed producer, a consumer that dies mid-processing — what happens to the message that was in flight at that moment? The answer is not one universal guarantee. It is a choice, made per producer and per consumer, among three distinct delivery semantics. Getting this choice right, and understanding exactly what it promises and does not promise, is one of the most consequential decisions in any Kafka-based system, because it directly determines whether a bug shows up as "we silently lost some data" or "we occasionally double-charged a customer."
At-most-once means a message is delivered zero or one times — never more, but sometimes not at all. This is what you get from a fire-and-forget producer: send, and move on without confirming. At-least-once means a message is delivered one or more times — never lost, but sometimes duplicated. This is what you get from a producer that retries until it receives an acknowledgement, and a consumer that processes a record before committing its offset. Exactly-once means a message is delivered — and, more precisely, its effect is applied — exactly one time: never lost, never duplicated, from the point of view of whoever is observing the result.
Beginner model: exactly-once is a mode you turn on, like a checkbox, and then every message is guaranteed to happen exactly once, everywhere, automatically.
Production model: at-most-once and at-least-once are properties of the transport — whether a producer waits for acknowledgement and retries, and whether a consumer commits before or after processing. Exactly-once is a much narrower, harder-won guarantee that Kafka provides at the transport layer through the idempotent producer and transactions (Part 03 and Part 04), and that only becomes true end-to-end exactly-once if the side effect your consumer produces — a database write, an API call, a file write — is also made idempotent or transactional. Kafka cannot make an external system idempotent on your behalf.
| Semantic | Can a message be lost? | Can a message be duplicated? | What guarantees it |
|---|---|---|---|
| At-most-once | Yes | No | Producer sends without waiting for, or without retrying on, acknowledgement failure. |
| At-least-once | No | Yes | Producer retries until acknowledged; consumer commits offsets only after processing. |
| Exactly-once (effectively-once) | No | No, as observed | Idempotent producer deduplicates retries; transactions make read-process-write atomic; or consumer-side idempotency absorbs any remaining duplicates. |
acks and min.insync.replicas determine whether a write that reaches the broker survives a broker failure. This module is about a different question: given that the write did or did not reach the broker, what does the producer do about acknowledgement and retries, and what does the consumer do about processing and offset commits — the layer where loss and duplication actually get introduced or prevented.Both Weaker Semantics Come From the Same Two Knobs
At-most-once and at-least-once are not exotic modes with their own configuration flag. They fall directly out of two ordinary decisions: does the producer wait for and retry on a failed acknowledgement, and does the consumer commit its offset before or after it finishes processing a record. Every Kafka pipeline is implicitly choosing one of these, whether or not the team building it thought about it explicitly.
At-most-once, on the producer side
A producer configured with acks=0, or one that simply does not check the result ofsend() (a pattern already flagged as dangerous in the producers module), does not know whether a write actually succeeded. If the network drops the request, or the broker rejects it, the message is gone and nothing retries it. This is at-most-once by construction: the message was sent zero or one times from the broker's point of view, and the producer has no way to tell which.
producer = Producer({'bootstrap.servers': 'broker:9092', 'acks': 0})
producer.send('metrics.pageviews', key=session_id, value=event)
# no callback, no future check
# if this send fails silently, the event is gone -- nobody retries it
# nobody even logs that it happenedAt-most-once, on the consumer side
A consumer that commits its offset immediately after poll() returns records — before actually processing them — is also at-most-once with respect to that processing. If the consumer crashes between the commit and finishing the work, Kafka believes those records are done. On restart, the consumer resumes past them. They are never reprocessed, and whatever partial or nonexistent work happened before the crash is the final state.
records = consumer.poll(timeout=1.0)
consumer.commit() # offset advances immediately
for record in records:
process(record) # if this crashes here, the record is skipped foreverAt-least-once, on the producer side
A producer with acks=all (or acks=1) and retries set to a high value will keep resending a batch until it receives a successful acknowledgement or exhausts its retry budget. This closes the loss gap, but opens a duplication gap: if the broker actually wrote the batch and the acknowledgement was lost on the way back — a dropped response packet, a timeout that fires just before the ack arrives — the producer has no way to distinguish "the write never happened" from "the write happened but I didn't hear about it," so it retries, and the broker ends up with the same logical message written twice.
At-least-once, on the consumer side
A consumer that processes records first and commits its offset only afterward is at-least-once: if it crashes after finishing the work but before the commit lands, the commit never happened, and on restart the consumer resumes from the last committed offset — reprocessing records it already handled. The work is never skipped, but it can happen more than once.
| Choice | Producer side | Consumer side |
|---|---|---|
| At-most-once | Fire-and-forget send, no retry on failure. | Commit offset before or without regard to processing outcome. |
| At-least-once | Retry until acknowledged (acks=1 or acks=all, retries > 0). | Commit offset only after processing succeeds. |
Kafka's Idempotent Producer Deduplicates Retries at the Broker
The retry-induced duplication described in Part 02 has a specific, mechanical fix built directly into Kafka: the idempotent producer, enabled with enable.idempotence=true (the default in current client versions when not explicitly disabled). It does not prevent retries — retries are still necessary for at-least-once delivery. Instead, it makes retries safe by letting the broker recognize and discard a retried write it has already committed.
Producer ID and per-partition sequence numbers
When an idempotent producer initializes, the broker assigns it a unique Producer ID (PID) for that session. Every batch the producer sends to a given partition carries a monotonically increasing sequence number, scoped to that (PID, partition) pair. The partition leader keeps track of the last sequence number it successfully wrote for each PID. When a new batch arrives, the broker compares its sequence number against what it has already seen: a sequence number it already wrote is a retry of an already-committed batch, and the broker discards it silently while still returning a success acknowledgement to the producer — which never even learns a duplicate was thrown away.
# Producer initializes: broker assigns PID = 7042
# Producer sends batch [M1, M2, M3] to orders-partition-2
# PID=7042, sequence=[301, 302, 303]
# Broker appends M1 M2 M3, records "PID 7042 last wrote seq 303 on partition 2"
# Broker sends ack -- but the ack is lost on the way back (network blip)
# Producer times out waiting for the ack, retries the same batch
# PID=7042, sequence=[301, 302, 303] -- identical sequence numbers
# Broker checks: "PID 7042 already wrote sequence 303 on this partition"
# Broker discards the retried batch WITHOUT writing it again
# Broker sends a success ack for the discarded retry
# Partition still contains exactly one copy of M1, M2, M3 -- no duplicateIt is worth being precise about the scope of this guarantee, because it is narrower than it sounds. The idempotent producer deduplicates retries within one producer session, againstone partition. The broker only tracks a small window of recent sequence numbers per (PID, partition) — not an unbounded history. And if the producer process itself crashes and restarts, it receives a brand-new PID on reconnect; the broker has no way to correlate the new PID with the old one, so any message the old producer instance sent but never confirmed before crashing could be resent by the new instance as a fresh, undeduplicated write.
| What idempotent producer does | What it does NOT do |
|---|---|
| Deduplicates retried batches within one producer session and partition. | Deduplicate across a producer restart — a new session gets a new PID. |
| Prevents reordering when combined with in-flight requests (up to 5 safely). | Guarantee atomicity across multiple partitions or topics — that requires transactions (Part 04). |
| Requires no application code changes — it is a broker/client protocol feature. | Make an external side effect (a database write, an API call) idempotent on your behalf. |
Transactions Make a Consume-Transform-Produce Cycle Atomic
The idempotent producer solves duplication for a single producer writing to a single partition. A very common Kafka pattern is more complex than that: a service reads a record from an input topic, transforms it, writes the result to an output topic, and needs its consumer offset on the input topic to advance — all as one unit. If the write to the output topic succeeds but the offset commit fails, or vice versa, a naive implementation ends up either reprocessing the same input record into a duplicate output record, or losing track of what was already processed. Kafka's transactional API exists specifically to make this three-part operation atomic: either all three things happen, or none of them do.
The transactional API, piece by piece
- ✓initTransactions() — called once at producer startup, registers the transactional.id and fences off any earlier, possibly-zombie producer instance using the same ID.
- ✓beginTransaction() — opens a new transaction before producing any records that belong to it.
- ✓send() / produce() — writes records as normal, but they are not visible to read_committed consumers until the transaction commits.
- ✓sendOffsetsToTransaction() — includes the consumer offset commit as part of the same transaction, instead of committing it separately through the consumer.
- ✓commitTransaction() — atomically makes every write and the offset commit visible together.
- ✓abortTransaction() — on any failure, rolls the whole transaction back; none of its writes become visible and the offset does not advance.
producer = Producer({
'bootstrap.servers': 'broker:9092',
'transactional.id': 'order-enrichment-service-1', # unique per producer instance
'enable.idempotence': True, # transactions require idempotence
})
producer.init_transactions()
consumer = Consumer({
'bootstrap.servers': 'broker:9092',
'group.id': 'order-enrichment-group',
'isolation.level': 'read_committed', # hide uncommitted/aborted writes
'enable.auto.commit': False, # offset commits happen inside the transaction
})
consumer.subscribe(['orders.raw'])while running:
msg = consumer.poll(timeout=1.0)
if msg is None:
continue
try:
enriched = enrich(json.loads(msg.value()))
producer.begin_transaction()
producer.produce('orders.enriched', value=json.dumps(enriched))
offsets = [{'topic': msg.topic(), 'partition': msg.partition(),
'offset': msg.offset() + 1}]
producer.send_offsets_to_transaction(offsets, consumer.consumer_group_metadata())
producer.commit_transaction()
# the write to orders.enriched AND the offset commit on orders.raw
# become visible together, atomically -- or neither does
except Exception as error:
producer.abort_transaction()
# offset was not committed -- this record will be re-read and retried
log_and_alert('transaction aborted', error)The consumer setting isolation.level=read_committed is the other essential half of this guarantee. Without it, a consumer reading orders.enriched would see records from transactions that later aborted — partial, thrown-away work becoming visible anyway. Withread_committed, the broker withholds records belonging to an open or aborted transaction until (and unless) that transaction commits; an aborted transaction's writes are never exposed to a read_committed consumer at all.
| isolation.level | What the consumer sees | Typical use |
|---|---|---|
| read_uncommitted (default in many clients) | Every write, including ones from transactions that later abort. | Non-transactional pipelines, or pipelines where seeing an eventually-aborted write briefly does not matter. |
| read_committed | Only writes from transactions that have actually committed; open or aborted transaction writes are withheld. | Any consumer downstream of a transactional producer where correctness depends on not seeing partial work. |
The Guarantee Stops at Kafka's Boundary — Your Side Effects Are Still Your Job
Part 03 and Part 04 describe real, mechanically enforced guarantees: exactly one copy of a message lands in a partition despite retries, and a read-process-write cycle across topics is atomic. But neither of these makes an arbitrary external side effect exactly-once. If your consumer's job is to call a payment API, write a row to a Postgres table, or send an email, Kafka's transactional guarantees cover the Kafka-to-Kafka part of the pipeline — they say nothing about whether that external call itself happens exactly once.
This is why practitioners increasingly prefer the term effectively-once over exactly-once: it is more honest about what is actually guaranteed. The message is delivered exactly once as far as Kafka's own bookkeeping is concerned. Whether the effect of processing that message happens exactly once in the outside world depends on whether that external system was also made idempotent or included in a transaction of its own — something Kafka cannot do for you, because it has no visibility into a payment gateway's internal state or a database's transaction log.
Kafka's transactional guarantee covers this boundary:
[ input topic ] --consume--> [ processing ] --produce--> [ output topic ]
\
-- atomic with --> [ offset commit ]
It does NOT cover this boundary:
[ output topic ] --consume--> [ processing ] --side effect--> [ external system ]
(payment API, DB write, email)
A consumer reading orders.enriched and calling a payment API on each record
can still call that API twice for the same record if it crashes between
the API call succeeding and its own offset commit -- Kafka transactions
covered the FIRST boundary, not this one.When Kafka Transactions Aren't in Play, Make the Side Effect Itself Idempotent
Most consumers writing to something outside Kafka — a relational database, a search index, a third-party API — cannot lean on Kafka transactions at all, because the external system is not a participant in Kafka's transaction protocol. For these, at-least-once plus consumer-side idempotency is the standard, practical way to get effectively-once behavior without needing Kafka's transactional machinery.
Pattern 1 — dedup by a stable business key
If every record carries a natural, stable identifier — an order ID, a payment ID, an idempotency key the upstream system generated — the consumer can check whether it has already processed that identifier before doing the side effect, and skip it if so. The dedup check itself needs to be reliable, which usually means backing it with a database unique constraint or a dedicated already-processed table, not an in-memory set that a restart would wipe clean.
CREATE TABLE processed_events (
event_id TEXT PRIMARY KEY, -- the business key from the record
processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- inside the consumer, for each record:
BEGIN;
INSERT INTO processed_events (event_id) VALUES (:event_id)
ON CONFLICT (event_id) DO NOTHING;
-- if 0 rows were inserted, this event_id was already processed -- skip the side effect
-- if 1 row was inserted, this is genuinely new -- do the side effect, in the same
-- database transaction if the side effect is also a database write
COMMIT;Pattern 2 — upsert instead of insert
For side effects that write current state rather than append a log of events, an upsert (insert if absent, overwrite if present, keyed by the record's natural key) is often naturally idempotent without needing an explicit dedup table at all: writing the same inventory-count update twice produces the same final row either way, because the second write simply overwrites the first with identical data. This only works when the side effect is a pure "set current value" operation, not an operation with cumulative effect like "increment the count by this amount" — an increment applied twice is not idempotent even as an upsert.
-- SAFE: upsert sets an absolute value, replaying it is harmless
INSERT INTO inventory (sku, quantity_on_hand, updated_at)
VALUES (:sku, :quantity, :event_timestamp)
ON CONFLICT (sku) DO UPDATE
SET quantity_on_hand = EXCLUDED.quantity_on_hand,
updated_at = EXCLUDED.updated_at
WHERE inventory.updated_at < EXCLUDED.updated_at; -- guard against out-of-order replays too
-- UNSAFE: an increment is not idempotent -- replaying it double-counts
UPDATE inventory SET quantity_on_hand = quantity_on_hand - :quantity_sold
WHERE sku = :sku;
-- if this record is processed twice, the sku loses quantity_sold TWICE| Pattern | When it works | When it does not |
|---|---|---|
| Dedup table keyed by business key | Any event with a stable, unique identifier; works for both event-log and current-state side effects. | No usable stable key exists, or the key is only unique within a short window. |
| Upsert on current state | The side effect sets an absolute value (current price, current status, current quantity). | The side effect is a relative change (increment, decrement, append) — replays double-apply. |
| Idempotency key sent to an external API | The external API explicitly supports an idempotency key parameter (many payment APIs do). | The external API has no idempotency concept — a duplicate call is indistinguishable from a new one. |
Tracing One Bug Through All Three Semantics
Concrete numbers and a concrete bug make the abstract three-way distinction much easier to reason about under pressure. Consider a payments service: a consumer reads payments.authorizedevents and, for each one, calls a downstream billing API to actually charge the customer's card.
consumer.subscribe(['payments.authorized'])
while running:
records = consumer.poll(timeout=1.0)
for record in records:
charge_customer_card(record.customer_id, record.amount) # NOT idempotent
consumer.commit() # after processing -- at-least-once by construction
# scenario: charge_customer_card() succeeds, but the consumer process
# crashes before consumer.commit() executes
# on restart, the consumer resumes from the LAST COMMITTED offset,
# which is BEFORE this record -- it is re-read and re-processed
# charge_customer_card() is called a second time for the same authorization
# the customer is charged twiceNow trace how each of the three semantics would have handled this exact crash.
| Semantic | What happens on this crash | Does it prevent the double charge? |
|---|---|---|
| At-most-once (commit before processing) | The offset would have already advanced before the charge call — the crash loses nothing to reprocess, but if the crash had instead happened before the charge call completed, the customer would never be charged at all. | Prevents the duplicate, but by risking silent loss instead — not an acceptable trade for a payment. |
| At-least-once, no consumer idempotency (as shown above) | The record is reprocessed on restart; charge_customer_card() runs twice. | No — this is exactly the bug. |
| At-least-once + idempotency key sent to the billing API | The record is reprocessed, but the billing API recognizes the same idempotency key and returns the original charge result without charging again. | Yes — the duplicate delivery still happens, but its effect does not. |
| Kafka transactions (offset commit atomic with a Kafka write) | Only helps if the side effect is itself a Kafka write in the same transaction — a call to an external billing API is outside the transaction boundary (Part 05) and is not protected by this alone. | Not by itself — transactions do not reach an external API call. |
The fix that actually holds up in production combines two of these: at-least-once delivery (so a payment is never silently dropped), plus an idempotency key passed to the billing API, generated from a stable field on the event itself — the authorization ID, not a value generated fresh on each processing attempt, since a freshly generated key would be different on the retry and defeat the whole purpose.
for record in records:
charge_customer_card(
customer_id=record.customer_id,
amount=record.amount,
idempotency_key=record.authorization_id, # stable across retries and reprocessing
)
consumer.commit()
# if this crashes before commit, the record is reprocessed --
# but charge_customer_card() with the same idempotency_key is now a safe no-op
# on the billing API's side for the second callThe Choice Is a Business Decision, Not a Performance Setting
The right delivery semantics for a given topic depend entirely on what the cost of loss versus the cost of duplication actually is for that specific data — the same "business decision, not a performance setting" framing the producers module applies to acks. A blanket "always use exactly-once" policy wastes latency and coordination overhead on data where a duplicate or an occasional dropped record costs nothing. A blanket "always use at-most-once for speed" policy is a silent data-loss incident waiting to happen on anything with real consequence.
| Workload | Cost of losing a record | Cost of a duplicate | Recommended semantics |
|---|---|---|---|
| Metrics and application logs | Low — a gap in a dashboard is rarely investigated individually. | Low — an occasional double-counted metric is statistically invisible in aggregate. | At-most-once is often acceptable; at-least-once with no dedup is fine too. |
| Clickstream / analytics events | Low to moderate — matters in aggregate, not per-event. | Low — duplicate events wash out in large-scale analytics. | At-least-once, no consumer idempotency required in most cases. |
| Order and inventory state changes | High — a lost update corrupts current-state accuracy. | Moderate to high — a duplicate can double-decrement stock. | At-least-once + consumer-side idempotency (dedup key or upsert pattern from Part 06). |
| Payments and billing | Unacceptable — a lost charge is a revenue and trust problem. | Unacceptable — a duplicate charge is a direct customer harm and compliance issue. | At-least-once + idempotency key on the side effect, or full Kafka transactions where the whole pipeline stays inside Kafka. |
| Multi-topic read-process-write pipelines (Kafka Streams-style) | Unacceptable if the pipeline is the system of record. | Unacceptable for the same reason. | Kafka transactions (Part 04) — the pipeline never leaves Kafka, so transactional exactly-once fully applies. |
A short decision process
- ✓Does the pipeline stay entirely inside Kafka (consume from one topic, produce to another)? If yes, Kafka transactions give you a real, mechanically enforced exactly-once guarantee — use them.
- ✓Does the consumer call an external system with a side effect? If yes, transactions cannot reach it — you need at-least-once delivery plus an idempotent side effect (dedup key, upsert, or an idempotency key on the external API).
- ✓Is duplication genuinely harmless for this specific data, evaluated honestly rather than assumed? If yes, plain at-least-once with no dedup logic is simpler and cheaper — do not build idempotency machinery nothing needs.
- ✓Is occasional silent loss genuinely acceptable, evaluated with the same honesty? Only then is at-most-once a reasonable, deliberate choice — never a default reached for out of laziness about acknowledgements.
enable.idempotence=true on the producer has essentially no downside — it prevents a specific, narrow class of duplicate writes at the broker with negligible overhead, and it is required for transactions anyway. There is very little reason to leave it disabled on any modern producer, regardless of which end-to-end semantics the workload ultimately needs.Delivery Semantics Are Only Real If You Can Verify Them
Every semantics choice covered so far is a design-time decision. In production, the only way to know whether that design decision is actually holding is to measure it directly — loss and duplication are both silent by default, and a pipeline that has been quietly losing or duplicating records for weeks will not announce itself unless something is specifically watching for it.
Detecting loss
The most reliable way to detect loss end to end is a count reconciliation: track how many records a producer believes it successfully sent (from delivery callbacks or futures, per the producers module's callback-handling guidance) against how many records a consumer, or a downstream sink, believes it actually processed, over the same window of time. A persistent, unexplained gap between those two counts is the clearest possible signal that at-most-once behavior is happening somewhere it should not be — usually an unhandled send failure, a producer using acks=0 on data that needed stronger durability, or a consumer silently dropping records it fails to parse instead of routing them to a dead-letter topic.
producer emits a metric: events_sent_total (incremented only inside a
successful delivery callback, never optimistically before send())
consumer emits a metric: events_processed_total (incremented only after
a record's side effect has durably succeeded)
alert if:
events_sent_total(window) - events_processed_total(window) > expected_lag_tolerance
# a small, bounded gap is normal (in-flight processing lag);
# a gap that keeps growing, or never closes, points to real lossDetecting duplication
Duplication is often easier to catch than loss, precisely because at-least-once systems are expected to produce some duplicates under specific failure conditions — the question is whether the rate is bounded and explainable, or unbounded and silently corrupting downstream state. For a topic with a stable business key, counting distinct keys processed versus total records processed over a window gives a direct duplication rate; a consumer relying on the dedup or upsert patterns from Part 06 should also emit a metric specifically for how often the dedup check actually caught something, since a dedup path that never fires might mean duplicates genuinely aren't happening, or might mean the dedup logic itself is broken and silently letting everything through.
| Signal | What it measures | What an anomaly suggests |
|---|---|---|
| sent count vs. processed count gap | Records the producer believes succeeded vs. records the consumer believes it finished. | A persistent, growing gap suggests loss somewhere between send and durable processing. |
| distinct keys vs. total records ratio | How many logically distinct events are hiding behind a larger raw record count. | A ratio far from 1:1 where it should be close suggests unexpected duplication. |
| dedup-hit rate | How often a consumer's own idempotency check actually rejects a repeat. | A rate of exactly zero over a long window is suspicious — either duplicates truly never happen (verify independently) or the dedup check is silently broken. |
| dead-letter topic volume | Records a consumer could not process and routed aside instead of silently dropping. | A rising trend often points to a schema mismatch or an upstream data-quality regression, not a delivery-semantics bug per se — but it is the alternative to silent at-most-once loss. |
A Duplicate or a Retry Can Also Reorder Records If You Are Not Careful
Delivery semantics are usually discussed purely in terms of loss and duplication, but a closely related failure mode rides along with retries: reordering. If a producer has more than one request in flight to the same partition at once, and an earlier request needs to be retried while a later one already succeeded, the retried (earlier) record can land after the (later) one that originally succeeded first — even though nothing about acks or idempotence was misconfigured.
max.in.flight.requests.per.connection = 5 (default before idempotence became default-on)
enable.idempotence = false
producer sends batch A (records 1-3), batch B (records 4-6) -- both in flight at once
batch A's first attempt times out on the network (but the broker DID receive it)
batch B succeeds immediately, writes records 4-6
batch A is retried, succeeds on retry, writes records 1-3 -- AFTER records 4-6
partition ends up with: [4, 5, 6, 1, 2, 3] instead of [1, 2, 3, 4, 5, 6]
-- this is a reordering bug, not a loss or duplication bug, but it can be
just as damaging for any consumer relying on per-key event orderingThe idempotent producer (Part 03) closes this gap as a side effect of the same sequence-number mechanism that prevents duplication: when idempotence is enabled, the broker also uses those sequence numbers to detect and reject out-of-order batches, forcing the producer to resend them in the correct order rather than silently accepting a batch that arrived out of sequence. This is one of the reasons enable.idempotence=true is now the client default rather than an opt-in setting — it fixes both duplication and a class of reordering bugs with the same underlying mechanism, at negligible cost.
| Configuration | Reordering risk on retry |
|---|---|
| enable.idempotence=false, max.in.flight > 1 | Real risk — a retried earlier batch can land after a later batch that succeeded first, exactly as shown above. |
| enable.idempotence=false, max.in.flight = 1 | No reordering, but a much lower ceiling on throughput since only one request can be outstanding to a partition at a time. |
| enable.idempotence=true, max.in.flight up to 5 | No reordering — the broker enforces correct sequence ordering per (PID, partition), safe at higher concurrency. |
Stream Processing Frameworks Build on the Same Primitives, Not a New Guarantee
Kafka Streams (and stream-processing layers built on top of it, like ksqlDB) advertise "exactly-once processing" as a configuration flag — processing.guarantee=exactly_once_v2 — which can make it sound like a fundamentally different, stronger guarantee than anything covered in this module. It is not. Under the hood, Kafka Streams' exactly-once mode is implemented using exactly the transactional producer mechanics from Part 04: every state update and every output record a stream task produces is wrapped in a Kafka transaction, atomic with the consumer offset commit for the input record that triggered it.
processing.guarantee = exactly_once_v2
for each input record consumed by a stream task:
begin transaction
update local state store (if the topology is stateful, e.g. an aggregation)
write the state store's changelog topic update
produce any output records to downstream topics
commit the input topic's offset
commit transaction
# all of the above becomes visible together, or none of it does --
# this is Part 04's read-process-write atomicity, applied automatically
# by the Streams runtime instead of hand-written by application codeThe same Part 05 boundary still applies: if a Kafka Streams topology's final step calls out to an external system — writes to a database via a custom sink, calls a third-party API inside aProcessor — that call is not covered by exactly_once_v2's transactional guarantee, for exactly the same reason a hand-written transactional producer's guarantee stops at Kafka's boundary. Kafka Connect sink connectors (a common way data leaves a Streams topology) handle this with their own connector-specific idempotency strategies, which is a detail worth checking per-connector rather than assuming automatically inherits Streams' internal guarantee.
| Configuration | What it actually guarantees | What it does not reach |
|---|---|---|
| processing.guarantee=at_least_once (default in older versions) | Records are never lost, but reprocessing after a failure can produce duplicate state updates or output records. | Any external side effect the topology triggers, same as at-least-once anywhere else. |
| processing.guarantee=exactly_once_v2 | State store updates, changelog writes, downstream produces, and the input offset commit are all atomic together, using Kafka transactions under the hood. | External sinks and side effects outside the Kafka transaction boundary — same limitation as Part 05, just automated for the Kafka-internal part. |
Chaos-Style Failure Injection Is the Only Reliable Way to Verify a Semantics Choice
Everything this module has covered so far is a design and a mental model. None of it substitutes for actually verifying, before shipping, that a pipeline behaves the way its delivery-semantics design claims it does under the specific failure conditions that matter — a producer retry, a consumer crash mid-batch, a broker restart during an in-flight transaction. Code review can confirm a design intends to be at-least-once with idempotent handling; only forcing the actual failure and observing the result confirms it really is.
A minimal test matrix worth running before trusting a pipeline's semantics
- ✓Kill the consumer process mid-batch, after some records in the current poll() have been processed but before the offset commit — confirm the expected records are reprocessed, and confirm the idempotency mechanism (Part 06) actually prevents a duplicate side effect on replay.
- ✓Kill the producer process after a send() call returns but before its delivery callback fires — confirm the application-level retry or restart behavior does not silently drop the in-flight record.
- ✓Introduce a network partition between the producer and the current partition leader mid-write, then heal it — confirm the producer's retry-and-refresh-metadata behavior (from the producers module) resumes correctly rather than stalling or duplicating.
- ✓For a transactional pipeline, kill the producer process between beginTransaction() and commitTransaction() — confirm the transaction is correctly aborted, that read_committed consumers never see the partial writes, and that the input offset was not advanced.
- ✓Deliberately replay a batch of already-processed records from an earlier offset — confirm the consumer's dedup or upsert logic produces the same final state as processing them once, not a corrupted or doubled one.
# the single most valuable automated test for at-least-once + idempotency:
# process the same batch of records TWICE, back to back, and assert
# the observable end state after the second run is identical to the
# end state after the first run
def test_replay_is_idempotent():
records = load_fixture_batch("orders_batch_1.json")
process_batch(records)
state_after_first_run = snapshot_downstream_state()
process_batch(records) # exact same records, simulating a Kafka replay
state_after_second_run = snapshot_downstream_state()
assert state_after_first_run == state_after_second_run
# if this fails, the consumer's idempotency claim is false --
# better to find out here than from a duplicate-charge incidentTurning This Module Into a Repeatable Checklist
Every concept in this module ultimately exists to answer one practical question a team faces when standing up a new producer or consumer: what should actually be configured, and why. Rather than re-deriving the reasoning from scratch each time, it helps to have a short, concrete checklist to run through before a new topic's producers and consumers go to production.
- ✓What is the real cost of losing one record on this topic, stated concretely rather than in the abstract — a support ticket, a wrong dashboard number, a missed payment? This determines whether at-most-once is even on the table (Part 08).
- ✓What is the real cost of processing one record twice? If genuinely zero, plain at-least-once with no dedup logic is the right, simplest answer — do not build idempotency machinery a workload does not need.
- ✓Is enable.idempotence=true set on every producer for this topic? There is essentially no reason it should not be, regardless of which broader semantics the workload needs (Part 03, Part 08).
- ✓Does the consumer's side effect leave Kafka entirely (a database write, an external API call)? If so, Kafka transactions cannot reach it, and the plan needs an explicit idempotency mechanism from Part 06 — a dedup key, an upsert, or an idempotency key passed to the external system.
- ✓Does the pipeline stay entirely inside Kafka (topic to topic, no external side effect)? If so, Kafka transactions (Part 04) are usually worth their added latency for the correctness they buy.
- ✓Is there a dead-letter topic (or equivalent) so an unprocessable record gets set aside instead of silently dropped, which would quietly reintroduce at-most-once behavior into an at-least-once design (Part 09)?
- ✓Is there a metric or reconciliation check that would actually surface silent loss or unexpected duplication, rather than relying on nobody noticing (Part 09)?
- ✓Has the idempotency claim actually been tested by replaying the same batch of records twice and comparing the resulting state, not just asserted in a design doc (Part 12)?
## Delivery semantics review — <topic-name>
Cost of losing a record: <concrete description>
Cost of duplicating a record: <concrete description>
Chosen semantics: [ ] at-most-once [ ] at-least-once [ ] transactional exactly-once
enable.idempotence=true? [ ] yes [ ] no -- justify:
External side effect present? [ ] yes -- idempotency mechanism: _______
[ ] no -- entire pipeline stays in Kafka
Dead-letter topic configured? [ ] yes [ ] no -- justify:
Loss/duplication monitoring? [ ] yes, via: _______ [ ] no -- justify:
Idempotency replay-tested? [ ] yes [ ] no -- schedule:None of This Is Free — Weigh the Overhead Honestly Against the Guarantee
It would be easy to close this module concluding that idempotent producers and transactions should simply always be on, everywhere, since Part 03 already noted idempotence has essentially no downside. Idempotence really is close to free. Transactions are a different story, and treating them as a costless upgrade over at-least-once leads to teams reaching for them on workloads where the overhead is not worth what it buys.
Where the overhead actually comes from
A transactional producer pays a coordination cost on every commitTransaction() call — a round trip to the transaction coordinator broker to record the commit marker, on top of the normal write path. A read_committed consumer pays a latency cost too: it must withhold records belonging to an open transaction until that transaction resolves, which means a slow-committing producer directly adds latency to every downstream consumer waiting on read_committed visibility, not just to the producer's own throughput.
| Mechanism | Overhead | Worth it when |
|---|---|---|
| Idempotent producer (enable.idempotence=true) | Negligible — a small header per batch, a sequence-number check on the broker. | Almost always — there is little reason to leave this off on a modern producer. |
| Kafka transactions | Real — coordinator round trips per commit, added latency for read_committed consumers waiting on transaction resolution, more moving parts to monitor and reason about on failure. | A read-process-write pipeline that stays entirely inside Kafka, where atomicity across the write and the offset commit is a genuine correctness requirement, not just a nice-to-have. |
| Consumer-side dedup table or upsert logic | A database write or lookup added to the processing path, plus the ongoing cost of retaining and maintaining the dedup table (Part 06's error-library entry on unbounded growth). | Any workload with an external side effect where duplication has a real cost — most business-critical pipelines with a side effect outside Kafka. |
The practical implication is that a pipeline should not reach for full Kafka transactions just because it technically could, if the pipeline's real bottleneck to correctness is an external side effect that transactions cannot reach anyway (Part 05). In that case, the transactional overhead is paid without buying the guarantee that actually matters for the workload — consumer-side idempotency on the external call is both cheaper and the thing that actually closes the gap.
The Whole Module, Condensed Into One Reference
Every mechanism covered above answers a specific question about what happens to a message under failure. It helps to have all of them side by side, as a single reference to check against when a new pipeline's requirements are being scoped out, rather than needing to re-read the full module each time.
| Question | Where it is answered | The short answer |
|---|---|---|
| Can a message be silently lost? | Part 01, Part 02 | Only under at-most-once — a producer that does not retry on failure, or a consumer that commits before processing. |
| Can a message be duplicated? | Part 01, Part 02 | Yes, under at-least-once, unless the duplication's effect is absorbed by idempotent handling. |
| How does Kafka prevent a retried write from duplicating? | Part 03 | Producer ID plus per-partition sequence numbers, tracked by the broker, deduplicating within one producer session. |
| How does Kafka make a multi-topic read-process-write cycle atomic? | Part 04 | Transactions — beginTransaction/sendOffsetsToTransaction/commitTransaction, paired with isolation.level=read_committed downstream. |
| Does exactly-once reach an external database or API call? | Part 05 | No — Kafka's guarantee stops at Kafka's own boundary; the external call needs its own idempotency mechanism. |
| How do you make an external side effect idempotent without transactions? | Part 06 | Dedup by a stable business key, or an upsert that sets an absolute value rather than a relative change. |
| How do you pick the right semantics for a given workload? | Part 08 | Weigh the real cost of loss against the real cost of duplication for that specific data, honestly, not by copying another topic's defaults. |
| How do you know loss or duplication is actually happening in production? | Part 09 | Sent-vs-processed count reconciliation, dedup-hit rate, and a dead-letter topic instead of silent drops. |
| Can retries reorder messages even without loss or duplication? | Part 10 | Yes, without idempotence enabled and multiple in-flight requests — idempotent producer fixes this too. |
| Does Kafka Streams' exactly_once_v2 add a new guarantee? | Part 11 | No — it automates the same transactional mechanics from Part 04, with the same external-boundary limitation from Part 05. |
| How do you verify an idempotency claim is actually true? | Part 12 | Replay the same batch of records twice and assert the resulting state is identical — automate this in CI. |
Five Misconceptions About Delivery Semantics
What This Looks Like on Day One
At Plaid, a team building a transaction-categorization pipeline discovers during a load test that restarting the categorization service mid-run produces a small number of duplicate category-assignment writes to their downstream database. The consumer was using at-least-once delivery with no consumer-side idempotency — a defensible choice for many workloads, but not for one writing financial transaction metadata that other services trust as authoritative. The fix is not to chase exactly-once through Kafka transactions, since the actual side effect is a database write outside Kafka entirely; it is adding a dedup table keyed on transaction ID, per Part 06's Pattern 1, so a reprocessed record becomes a safe no-op instead of a duplicate row.
At Shopify, a checkout team designs a strictly Kafka-to-Kafka pipeline: an order-validation service reads from orders.submitted, enriches each order with inventory and tax data, and writes to orders.validated, which downstream fulfillment services consume. Because this entire pipeline never leaves Kafka — no external database write, no API call — the team reaches for full Kafka transactions exactly as described in Part 04: the enrichment write and the input offset commit happen atomically, and every downstream consumer sets isolation.level=read_committed. This is the one case in the whole system where true exactly-once, not just effectively-once, is both achievable and worth the added transactional latency.
In a system design interview, a candidate is asked to design an event pipeline for a ride-hailing app's fare-charging flow and says "I'll just turn on exactly-once semantics for the whole thing." The interviewer presses: "the charge happens through a third-party payment processor's REST API — how does Kafka's exactly-once semantics reach that call?" The strong answer recognizes, per Part 05, that it does not — Kafka's transactional guarantee stops at Kafka's own boundary, and the actual fix for the fare-charging call is at-least-once delivery combined with an idempotency key derived from the ride ID, sent to the payment processor's API, exactly as worked through in Part 07.
5 Interview Questions — With Complete Answers
Mistakes Beginners Make Constantly
Errors You Will Hit — And Exactly Why They Happen
🎯 Key Takeaways
- ✓At-most-once can silently lose data but never duplicates it; at-least-once never loses data but can duplicate it; exactly-once (more honestly, effectively-once) aims for neither, as observed by the consumer.
- ✓The idempotent producer (enable.idempotence=true) uses a Producer ID and per-partition sequence numbers to let the broker discard retried writes it already committed — but only within one producer session, against one partition.
- ✓Kafka transactions make a read-process-write cycle across topics atomic with the consumer offset commit, using initTransactions/beginTransaction/sendOffsetsToTransaction/commitTransaction, paired with isolation.level=read_committed on downstream consumers.
- ✓Kafka's exactly-once guarantees stop at Kafka's own boundary — an external side effect like a database write or an API call needs its own idempotency mechanism, since Kafka cannot make an outside system idempotent on your behalf.
- ✓Consumer-side idempotency without transactions usually means one of two patterns: dedup by a stable business key backed by a unique constraint, or an upsert that sets an absolute value rather than applying a relative change like an increment.
- ✓Choosing delivery semantics is a business decision weighing the real cost of loss against the real cost of duplication for that specific data — not a performance setting to copy from another topic without re-evaluating.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.