Retention and Log Compaction

Kafka Deletes Old Data By Default — It Is Not a Database

A very common mental model, especially for engineers coming from traditional databases, is that Kafka is a durable store that keeps everything you write to it forever, the way a table in Postgres keeps every row until you explicitly delete it. That model is wrong, and it matters because it leads teams to build systems that quietly assume history is available long after Kafka has already deleted it.

Every Kafka topic has a retention policy, controlled primarily by two settings:retention.ms (how long, in milliseconds, records are kept before becoming eligible for deletion) and retention.bytes (a size cap per partition, after which the oldest data is deleted regardless of age). The out-of-the-box default for retention.ms is seven days. Unless a topic is explicitly configured otherwise, data written to it today will be gone in a week.

# Default retention — 7 days, whichever limit is hit first also applies
retention.ms=604800000          # 7 days in milliseconds
retention.bytes=-1              # -1 means "no size limit", so time is the only bound here

# A topic explicitly kept longer, e.g. for compliance/audit replay
retention.ms=2592000000         # 30 days

# A topic explicitly bounded by size instead of time (e.g. high-volume debug topic)
retention.bytes=10737418240     # 10 GB per partition — oldest data deleted once exceeded

# Both set together — whichever threshold is reached first triggers deletion
retention.ms=604800000
retention.bytes=53687091200     # 50 GB per partition

Whichever of these two limits is reached first is the one that triggers deletion. If a topic has both a time limit and a size limit configured, a sudden burst of volume can cause the size limit to delete data well before the time limit would have. This is a frequent source of "why is my data gone earlier than I expected" incidents — someone configured both without realizing the size limit was the tighter constraint under real production volume.

It's worth sizing retention against a real, concrete number rather than picking a round figure like "7 days" out of habit. The right question is: how long does the slowest legitimate consumer of this topic need to be able to catch up after downtime, and how far back does replay ever realistically need to reach — a nightly batch job that only ever reads the last 24 hours needs far less retention than a topic that occasionally needs a full historical reprocess for a new consumer being onboarded. Retention set shorter than the real requirement causes silent, permanent data loss the moment a consumer falls behind by more than the window; retention set far longer than necessary just spends disk budget for no benefit.

Deletion Happens by Segment, Not by Individual Record

Understanding how retention is physically implemented explains a lot of Kafka's behavior that otherwise seems surprising. A partition's log is not one giant file that Kafka scans record by record looking for expired entries. It is split into segments — fixed-size files, commonly capped around a gigabyte by default, plus companion index files. Only one segment per partition, the active segment, is ever being written to; every other segment is closed and immutable.

The broker's log cleaner (or, for time-based deletion specifically, a background retention thread) periodically checks each closed segment's newest record timestamp. If that timestamp is older than the retention window, the entire segment file becomes eligible for deletion and is removed from disk as one operation. This means deletion is granular at the segment level, not the individual record level — there is no operation in Kafka that deletes "just this one record" out of the middle of a segment while keeping the rest.

# Partition on disk, retention.ms = 7 days, "today" = day 10

/data/kafka/freshcart.clicks-2/
    00000000000000000000.log   # covers day 1 - day 2 records  → newest record: day 2
    00000000000000480210.log   # covers day 2 - day 4 records  → newest record: day 4
    00000000000001102873.log   # covers day 4 - day 7 records  → newest record: day 7
    00000000000001987340.log   # covers day 7 - day 10 records → newest record: day 10 (ACTIVE)

# retention boundary: anything with newest-record-timestamp older than day 3 (10 - 7) is eligible
# (using day 10 as "now" and a 7-day window for this illustration)

# segment 1 (newest record: day 2) → ENTIRELY past the boundary → deleted as one file
# segment 2 (newest record: day 4) → newest record still within the window → KEPT ENTIRELY
#     even though this segment also contains some day-2 and day-3 records that are
#     individually "expired" by the 7-day rule — they are NOT deleted separately,
#     because the segment as a whole is not yet eligible
# segment 3, 4 → clearly within the window, kept

# Consequence: a record can sit on disk somewhat past its "logical" retention age
# if it shares a segment with newer records. Retention is a segment-boundary
# approximation of the configured time window, not an exact per-record guarantee.

This also explains why shrinking retention.ms on a topic doesn't immediately free disk space in a smooth, proportional way. Space is freed in segment-sized chunks whenever a whole segment's newest record finally ages past the new boundary — not continuously as each record crosses the line.

The delete Policy: Time and Size, Nothing About Keys

The behavior described in Parts 01 and 02 — segments aging out based on time or size — is controlled by a topic-level setting called cleanup.policy, and the value that enables this behavior is cleanup.policy=delete, which is the default for ordinary topics. Under this policy, Kafka has no concept of "keys" at all when deciding what to remove. It only looks at how old a segment is (or how much total data has accumulated) — every record, regardless of its key or whether other records share that key, is deleted purely by age or size.

This is the right policy for what you'd intuitively call an "event stream" — a sequence of discrete things that happened, where each event is meaningful on its own and there is no notion of one event "superseding" another. A page-view event, a payment-captured event, a sensor reading, a click — each is a fact about a specific moment in time. Once that time window has passed and the data has been consumed and/or exported elsewhere, there's nothing more that record is "for."

cleanup.policy=delete
retention.ms=604800000   # 7 days

# freshcart.clicks — key = session_id, value = {page, timestamp, ...}

# offset 0: key=session_A  value={page:"/home", ts: day 1}
# offset 1: key=session_B  value={page:"/cart", ts: day 1}
# offset 2: key=session_A  value={page:"/checkout", ts: day 1}

# After 7 days pass and these records' segment ages out:
# ALL THREE are deleted together, once their segment crosses the retention boundary.
# Kafka does NOT look at the fact that session_A appears twice and keep only
# the newer session_A record — delete policy has no per-key awareness whatsoever.
# It is purely time/size-based, full stop.

The compact Policy: Keep Only the Latest Value Per Key, Forever

Some data isn't a stream of independent events — it's a series of updates to the current state of something. A customer's current shipping address. A product's current price. A user's current account-status flag. For data like this, what you actually want to retain is not "everything that ever happened," but "the most recent value for every key, indefinitely," regardless of how old that latest value is. Time-based retention is the wrong tool here — it would eventually delete even the single most recent, still-current value once it aged past the retention window.

cleanup.policy=compact is built exactly for this. Instead of deleting whole segments by age, the log cleaner periodically scans a compacted topic's older segments and removes any record whose key has a newer record later in the log — keeping only the record with the highest offset for each key. The result is a log that, over time, shrinks down to hold at most one record per distinct key: the latest one.

# freshcart.product-prices — cleanup.policy=compact
# key = product_id, value = current price snapshot

# BEFORE compaction (chronological order, by offset):
# offset 0: key=P1001  value={price: 18900}
# offset 1: key=P1002  value={price: 67500}
# offset 2: key=P1001  value={price: 19500}   ← price changed
# offset 3: key=P1003  value={price: 23400}
# offset 4: key=P1002  value={price: 69000}   ← price changed again
# offset 5: key=P1001  value={price: 21000}   ← price changed a third time

# AFTER compaction — the log cleaner keeps only the HIGHEST offset per key:
# offset 3: key=P1003  value={price: 23400}   ← only version that ever existed
# offset 4: key=P1002  value={price: 69000}   ← latest of two versions
# offset 5: key=P1001  value={price: 21000}   ← latest of three versions

# offsets 0, 1, 2 are gone — not because they aged out by time, but because
# a NEWER record for the same key made them obsolete

# A new consumer reading this topic from the beginning after compaction sees
# exactly one record per product — a complete, current snapshot of every
# product's price, reconstructed just by reading the log start to finish.

This pattern — a compacted topic acting as a "changelog" that any consumer can replay from the beginning to rebuild current state — is the foundation of Kafka Streams and ksqlDB state stores. When a streaming aggregation needs to recover after a crash or restart, it doesn't need to re-process the entire history of raw events; it rebuilds its in-memory state by replaying the much smaller compacted changelog topic that already holds just the latest aggregated value per key.

The head and tail of a compacted log — and why compaction isn't instantaneous

A compacted partition is conceptually split into two regions. The tail is the portion the log cleaner has already compacted — guaranteed to hold at most one record per key. The head is the active segment (and possibly a few recent segments) still being written to, which the cleaner has not processed yet and which can still contain multiple records for the same key. This means a consumer reading a compacted topic in real time can briefly see more than one value for a key before that region is compacted — compaction is a background process running on a cycle, not an instantaneous rewrite on every new record.

# The log cleaner doesn't compact continuously on every write — it runs
# periodically, triggered by a "dirty ratio" threshold per partition:
#
#   dirty ratio = (bytes in the uncompacted "head" region)
#               / (total bytes in the partition)
#
# min.cleanable.dirty.ratio = 0.5 (default)
#
# Once the uncompacted portion exceeds 50% of the partition's total size,
# the log cleaner schedules a compaction pass for that partition.
#
# Lowering min.cleanable.dirty.ratio (e.g. to 0.1) triggers compaction more
# often — keeps the topic smaller and closer to true steady-state, at the
# cost of more constant background I/O from the cleaner.
#
# Raising it (e.g. to 0.8) compacts less often — less cleaner overhead, but
# the topic can temporarily grow larger than the steady-state estimate from
# Part 09 before the next pass catches up.

Deleting a Key From a Compacted Topic: The Tombstone Record

Log compaction as described so far only ever keeps the latest value per key — it has no way to represent "this key should be removed entirely." If a customer closes their account, or a product is discontinued, you need a way to eventually make that key disappear from the compacted topic altogether, not just stop updating it (which would leave its last known value sitting there forever, looking current).

The mechanism is the tombstone: a record with the key you want to remove, and a nullvalue. Compaction treats a null-value record specially. First, it acts as the "latest value" for that key during compaction, so any earlier real values for that key get compacted away as normal. Then, after a further configurable delay — delete.retention.ms — the tombstone record itself is also removed from the log, and at that point the key is gone from the topic entirely, as if it had never existed.

# freshcart.product-prices — cleanup.policy=compact, delete.retention.ms=86400000 (24h)

# Product P1002 is discontinued. Producer writes:
producer.send(topic="freshcart.product-prices", key="P1002", value=None)   # tombstone

# Immediately after compaction runs:
# offset 3: key=P1003  value={price: 23400}
# offset 6: key=P1002  value=null              ← tombstone — marks P1002 for deletion
# offset 5: key=P1001  value={price: 21000}
# (P1002's earlier real price records are gone, same as any other compacted-away version)

# A consumer reading the topic right now still sees the tombstone itself —
# this is intentional: downstream consumers (e.g. a cache, a materialized view)
# need to SEE the null value so they know to remove P1002 from their own state.
# If the tombstone were invisible, downstream systems could never learn about the deletion.

# After delete.retention.ms (24 hours) passes:
# offset 6 (the tombstone) is ALSO removed by the log cleaner.
# P1002 no longer appears anywhere in the topic — not as a price, not as a tombstone.
# A brand new consumer starting today would never even know P1002 once existed.

Combining Both: cleanup.policy=compact,delete

The two policies are not mutually exclusive. Setting cleanup.policy=compact,deleteapplies both behaviors to the same topic: the log cleaner still compacts away superseded values for each key (keeping the latest per key), and time- or size-based retention still applies on top of that, eventually deleting even the latest surviving record for a key once it's old enough.

This combination is useful when you want "latest value per key" semantics but also want a hard upper bound on how long a key's value can survive without being refreshed — effectively an automatic expiry for entities that stop being updated. A session-state topic is a good example: you want the latest state per session_id (compact), but you also don't want session records for users who vanished a year ago sitting around forever just because they were never explicitly tombstoned (delete).

cleanup.policy=compact,delete
retention.ms=2592000000   # 30 days

# freshcart.active-sessions — key = session_id, value = current session state

# Compaction behavior: only the latest state per session_id is kept as usual
# Deletion behavior: even that latest record is deleted if its segment ages
#                     past 30 days without a newer update for that key

# Net effect: a session that keeps getting updated (user still active) keeps
# its latest record indefinitely, refreshed by compaction each time.
# A session that goes quiet for 30+ days with no new updates eventually
# has even its last known state deleted by the retention side of the policy —
# an automatic cleanup for entities nobody is updating anymore.

What Compaction Is Not: A Full History, or a Time-Ordering Tool

It's worth stating a limitation of compaction clearly, because it's easy to reach for compaction as a general-purpose retention upgrade and be surprised later. Compaction is specifically for "what is the latest value for this key" use cases. It is not a way to retain full history more cheaply, and it is not a mechanism for preserving or reconstructing the sequence of changes over time.

Once compaction removes the older records for a key, they are gone — there is no way to ask "what was P1001's price at offset 0" after compaction has run, because that record no longer exists on disk. If your use case needs the full change history of an entity — every price P1001 has ever had, with timestamps, for an audit trail or a time-series analysis — a compacted topic is the wrong tool entirely. You'd want an ordinary delete-policy topic (or an external store like a data warehouse) that retains every individual change event, not just the latest one.

Guidance: Choosing cleanup.policy for a New Topic

The decision usually comes down to one question, asked honestly about the data: does a new record for a given key represent a completely new, independently meaningful fact, or does it represent an update that supersedes the previous value for that key? The first case wants delete. The second wants compact, possibly combined with delete if you also want an expiry for stale, unrefreshed keys.

• ✓Order-placed, payment-captured, page-view, click, sensor-reading events → delete. Each record is a fact about a moment in time; nothing "supersedes" it.
• ✓Current customer address, current product price, current account status, current inventory count → compact. Only the latest value matters; history of prior values is not the point of this topic.
• ✓Kafka Streams / ksqlDB state store changelog topics → compact. These are automatically managed by the stream processing framework specifically because they need cheap, forever-latest-value semantics for fast recovery.
• ✓CDC (change-data-capture) "latest row state" topics mirroring a database table → compact, keyed by primary key. This is the canonical compacted-topic use case — the topic becomes a live mirror of current table state.
• ✓A session-state or presence topic where stale, unrefreshed entities should eventually vanish → compact,delete together, so quiet keys expire automatically instead of accumulating forever.

Ask: "does a new record for this key REPLACE the meaning of the old one,
       or is it a NEW, separate fact alongside the old one?"

REPLACES  → cleanup.policy=compact  (optionally + delete for stale-key expiry)
NEW FACT  → cleanup.policy=delete   (set retention.ms/.bytes to your real need)

# Common tell: if you would ever write application code that does
# "look up the LATEST record for this key" — that's a compaction signal.
# If you would ever write code that does
# "give me EVERY record for this key in order" — that's a delete-policy signal.

It's worth revisiting this decision whenever a topic's actual usage pattern changes, not just at creation time. A topic originally designed as a straightforward event stream sometimes grows an unplanned second use case — a team starts treating "the latest event per entity" as a de facto current-state signal, reading the stream and keeping only the newest record they've seen for each key in application memory. That's a strong sign the topic itself should have been (or should become) a compacted changelog rather than leaving every consumer to reinvent the same latest-value logic on top of a delete-policy topic that was never designed for it.

Operational Sizing: Disk Planning for Retention and Compaction

Retention configuration is not just a data-lifecycle decision — it is directly a disk capacity decision. Every byte a topic retains is a byte sitting on broker disks, multiplied by the topic's replication factor, because every replica holds a full copy of the partition's retained data. Sizing this correctly up front avoids the worst kind of Kafka incident: a broker running out of disk space and refusing all writes for every topic it hosts, not just the one that grew unexpectedly.

For a delete-policy topic, the calculation is straightforward: retained size is roughly bytes-per-second times retention window in seconds, multiplied by replication factor. For acompact-policy topic, sizing is different and, in a healthy design, usually much smaller — the steady-state size approaches roughly one record per distinct key (plus some slack for records awaiting the next compaction pass), regardless of how many total updates have ever been written. This is one of the practical operational benefits of choosing compaction correctly: the topic's disk footprint stops growing with total event volume and instead tracks key cardinality.

# ── delete-policy topic: freshcart.clicks ──────────────────────────────────
# Throughput: 40 MB/second sustained
# retention.ms: 7 days = 604,800 seconds
# replication.factor: 3

# raw retained data per partition (topic-wide, summed across partitions):
# 40 MB/sec * 604,800 sec = ~23.6 TB

# with replication factor 3, total disk footprint across the cluster:
# 23.6 TB * 3 = ~70.8 TB

# this number keeps growing if throughput grows, and resets the calculation
# every time retention.ms changes — this topic's disk footprint is coupled
# directly to (throughput * retention window), forever

# ── compact-policy topic: freshcart.product-prices ──────────────────────────
# Distinct keys (products): ~2 million
# Average record size: ~400 bytes
# replication.factor: 3

# steady-state size, AFTER compaction has caught up, regardless of how many
# total price-change events have ever been produced:
# 2,000,000 * 400 bytes = ~800 MB per full copy

# with replication factor 3:
# 800 MB * 3 = ~2.4 GB

# this number tracks the number of distinct products, NOT total historical
# volume of price changes — a product can be repriced a million times and
# the topic's steady-state size barely moves, because compaction keeps
# collapsing it back down to one record per key

What a Fresh Consumer Actually Sees When It Replays From the Beginning

A useful way to consolidate everything in this module is to trace through what a brand-new consumer sees when it starts reading a topic from the very beginning (offset 0, or the earliest surviving offset), for each cleanup policy. This is exactly the scenario that matters for rebuilding a cache, bootstrapping a new service, or recovering a Kafka Streams state store after a failure.

For a delete-policy topic, "the beginning" is not offset 0 in the absolute sense — it's whatever the oldest surviving offset happens to be after retention has deleted everything older. The consumer sees every event that hasn't yet aged out, in write order, and nothing more. There is no way to recover events that were deleted before the consumer started reading, even if the consumer has never read this topic before.

For a compact-policy topic, "the beginning" is similarly the oldest surviving offset, but what's sitting there is fundamentally different: it's the latest surviving value for whichever keys have not been superseded again since the last compaction pass reached that region, plus, deeper in the tail, exactly one record per key overall. A fresh consumer reading start to finish ends up with a complete, current snapshot of every key that has ever existed and not been tombstoned — not a history of changes, but a full current-state table reconstructed purely by reading records key by key and keeping whichever one it saw last for each key (which, in the compacted tail, is by definition the only one it will see).

# ── delete-policy topic, freshcart.clicks, retention.ms=7 days ──────────────
# Consumer starts reading from the oldest surviving offset.
# Sees: every click event from the last (up to) 7 days, in the order they happened.
# Does NOT see: anything older than 7 days — permanently gone, not reconstructable.
# Use case this supports: "replay recent activity", NOT "what is the full history".

# ── compact-policy topic, freshcart.product-prices ───────────────────────────
# Consumer starts reading from the oldest surviving offset.
# Sees: one record per product_id — its latest known price — regardless of
#       how long ago that product was first created or last repriced.
# Does NOT see: any of the intermediate price changes for a product that has
#               been repriced since — those were compacted away.
# Use case this supports: "give me current state for everything", NOT
#                          "give me the history of how prices changed".

# The mental shortcut: delete-policy replay reconstructs recent HISTORY.
# compact-policy replay reconstructs current STATE. Neither one does the other's job.

This is precisely why Kafka Streams state stores, ksqlDB materialized views, and CDC "current row state" topics all lean on compaction rather than long retention — their entire purpose is answering "what does this look like right now," and a compacted topic answers that question directly on replay, in a bounded amount of data, no matter how much write volume has accumulated over the topic's lifetime.

Using Tombstones for GDPR-Style Deletion Requests

Tombstones aren't just an internal cleanup mechanism — they map directly onto a real business requirement that any team running Kafka in a regulated industry eventually has to solve: a customer exercising a right-to-erasure request (GDPR "right to be forgotten," CCPA deletion requests, or similar). Understanding exactly what a tombstone does and doesn't guarantee is essential to answering "can we actually comply with this request" honestly.

A tombstone, once compacted and once delete.retention.ms has passed, removes that key from the compacted topic entirely — no consumer reading the topic afterward will ever see any trace of that key again. That is a genuine, durable deletion, well-suited to a compacted "current customer state" topic. But it's critical to be precise about scope: the tombstone only removes that key's data from that one compacted topic. It says nothing about copies of that customer's data sitting in non-compacted event-stream topics (which contain that customer's historical order-placed events, click events, and so on, under delete policy), in downstream data warehouse tables, in consumer application caches, or in backups.

# Customer requests deletion: customer_id = "cust_88213"

# freshcart.customer-profile (cleanup.policy=compact, keyed by customer_id)
producer.send(topic="freshcart.customer-profile", key="cust_88213", value=None)
# → after compaction + delete.retention.ms: cust_88213 is GONE from this topic. ✓

# freshcart.orders (cleanup.policy=delete, keyed by order_id, NOT customer_id)
# → cust_88213's historical orders are scattered across this topic under
#   different keys (order IDs). A tombstone on customer_id does NOTHING here —
#   this topic doesn't even use customer_id as its key. ✗ not covered

# Downstream data warehouse table built from freshcart.orders
# → still has every historical row referencing cust_88213 until a SEPARATE
#   deletion/anonymization job runs against the warehouse. ✗ not covered

# A consumer's local in-memory cache, built by replaying freshcart.customer-profile
# BEFORE the tombstone was produced
# → still holds the old value until that consumer restarts and replays again,
#   or explicitly handles the tombstone it eventually receives. ✗ not covered until then

A Note on Schema Changes for Long-Lived Compacted Data

Because a compacted topic can hold a record written years ago sitting right next to one written seconds ago — whichever happens to be the latest surviving value for each key — schema compatibility matters more here than on an ordinary event-stream topic. A delete-policy topic's oldest surviving record is at most as old as its retention window; a compact-policy topic's oldest surviving record can be arbitrarily old, for any key that simply hasn't been updated in a long time.

# freshcart.customer-profile — compact policy, no retention.ms limit on the latest value

# A customer's profile record was last written 3 years ago (schema v1) and has
# never been updated since — compaction keeps that v1 record indefinitely,
# because it's still the latest (and only) value for that key.

# Today, the schema evolves to v4, and every NEW write uses v4.

# A consumer reading this topic from the beginning today sees a MIX:
#   - some keys with v1 records (never updated since)
#   - some keys with v2 or v3 records (updated at some point in between)
#   - some keys with v4 records (updated recently)
# ALL AT ONCE, in a single pass — because compaction doesn't touch a key's
# record just because the schema evolved; it only acts when a NEWER record
# for that key arrives.

# On a delete-policy topic with 7-day retention, the oldest schema version a
# consumer could ever see is bounded by how recently a schema changed within
# that window. On a compact-policy topic, there is no such bound — the
# consumer's deserializer needs to handle every schema version that was ever
# written to a key that hasn't been updated since, indefinitely.

The practical takeaway is that a schema registry with enforced backward compatibility (new fields optional with defaults, never required) matters even more for compacted topics than for event streams, precisely because there's no retention window eventually flushing out old schema versions the way there is on a delete-policy topic. A consumer of a long-lived compacted topic should assume it may need to deserialize the very first schema version that topic ever had, for as long as that topic exists.

The Metrics That Tell You Retention and Compaction Are Actually Working

Retention and compaction are background processes running on their own schedules, independent of producer and consumer activity. That independence is exactly why they need their own monitoring — a topic can look completely healthy from a producer/consumer throughput dashboard while retention silently fails to reclaim disk, or while a compacted topic's log cleaner falls further and further behind, both of which eventually surface as a disk-exhaustion incident with very little warning if nobody was watching the right metrics.

# Alert 1 — broker disk usage trending toward capacity
# ALERT if disk_used_percent > 75% AND slope over last 24h > 0
# (catches a slow leak before it becomes an emergency, same pattern as the
#  consumer-lag alerting logic from the companion module on producers/consumers)

# Alert 2 — log cleaner falling behind on a compacted topic
# ALERT if uncleanable_partitions_count > 0
#   (this specifically means the cleaner has HIT AN ERROR on a partition and
#    stopped compacting it — a corrupted segment, for example — not just that
#    it's running a bit behind; this one needs immediate investigation)
# ALERT if max_dirty_ratio_across_partitions > 0.7 for more than 30 minutes
#   (the cleaner is technically working but losing ground against write volume)

# Alert 3 — a compacted topic significantly exceeding its size estimate
# baseline = distinct_key_count * avg_record_size * replication_factor
# ALERT if actual_topic_size > baseline * 2 for more than 2 hours
#   (either key cardinality grew far beyond plan, or the cleaner is behind —
#    either way, worth a human looking at it before it becomes a capacity issue)

Verifying Cleanup Behavior Before It's Load-Bearing in Production

Both retention and compaction are the kind of behavior that's easy to configure wrong and not notice for a long time — segments only get deleted when they age out, and compaction only runs once the dirty ratio threshold is crossed, so a misconfigured topic can look completely fine for hours or days before the problem becomes visible. It's worth being able to verify the actual behavior directly, on a local or test cluster, rather than trusting a config file alone.

For fast local testing, the trick is to configure very short, unrealistic settings — retention measured in seconds and a small segment size instead of days and a gigabyte — so the behavior you'd normally wait days to observe happens in minutes. This is purely a testing technique; production topics should never run with these aggressive settings.

# Create a test topic with deliberately tiny segment size and short retention
# so deletion is observable in minutes rather than days
kafka-topics --create --topic test.delete-behavior \
  --partitions 1 --replication-factor 1 \
  --config retention.ms=60000 \
  --config segment.ms=10000

# Produce some records, wait ~90 seconds, then check what's left:
kafka-console-producer --topic test.delete-behavior --bootstrap-server localhost:9092
# (type a few test messages, Ctrl+D to exit)

# After waiting: the earliest available offset should have moved forward,
# confirming segments aged out and were deleted as expected
kafka-run-class kafka.tools.GetOffsetShell \
  --broker-list localhost:9092 --topic test.delete-behavior --time -2   # earliest

# --- compaction test ---
kafka-topics --create --topic test.compact-behavior \
  --partitions 1 --replication-factor 1 \
  --config cleanup.policy=compact \
  --config segment.ms=5000 \
  --config min.cleanable.dirty.ratio=0.01   # compact aggressively for the test

# Produce several updates for the SAME key, wait a bit for a compaction pass,
# then consume from the beginning — you should see only the LATEST value:
kafka-console-consumer --topic test.compact-behavior --bootstrap-server localhost:9092 \
  --from-beginning --property print.key=true

# --- tombstone test ---
# Produce a null-value record for a key you already wrote above:
kafka-console-producer --topic test.compact-behavior --bootstrap-server localhost:9092 \
  --property "parse.key=true" --property "key.separator=:"
# type: cust_1: <press Ctrl+D or send an explicit null depending on client>
# after another compaction pass + delete.retention.ms, that key should no
# longer appear when consuming from the beginning

Defaults Live on the Broker, Overrides Live on the Topic

One last practical detail worth being precise about: every setting covered in this module —retention.ms, retention.bytes, cleanup.policy,delete.retention.ms, min.cleanable.dirty.ratio — exists at two levels. The broker has a cluster-wide default (configured as log.retention.ms,log.retention.bytes, log.cleanup.policy, and so on, in the broker's own configuration), and any individual topic can override that default with its own per-topic configuration. A topic created without an explicit override simply inherits whatever the broker's cluster-wide default happens to be.

# Broker-level defaults (server.properties) — apply to every topic
# that doesn't explicitly override them:
log.retention.hours=168          # 7 days — note: HOURS at the broker level
log.retention.bytes=-1
log.cleanup.policy=delete
log.segment.bytes=1073741824     # 1 GB

# A specific topic overriding just the setting it needs:
kafka-configs --alter --entity-type topics --entity-name freshcart.product-prices \
  --add-config cleanup.policy=compact,delete.retention.ms=86400000

# freshcart.product-prices now uses cleanup.policy=compact and
# delete.retention.ms=86400000 explicitly, while still inheriting the
# broker's log.segment.bytes=1073741824 for its segment size, since that
# wasn't overridden at the topic level.

# Checking what's actually in effect for a topic (override + inherited defaults):
kafka-configs --describe --entity-type topics --entity-name freshcart.product-prices \
  --bootstrap-server broker:9092

The practical habit worth building: never assume a topic's effective retention or cleanup policy just because you remember what the cluster default is supposed to be. Topics accumulate overrides over their lifetime — a one-off change made during an incident two years ago, a setting copied from a template topic that no longer matches the current cluster defaults — and kafka-configs --describe is the only reliable source of truth for what a specific topic is actually doing right now, as opposed to what it was configured to do when it was created.

Five Misconceptions About Retention and Compaction

What This Looks Like on Day One

5 Interview Questions — With Complete Answers

Mistakes Beginners Make Constantly

Errors and Symptoms You Will Hit — And Exactly Why