Events, Topics, and Partitions

Kafka Is Built from Events, Topics, Partitions, and Offsets

Module 01 explained why Kafka exists. This module explains, in real depth, what Kafka is made of. Every Kafka system, from a tiny local demo to a multi-region enterprise platform, depends on the same data model: events are written to topics, topics are split into partitions, and records inside each partition receive offsets. If you understand this model deeply, Kafka becomes predictable — you can look at almost any production incident and reason about it from first principles. If you only memorize the words, Kafka feels arbitrary and its errors feel like bad luck.

The common beginner mistake is to imagine a topic as one simple list, like an array or a single file. That picture is useful for the first five minutes of learning Kafka, but it becomes wrong almost immediately once real volume enters the picture. A Kafka topic is not one list. A topic is, physically, a set of independent partitions, and each partition is its own ordered log, usually stored on a different broker. Kafka guarantees order inside one partition. It makes no promise about order across the different partitions that together make up a topic.

topic: orders  (created with 3 partitions)

partition 0:
  offset 0 -> OrderPlaced(order_id=100)
  offset 1 -> PaymentAuthorized(order_id=100)
  offset 2 -> ShipmentPrepared(order_id=100)

partition 1:
  offset 0 -> OrderPlaced(order_id=101)
  offset 1 -> OrderCancelled(order_id=101)

partition 2:
  offset 0 -> OrderPlaced(order_id=102)
  offset 1 -> PaymentAuthorized(order_id=102)
  offset 2 -> OrderPlaced(order_id=103)

# Notice: order_id=100's three events are all in partition 0, in order.
# order_id=101's two events are both in partition 1, in order.
# But there is no defined relationship between, say,
# partition 0 offset 2 and partition 2 offset 1 — Kafka does not
# claim to know which one happened first in real-world time.

Everything else in this module — what exactly an event contains, why topics get split this way, how a record ends up in one partition versus another, and what retention actually means at the partition level — is really just unpacking the consequences of this one picture.

An Event Is an Immutable Fact With a Key, a Value, a Timestamp, and Headers

Module 01 introduced the event loosely as "a fact that already happened." Here is the precise, physical shape of a Kafka record, because every downstream design decision — how you choose a key, how consumers deduplicate, how ordering works — depends on understanding exactly what fields a record carries.

Key

An optional but usually present field. The key is what Kafka's default partitioner uses to decide which partition a record goes into (Part 05 covers this fully). Records with the same key are always routed to the same partition, which is how per-entity ordering is achieved. A key is typically a business identifier: an order ID, a customer ID, a device ID.

Value

The actual payload — the business facts the event carries. This is usually JSON, Avro, or Protobuf-encoded bytes. Kafka itself does not care what format the value is in; it stores and transmits raw bytes. Interpreting those bytes correctly is the producer and consumer's shared responsibility, which is why schema contracts matter so much in real systems.

Timestamp

Every record carries a timestamp, set either by the producer at write time (event time) or by the broker at append time (ingestion time), depending on topic configuration. This timestamp is distinct from a timestamp field you might also put inside the value's JSON body — Kafka's own record timestamp is a first-class part of the record format, used for retention decisions and for the time-based index that lets a consumer say "start reading from 3 hours ago."

Headers

Optional key-value metadata attached to a record, separate from the value payload — similar in spirit to HTTP headers. Commonly used for cross-cutting concerns that shouldn't pollute the business payload: a trace ID for distributed tracing, a schema version identifier, the name of the producing service, or a content-type marker.

key:       "order-1024"                       (bytes — usually a string or serialized ID)

value:     {                                   (bytes — usually JSON/Avro/Protobuf)
             "event_id": "a2b1022e-8fb5-43a2-86be-8e07c56f1b10",
             "event_type": "OrderPlaced",
             "order_id": "order-1024",
             "customer_id": "customer-88",
             "total_usd": 149.00,
             "currency": "USD"
           }

timestamp: 2026-09-10T18:42:11.203Z            (set by producer or broker, per topic config)

headers:   {
             "trace-id": "9f1c2a-b3e7-44d1",
             "schema-version": "3",
             "producer-service": "checkout-service"
           }

# Kafka stores this whole record at, say, topic=orders, partition=2, offset=88112
# The (topic, partition, offset) triple is how you address this exact record —
# the key and headers do not identify it uniquely by themselves.

Serialization — what "bytes" actually means in practice

It is worth being concrete about how a value actually becomes bytes, since this matters for Part 09's design discussion later and for every producer/consumer pair that has to agree on it. The three formats you will encounter constantly are JSON (human-readable, flexible, no built-in schema enforcement, larger on the wire), Avro (compact binary format with strong schema-evolution support, the most common choice in mature Kafka platforms), and Protobuf (compact binary, schema-driven, common where the organization already uses gRPC elsewhere). Kafka itself is agnostic to all three — it stores and transmits opaque bytes and never inspects the value's contents. The producer and every consumer of a topic must independently agree on which format is in use and how to interpret it; Kafka provides no enforcement of this by itself; that is what a schema registry, covered in a later module, is for.

The single most important property of an event, underneath all four of these fields, is immutability. Once a record is appended to a partition, it is never edited in place. If a customer changes their shipping address, you do not go back and mutate the originalOrderPlaced record — you produce a new event, OrderAddressUpdated, with its own key, value, timestamp, and offset. The original record stays exactly as it was written, forever (or until retention removes it). This is what makes the log a trustworthy, append-only history rather than a mutable snapshot.

Event versus command — a distinction worth holding onto

A useful test for whether something belongs in Kafka as an event: is it phrased as a fact that already happened, or as an instruction telling one specific system what to do? OrderPlacedis a fact — it describes what happened and lets billing, warehouse, analytics, and fraud each decide independently what it means for them. ChargeCustomerNow is a command — it only makes sense to one consumer, the billing service, and coupling a Kafka topic to one consumer's specific action defeats the entire purpose of publishing a broadly reusable stream of facts. This is the same idea Module 01 introduced at a high level; here it becomes a concrete filter you can apply to every event you design.

A Topic Is a Named Stream — a Category of Events, and a Contract

A topic is the name producers write to and consumers read from — the organizing unit above partitions. Topic names look like simple strings, but in a real Kafka platform they function as contracts between teams. The topic name communicates ownership, domain, data meaning, and sometimes environment or privacy level, long before anyone reads a single record from it.

Poor topic names create real confusion. A topic named data tells a new consumer nothing. A topic named orders is better, but still ambiguous — does it contain order commands, raw database row changes, business events, or periodic snapshots? A topic namedcommerce.orders.events is much clearer: the domain is commerce, the entity is orders, and the content is discrete events (as opposed to, say, full-row change-data-capture records).

A topic is a category of events — one coherent stream, not a dumping ground

There is no universal number of topics an organization should have. Too few topics create mixed streams where every consumer must inspect each record just to figure out what kind of event it even is — defeating the purpose of a clear contract. Too many topics create operational noise: access-control complexity, metadata overhead across the cluster, and unclear ownership. A good topic represents one coherent category of events with a clear owner and a stable contract — think of it as one well-defined table in a database, not as a folder you drop arbitrary files into.

A subtle but important consequence of treating a topic as a contract: renaming a topic is not a simple find-and-replace the way renaming a variable is. Every producer and every consumer group across every team that touches the topic needs to be updated in coordination, or you end up with two parallel, diverging topics — an old one that some consumers still read, and a new one that others have already switched to. This is exactly why topic-naming conventions, decided early and applied consistently, save real coordination pain later — a topic name is closer to a public API endpoint than to an internal implementation detail.

Partitions Exist for One Reason: Parallelism — at the Cost of a Trade-Off

A partition is one ordered log inside a topic, physically stored (with its replicas) on specific brokers. Kafka splits topics into partitions for exactly one fundamental reason: parallelism. A single log, on a single machine, can only be written to and read from so fast — bounded by that one machine's disk and network. Splitting a topic into multiple partitions lets Kafka spread that load across multiple brokers for writes, and across multiple consumers for reads, simultaneously.

Without partitions, one topic would be limited to whatever throughput a single log on a single broker can sustain — typically somewhere in the range of 100 MB/second on solid modern hardware, depending on record size and disk. And no matter how many consumer instances you deploy, only one of them could ever be actively reading that single log at a time within a consumer group, because Kafka assigns each partition to exactly one consumer per group. More partitions is literally what buys you both more write throughput and more consumer parallelism — there is no other lever.

orders topic with 1 partition, consumer group "billing":
  billing-consumer-1 -> partition 0   (the only active consumer)
  billing-consumer-2 -> IDLE          (nothing to assign it — sits and waits)
  billing-consumer-3 -> IDLE

orders topic with 6 partitions, same consumer group:
  billing-consumer-1 -> partition 0, partition 1
  billing-consumer-2 -> partition 2, partition 3
  billing-consumer-3 -> partition 4, partition 5
  (all three consumers actively working — roughly 3x the throughput of the single-partition case,
   assuming even load across partitions)

It is worth being precise about the unit of scaling here too: partitions scale a topic's throughput across brokers because different partitions of the same topic are typically led by different brokers, spreading disk and network load. They scale consumer-side throughput because Kafka's consumer group protocol assigns each partition to exactly one consumer instance within a group at a time — so N partitions can be actively processed by up to N consumer instances simultaneously, each handling a disjoint slice of the topic's total traffic independently.

The trade-off partitions introduce is the single most important Kafka rule in this whole module: Kafka guarantees ordering within a partition, not across partitions. If record A and record B are written to the same partition, Kafka preserves their relative order absolutely — no exceptions, guaranteed by the sequential nature of an append-only log. If record A is in partition 0 and record B is in partition 1, Kafka makes no claim whatsoever about which one a consumer will see "first," even if they were produced a millisecond apart in the real world.

orders topic with 2 partitions

partition 0:
  offset 0 -> OrderPlaced(order_id=100)
  offset 1 -> PaymentAuthorized(order_id=100)

partition 1:
  offset 0 -> OrderPlaced(order_id=101)
  offset 1 -> PaymentAuthorized(order_id=101)

Kafka guarantees: within partition 0, OrderPlaced(100) happened before PaymentAuthorized(100).
Kafka guarantees: within partition 1, OrderPlaced(101) happened before PaymentAuthorized(101).
Kafka does NOT guarantee: that partition 0's PaymentAuthorized happened before or after
  partition 1's OrderPlaced, even though they are both "offset 1" and "offset 0" respectively —
  those offset numbers have no relationship to each other across partitions.

Why not just use one partition and accept the throughput ceiling?

For a genuinely low-volume topic, this is a completely reasonable choice, and it is worth stating plainly rather than treating multiple partitions as automatically correct. A single partition gives you the simplest possible mental model — one ordered log, one consumer processing it at a time, no cross-partition ordering questions to reason about at all. The moment throughput or consumer-parallelism requirements exceed what one partition and one consumer instance can sustain, that simplicity has to give way to the trade-off in the table above. The right question is never "how many partitions should every topic have," it is "what does this specific topic's traffic and consumer-scaling requirement actually demand."

Key Hashing, Round-Robin, and Custom Partitioners

Every record a producer sends must end up in exactly one partition. Kafka decides this using a partitioner, and understanding the three ways this decision gets made is essential — it directly determines whether related events end up ordered relative to each other or scattered randomly.

Key hashing (the default, when a key is present)

If a record has a key, Kafka's default partitioner hashes that key and uses the hash to deterministically choose a partition. Critically, the same key always hashes to the same partition (as long as the partition count doesn't change — more on that in Part 06). This is exactly what gives you per-entity ordering: every event with key=order-1024 lands in the same partition, in the order it was produced, regardless of which producer instance sent it or when.

Round-robin / sticky partitioning (when no key is present)

If a record has no key (a null key), Kafka does not guarantee any particular entity's events stay together, because there is no entity identifier to hash. Modern Kafka producers use a "sticky" strategy by default — batching a run of keyless records onto the same partition for a short window to improve batching efficiency, then moving to the next partition — which approximates even distribution across partitions over time without any ordering guarantee between individual records.

Custom partitioner

Producers can supply their own partitioning logic — for example, routing all events for a VIP customer tier to a dedicated, less-contended partition, or implementing a specific load-balancing scheme that the default hash doesn't provide. This is an escape hatch for advanced cases; most production topics use the default key-hash behavior.

# Simplified view of the default partitioner's logic:
partition = hash(record.key) % number_of_partitions

# Example: orders topic, 6 partitions
hash("order-1024") % 6  -> always resolves to the same partition number
                            every single time, for every producer instance,
                            as long as partition count stays at 6

# This is why:
key="order-1024" -> OrderPlaced        -> partition 2 (say)
key="order-1024" -> PaymentAuthorized  -> partition 2 (same key, same partition)
key="order-1024" -> ShipmentPrepared   -> partition 2 (same key, same partition)
# All three land in partition 2, in the order they were produced — ordered relative to each other.

key="order-1025" -> OrderPlaced        -> partition 5 (different key, likely different partition)
# order-1025's events are ordered relative to EACH OTHER, but have no defined
# relationship to order-1024's events, because they live in different partitions.

Hot partitions — when key choice goes wrong

A hot partition happens when one partition receives dramatically more traffic than its siblings, almost always because of a low-cardinality key choice. If you key by country in a US-heavy application, the partition that "US" hashes to absorbs a disproportionate share of all traffic — and adding more partitions to the topic does not fix this, because"US" still always hashes to exactly one of them.

Bad for a global shopping app:
  key = country
  result: "US" traffic may dominate a single partition, no matter the total partition count

Better when order histories just need to be independently ordered:
  key = order_id
  result: many unique, high-cardinality keys spread evenly across partitions

Better when customer-level history must stay ordered:
  key = customer_id
  result: each customer's own events stay ordered; overall load still spreads well
          as long as no single customer dominates total volume

Why Partition Count Can Only Increase, Never Safely Decrease

Kafka lets you increase a topic's partition count at any time, but it does not support decreasing it — and understanding why exposes exactly how fragile the key-hashing guarantee from Part 05 really is under a partition-count change.

Recall the hashing formula: partition = hash(key) % number_of_partitions. The moment the partition count changes, that modulo operation produces different results for the same key. A key that used to hash to partition 2 out of 6 might now hash to partition 5 out of 8. Every guarantee about "this key's events all land in the same partition" is only valid for a fixed partition count — change the count, and new events for an existing key can start landing in a different partition than that key's older events.

# Before: orders topic, 6 partitions
hash("order-1024") % 6 = 2   -> all of order-1024's events go to partition 2

# Topic is repartitioned to 8 partitions (scaling up for more throughput)
hash("order-1024") % 8 = 5   -> NEW events for order-1024 now go to partition 5

# Result: order-1024's OLD events (OrderPlaced, PaymentAuthorized) sit in partition 2.
#         order-1024's NEW events (ShipmentPrepared, going forward) sit in partition 5.
# A consumer reading partition 2 alone no longer sees this order's full history.
# This is true even when INCREASING partition count — it is a known, accepted cost.

Given that even increasing partition count already disrupts per-key ordering for existing keys, decreasing it would be strictly worse and is not supported at all by Kafka's tooling — there is no safe, built-in way to merge partitions back down while preserving each partition's internal order and the records already written to disk. The only way to effectively "shrink" partition count is to create an entirely new topic with the smaller count and migrate producers and consumers to it deliberately, accepting the same reordering consequences shown above.

It also helps to name the failure mode this asymmetry guards against directly: teams sometimes try to "fix" an over-provisioned topic by shrinking partition count to save on broker resource usage, discover Kafka's tooling refuses to do it, and then consider force-deleting and recreating the topic instead. That approach discards the entire event history unless it has been carefully exported and replayed back in first — a much larger operation than it initially sounds like, and exactly why getting partition count right at design time, per Part 04's throughput/parallelism reasoning, is worth the extra planning up front.

A Partition on Disk: An Append-Only Log, With Offsets as Position Markers

It helps to ground all of this in what a partition actually is on a broker's filesystem, not just as an abstract diagram. A partition is stored as a sequence of files on disk — an append-only log file, plus an index file that maps offsets to byte positions so a consumer requesting a specific offset doesn't have to scan the entire file from the start.

/data/kafka/orders-2/                    <- partition 2 of the orders topic
    00000000000000000000.log             <- raw bytes of sequentially appended records
    00000000000000000000.index           <- sparse offset -> byte-position lookup index
    00000000000000000000.timeindex       <- timestamp -> offset lookup index

# When a consumer asks to start reading from offset 88112:
# 1. broker binary-searches the .index file for the nearest indexed offset <= 88112
# 2. seeks directly to that byte position in the .log file
# 3. scans forward a short distance to the exact offset requested
# This stays fast even with millions of records in the partition, because it never
# needs to scan the whole file from the beginning.

An offset, in this physical picture, is simply the position number of a record within this one append-only file (conceptually — in reality large partitions are split into multiple smaller segment files, but the logical numbering is continuous). Offset 0 is the very first record ever written to this partition. Offset 1 is the next. They increase monotonically and are never reused, even after old records are deleted by retention — offset numbers are never recycled.

Consumers use offsets as their entire notion of progress. A consumer group that has processed through offset 88112 in partition 2 commits that number back to Kafka. If that consumer restarts, it resumes from offset 88113 — the next unread record. This is also exactly how replay works: resetting a consumer group's committed offset to an earlier number, or to a specific timestamp using the .timeindex file, makes the consumer re-read history from that point forward, as long as those records are still within the retention window.

It is worth being precise about one more subtlety here: offsets are tracked separately per consumer group, not per consumer or per topic globally. If billing-consumer-group and analytics-consumer-group both read the orders topic's partition 2, Kafka tracks two entirely independent committed offsets for that one partition — one per group. This is what makes it possible for one group to be fully caught up while another is intentionally replaying from the beginning, at the same moment, on the same partition, without either affecting the other's progress in any way.

How Retention Works — A Working Model, Not the Full Internals

Data in a Kafka topic is not kept forever by default. Every partition's log has a retention policy that determines when old records become eligible for deletion — typically time-based (for example, delete anything older than 7 days) or size-based (cap the partition at a maximum size on disk, whichever fills up first). A later module in this track, on retention and compaction, goes into the full internals — how deletion actually happens at the segment-file level and what log compaction does differently. Here, you need a correct working model, not the complete mechanics.

The working model is: retention operates on the whole log, deleting the oldest records first, and it happens independently of whether any consumer has actually read those records yet. A consumer that falls far enough behind can have data age out of retention before it ever gets read — this is not a bug, it is the direct, predictable consequence of a time-or-size-bounded log with consumers that were too slow to catch up in time.

orders partition 2, retention = 7 days

# Records older than 7 days become eligible for deletion, oldest first.
# This happens on a schedule, regardless of consumer read progress.

billing-consumer-group:    committed offset far behind -> at risk if lag exceeds 7 days of data
analytics-consumer-group:  caught up, near the latest offset -> not at risk

# If billing-consumer-group's lag ever exceeds what 7 days of retention holds,
# the oldest lagged records are already gone by the time billing tries to read them —
# not delayed, permanently unavailable to that consumer.

One related concept worth naming here, without going deep into it: some topics are configured for log compaction instead of (or alongside) time-based retention — retaining only the latest record per key forever, rather than deleting by age. That pattern is used for "current state" topics, like a changelog of a customer's current address. The full mechanics of compaction — how the log cleaner thread works, tombstones, and segment-level cleanup — belong to the dedicated retention-and-compaction module later in this track; the model above is everything you need to reason correctly about ordinary, time-based retention today.

Three Real Topic Designs, and the Reasoning Behind Each

Design 1: Order lifecycle events

topic: commerce.orders.events
partitions: 12
key: order_id
events:
  OrderPlaced
  PaymentAuthorized
  ShipmentPrepared
  ShipmentDispatched
  OrderCancelled

Why:
  Every order's lifecycle stays fully ordered (Part 04, Part 05 — same key, same partition).
  Different orders process independently in parallel across 12 partitions.
  12 chosen with headroom for a consumer group expected to run up to ~10 instances at peak.

Design 2: Customer activity stream

topic: growth.customer-activity.events
partitions: 24
key: customer_id
events:
  PageViewed
  ProductClicked
  SearchSubmitted
  CartUpdated

Why:
  Personalization consumers need each customer's own activity in order — keyed by customer_id.
  High cardinality of customer_id (millions of distinct values) spreads load evenly —
  no single customer dominates a partition the way "country" would (Part 05's hot-partition warning).

Design 3: Raw clickstream

topic: web.clickstream.raw
partitions: 96
key: null (or session_id, depending on whether session-level ordering is needed)
events:
  Click
  PageView
  Impression

Why:
  Very high volume needs many partitions purely for write and read throughput.
  Exact global ordering across all clicks is not realistic or required for this use case —
  keyless/round-robin distribution (Part 05) maximizes even spread across all 96 partitions.

Design 4: A low-volume, single-partition "current status" topic

topic: ops.feature-flags.changes
partitions: 1
key: flag_name
events:
  FlagEnabled
  FlagDisabled
  FlagRolloutPercentageChanged

Why:
  Volume is tiny (a handful of changes per day, made by engineers, not by end users).
  A strict, globally ordered history of every flag change matters more than throughput —
  one partition trivially gives perfect ordering with no cross-partition reasoning required.
  This is the case from Part 04 where the throughput ceiling of one partition is irrelevant.

These four designs are not a menu to copy from — they are a demonstration of the same reasoning process applied to different requirements: identify what needs to stay ordered relative to what, estimate real throughput and consumer parallelism needs, and only then choose a key and a partition count. A topic design review that cannot answer "why this key, why this partition count" in those terms has not actually been designed yet.

Explain This Like a Kafka Engineer

If someone asks you to explain Kafka topics and partitions, answer like this: a topic is a named stream of records — a category of events with a defined contract. A topic is physically split into one or more partitions for parallelism. Each partition is an independent, ordered, append-only log, stored on disk on a broker (and replicated to others). Records inside a partition receive monotonically increasing offsets that are never reused. Kafka guarantees ordering within a partition, not across a topic's partitions as a whole. Producers use a record's key, hashed against the current partition count, to choose which partition a record lands in — which is exactly what gives you per-entity ordering, and exactly why partition count can only safely increase, never decrease, without disrupting that guarantee for existing keys. Partitions are simultaneously the unit of parallelism and the boundary of ordering — that trade-off is the single most important idea in this module.

Checklist before creating a topic

• ✓What business facts belong in this topic, and what should NOT be mixed in?
• ✓Who owns this topic, and what is its schema/event contract?
• ✓What key preserves the ordering that consumers actually need?
• ✓How many partitions are needed for expected throughput AND expected consumer parallelism, with headroom?
• ✓How long should records be retained, and can consumers realistically stay within that window?
• ✓Which teams or applications are expected to read and write this topic?
• ✓What metrics and alerts (consumer lag, partition skew) prove this topic is healthy?

Carry one idea forward into Module 03: everything in this module described the shape of the data — events, topics, partitions, offsets, keys. Module 03 describes the mechanics of how producers and consumers actually talk to brokers over the network to read and write that data — metadata discovery, leader election, acknowledgement settings, and consumer group rebalancing. The data model you now understand is the foundation that machinery operates on.

Five Misconceptions About Events, Topics, and Partitions

Why This Module Matters on the Job

5 Interview Questions — With Complete Answers

Mistakes Beginners Make Constantly

Symptoms You Will Actually Hit — And Exactly Why They Happen