Keys, Ordering, and Partitioning Strategy

A Record Key Is a Routing Instruction, Not a Label

Every Kafka record has an optional key and a value. Beginners usually treat the key as metadata — something you attach to a record the way you'd attach a tag, useful for humans reading logs later. That is not what the key is for. The key's real job is to decide which partition the record lands on. That single fact — key chooses partition — is the mechanism behind almost everything else in this module: per-key ordering, hot partitions, custom partitioners, and the partition-count trap.

When a producer sends a record with a non-null key, the default partitioner hashes the key and maps it onto one of the topic's partitions using hash(key) % partition_count. Because hashing a given input always produces the same output, every record with the same key is guaranteed to hash to the same partition, every single time, for as long as the partition count stays fixed. That is the entire mechanism. There is no coordinator making a routing decision. There is no lookup table mapping keys to partitions. It is a pure, deterministic function evaluated independently by every producer.

topic: freshcart.orders, partition_count = 4

producer.send(topic="freshcart.orders", key="user_9271", value={...order 1...})
producer.send(topic="freshcart.orders", key="user_9271", value={...order 2...})
producer.send(topic="freshcart.orders", key="user_4410", value={...order 3...})

# default partitioner:
partition = hash("user_9271") % 4   # → always evaluates to the same number, e.g. 2
partition = hash("user_9271") % 4   # → 2 again, same input, same output
partition = hash("user_4410") % 4   # → some other number, e.g. 0 (probably)

# result:
# partition 2: order 1, order 2   (both records for user_9271, in send order)
# partition 0: order 3            (user_4410's record)

# every future record with key="user_9271" also lands on partition 2
# as long as partition_count stays 4 — see Part 06 for what happens if it doesn't

It is worth being precise about "hash" here. Kafka's default partitioner uses a hashing function (murmur2 by default, historically) over the serialized key bytes, then takes the result modulo the number of partitions. You don't need to know the internals of murmur2 to use Kafka correctly. You do need to internalize that the formula depends on two things: the key's bytes, and the current partition count. Change either one and the output changes.

What Happens With No Key: Round-Robin and Sticky Partitioning

Not every record needs a key. If you send a record with key=null, there is nothing to hash, so Kafka cannot route it based on key content. Instead, the producer distributes keyless records across partitions using a load-balancing strategy so that, over time, every partition gets a roughly even share of traffic.

Older Kafka producer clients did strict round-robin: record 1 to partition 0, record 2 to partition 1, record 3 to partition 2, and so on, cycling through every partition on every single send. Newer producer clients (since Kafka 2.4, the "sticky partitioner") instead stick to one partition for an entire batch before switching to the next, because that produces larger, more efficient batches without sacrificing even distribution — round-robin at the record level means every batch is tiny, since consecutive records rarely land in the same batch.

producer.send(topic="freshcart.clicks", key=null, value={...click A...})
producer.send(topic="freshcart.clicks", key=null, value={...click B...})
producer.send(topic="freshcart.clicks", key=null, value={...click C...})
producer.send(topic="freshcart.clicks", key=null, value={...click D...})

# sticky partitioner: picks partition 1, batches A, B, C together into one request
# once that batch is sent (or a new batch is needed), it picks a new partition, say 3
# click D and subsequent keyless records go to partition 3, batched together

# net effect over millions of clicks: partitions get roughly even volume
# but no ordering relationship is implied or preserved between any two keyless records

One easy way to check which behavior a topic is actually getting in practice is to look at how evenly bytes and message counts spread across partitions during ordinary traffic. A keyed topic with a well-chosen, high-cardinality key and a null-keyed topic should both show roughly even distribution across partitions — for the keyed topic because the key population is broad, and for the null-keyed topic because the load-balancing strategy guarantees it regardless of what the data looks like. A skewed distribution on a keyed topic is a signal worth investigating per Part 04; a skewed distribution on a genuinely null-keyed topic would point to a producer client bug rather than a data problem, since null-key distribution isn't supposed to depend on the data at all.

Kafka Orders Within a Partition — And Nowhere Else

This is the single most commonly misunderstood fact about Kafka, and it deserves to be stated as precisely as possible: Kafka guarantees that records within one partition are delivered to consumers in the exact order they were written to that partition. Kafka makes noguarantee whatsoever about the relative order of records that live in different partitions — even within the same topic, even if they were produced by the same producer, even if one was sent a full second before the other.

Why this specific scope, and not something broader? Because a partition is a single append-only log, written by whichever broker currently leads it, and read sequentially by consumers. There is exactly one physical sequence of bytes, so there is exactly one order. Two different partitions are two different logs, potentially led by two different brokers, written to independently, with no synchronization between them. There is no shared clock or shared sequence number spanning partitions — nothing to order them by even if Kafka wanted to.

topic: freshcart.orders, 4 partitions
key="user_9271" always hashes to partition 2 (per Part 01)

# Producer sends, in this exact order:
send(key="user_9271", value="order A")   # → partition 2, offset 40
send(key="user_4410", value="order B")   # → partition 0, offset 18
send(key="user_9271", value="order C")   # → partition 2, offset 41

# GUARANTEED: within partition 2, order A (offset 40) is seen before order C (offset 41)
#             — consumers of partition 2 will never see C before A

# NOT GUARANTEED: any ordering relationship between order B (partition 0)
#                  and order A or order C (partition 2)
#                  a consumer reading both partitions could see B before A,
#                  between A and C, or after C — all are valid

This is exactly why the key matters so much: the key is the mechanism by which you convert "I need these related records in order" into "these related records share a key, therefore they share a partition, therefore they are ordered." Ordering in Kafka is not something you get automatically — it is something you deliberately engineer by choosing which entity's identifier becomes the key.

Retries can silently reorder writes within the same partition

There's one more layer to the "what the log physically contains" point that's worth spelling out, because it directly connects this module to the producer configuration decisions covered elsewhere in this track. By default, a producer can have several requests in flight to the same broker at once (max.in.flight.requests.per.connection, default 5). If an earlier batch fails to get acknowledged and is retried while a later batch has already been sent, the later batch can be written to the partition first — even though both batches were destined for the same partition and the earlier one was sent first.

# max.in.flight.requests.per.connection = 5 (default), retries enabled

# Producer sends batch B1 (records for key="user_9271") to partition 2 — no ack yet
# Producer immediately sends batch B2 (also key="user_9271") to partition 2 — no ack yet
# Network hiccup: B1's ack is lost. B2 arrives cleanly and is written at offset 100.
# Producer retries B1. B1 is now written at offset 101 — AFTER B2, even though
# B1 was sent to the broker first.

# Both B1 and B2 are correctly in partition 2 — the partitioning was correct.
# But the WRITE ORDER inside that partition no longer matches SEND order.
# A consumer of partition 2 sees B2's records before B1's records.

# Fix: enable.idempotence=true
# This assigns each batch a sequence number and lets the broker enforce that
# sequence numbers are written in order per partition, even with retries and
# multiple in-flight requests — the broker rejects an out-of-order retry rather
# than silently accepting it out of sequence.

This is why, in practice, per-key ordering guarantees and idempotent producers travel together. Choosing a good key gets related records into the same partition; enable.idempotence=trueis what keeps the broker from letting a network retry scramble the order those records actually land in once they're there. Neither one alone is the complete guarantee.

Picking a Key: Even Distribution Versus Per-Entity Ordering

Choosing a partition key is a genuine design decision with real trade-offs, not a checkbox. The two things you're usually balancing are: (1) even distribution of load across partitions, so no single partition — and therefore no single consumer instance, since one partition is read by at most one consumer per group — becomes a bottleneck, and (2) grouping records that must stay ordered relative to each other under the same key.

Good partition keys for real workloads are almost always the identifier of the entity whose events must stay in order relative to each other, and which also has enough cardinality (enough distinct values, spread roughly evenly) that hashing spreads load well. user_id works well for a clickstream or account-activity topic — you usually want one user's events processed in order, and you have millions of roughly-equal-volume users. order_id works well for an order lifecycle topic — every state transition for one order (placed, paid, packed, shipped, delivered) needs to stay in sequence, and orders are numerous and roughly evenly sized.

The hot partition problem

A hot partition happens when your chosen key is skewed — a small number of key values account for a disproportionate share of traffic. Because every record for a given key always hashes to the same partition, a skewed key doesn't just create uneven load, it concentrates that imbalance permanently onto one physical partition, and therefore onto one broker and one consumer instance.

topic: freshcart.orders, keyed by customer_id, 8 partitions

# Normal customers: a few orders a month each, spread evenly across 8 partitions
# One customer: a large enterprise reseller placing 40% of all order volume through
#               a single automated integration account

# Every one of that reseller's orders hashes to the SAME partition (say, partition 5)
# because they all share customer_id="reseller_882"

# Result:
#   partitions 0,1,2,3,4,6,7 — light, evenly loaded, consumers idle much of the time
#   partition 5              — carrying 40% of total volume alone
#   the single consumer instance assigned to partition 5 cannot keep up
#   consumer lag grows on partition 5 while lag on every other partition sits near zero
#   this is invisible in an AVERAGE lag metric — see Part 07's Error Library entry

The same problem shows up with a single wildly popular product ID in an inventory-events topic, a single viral post ID in a social-activity topic, or any workload with a genuine power-law distribution of activity per key. There is no purely automatic fix, because the skew is inherent to the business data, not a configuration mistake. Common mitigations include salting the hot key (for example, appending a random suffix like reseller_882#3 so one logical entity's traffic spreads across several partitions, at the cost of losing strict per-entity ordering for that entity), increasing partition count so the hot key's share of any one partition shrinks somewhat (it helps only marginally, since the hot key still lands on exactly one of the now-larger set of partitions), or building a composite key that includes a natural sub-dimension of the entity (order line ID instead of just order ID, if line-level ordering is sufficient).

When You'd Write a Custom Partitioner

The default partitioner (hash the key, mod the partition count; sticky/round-robin for null keys) is correct for the overwhelming majority of topics. A custom partitioner is a deliberate override of that logic, and it is worth reaching for only when the default's behavior actively works against a specific, well-understood requirement.

The most common legitimate reasons to write one: explicit routing that has nothing to do with a hash at all — for example, sending all records tagged priority=critical to a dedicated subset of partitions served by a smaller, faster consumer pool, while routine records use the rest; mitigating a known hot key by applying custom salting logic inside the partitioner itself rather than in every producer call site; or maintaining compatibility with a partitioning scheme from another system during a migration, so that records that were co-located in the old system stay co-located in Kafka.

# Reserve partitions 0-1 of an 8-partition topic for critical records,
# spread everything else across partitions 2-7.

class PriorityPartitioner:
    def partition(self, key, value, partition_count, available_partitions):
        if value.get("priority") == "critical":
            # only 2 partitions for the critical lane — small, fast consumer pool
            return hash(key) % 2
        else:
            # remaining 6 partitions for everything else
            return 2 + (hash(key) % (partition_count - 2))

# This is NOT the default behavior — a hash of the key alone would never
# treat two records with the same key differently based on a value field.
# A custom partitioner can inspect the full record, not just the key.

Changing Partition Count Breaks the Key-to-Partition Mapping

This is the gotcha that catches teams who understand everything above but haven't connected it to operations. The formula is hash(key) % partition_count. Partition count is not a constant baked into the hash — it's a live input. If you increase a topic's partition count after the topic has been in production, the modulo changes, which means the formula's output changes for most keys, which means most keys now map to a different partition than they did before.

# freshcart.orders starts with 4 partitions
hash("user_9271") % 4 = 2    # historically, all of user_9271's records are in partition 2

# ... months of production traffic accumulate in partition 2 for this user ...

# Ops decides the topic needs more throughput and runs:
#   kafka-topics --alter --topic freshcart.orders --partitions 8

hash("user_9271") % 8 = 6    # same key, same hash, DIFFERENT partition — modulo changed

# What this means concretely:
# - All of user_9271's OLD records are still sitting in partition 2 (data is never moved)
# - All of user_9271's NEW records, from this point forward, go to partition 6
# - A consumer that groups by key across the topic's full history now sees the
#   same user's order sequence split across two partitions with no ordering
#   relationship between them
# - This applies to EVERY key whose hash % 4 != hash % 8 — which is most keys

Nothing about this is a bug or a Kafka misbehavior — it's the direct, correct consequence of a deterministic formula that includes partition count as an input. But it means that for any topic where key-based ordering genuinely matters, partition count is not a knob you can casually turn up later "for more throughput." Turning it up silently and permanently breaks the ordering guarantee that the whole system was designed around, for every key whose new modulo differs from its old one.

A Worked Design: Partitioning an Order-Events Topic

Walking through a full example ties the previous parts together. Suppose you're designingfreshcart.orders, a topic carrying every lifecycle event for every order: placed, payment_captured, packed, shipped, delivered, cancelled. Multiple consumers care about this topic — a fulfillment service, a customer-notification service, and an analytics pipeline.

• ✓Key choice: order_id. Every event for one order must be processed in the sequence it happened — a "shipped" event must never be seen before "payment_captured" for the same order.
• ✓Distribution check: order IDs are high-cardinality and roughly even in volume per order (an order generates a handful of events, not millions) — no single order_id dominates traffic the way one customer or one product might, so this key does not create a hot partition on its own.
• ✓Partition count: sized for projected peak throughput with headroom, decided before launch — per Part 06, this cannot be casually revised later without breaking ordering for orders already in flight.
• ✓Consumer groups: fulfillment, notifications, and analytics each run their own independent consumer group against this topic, each free to scale up to the partition count without affecting the others.
• ✓What is explicitly NOT guaranteed: the relative order between order A's events and order B's events. Nothing in the system needs that, so this is an acceptable, correct gap — not a bug.

Notice what this design does not need: a custom partitioner (the default hash-based routing is exactly right for order_id), and no null-key traffic (every event genuinely belongs to a specific order). The design is almost entirely a consequence of correctly answering "what needs to stay in order, and what is that entity's identifier" — which is the question this whole module has been building toward.

It's also worth stress-testing the design against a plausible future change before calling it done: what happens if one enterprise customer starts placing an unusually large share of all orders through an automated bulk-ordering integration? Because the key is order_id, notcustomer_id, that customer's orders are still spread across every partition — each order gets its own hash, so no single partition absorbs that customer's disproportionate volume. This is a direct, concrete payoff of choosing the right granularity of key in Part 04: a coarser key like customer_id would have made this design fragile against exactly the kind of real-world skew that eventually shows up in almost every production system.

Watching for Skew Before It Pages You

Everything in Parts 04 and 06 describes problems that develop gradually and are invisible in aggregate dashboards. A topic-wide throughput graph, a topic-wide average lag number, and a topic-wide disk usage figure can all look perfectly healthy while one partition quietly absorbs a disproportionate share of load. Catching this before it becomes an incident means monitoring at the partition level, not just the topic level, from the day a keyed topic goes live.

The three signals worth tracking per partition are: consumer lag, bytes-in rate, and message-rate. Any one of these consistently diverging from its siblings — not a brief spike, but a sustained gap — is the early warning for key skew described in Part 04. Catching it early, while the skew is moderate, gives you room to salt a key or redesign before the affected consumer is visibly falling behind in a way customers or downstream teams notice.

# topic-wide average — LOOKS fine, hides the problem
$ kafka-consumer-groups --describe --group billing-group --bootstrap-server broker:9092
# TOPIC             PARTITION  LAG
# freshcart.orders  (avg across 8 partitions): 1,240   ← looks totally reasonable

# same data, per partition — the real picture
# TOPIC             PARTITION  LAG
# freshcart.orders  0          140
# freshcart.orders  1          95
# freshcart.orders  2          9,680   ← one partition carrying nearly all the lag
# freshcart.orders  3          60
# freshcart.orders  4          110
# freshcart.orders  5          130
# freshcart.orders  6          75
# freshcart.orders  7          90

# the average of these eight numbers is ~1,297 — deceptively unremarkable
# alerting on "average lag > threshold" would have caught this late, if at all
# alerting on "max partition lag > threshold" or "stddev across partitions" catches it early

How Partitioning Decisions Interact With Consumer Group Scaling

Everything covered so far is about the producer side — how a key gets a record onto a specific partition. It's worth closing the loop on the consumer side, because partitioning decisions directly determine the ceiling on how much a consumer group can scale, and a good key choice on the producer side can still be undermined by a bad partition-count decision that ignores this ceiling.

Within one consumer group, each partition is assigned to exactly one consumer instance at a time. This means the partition count set at topic-design time is a hard cap on how many consumer instances in a single group can ever be simultaneously active and doing useful work. A topic with 6 partitions cannot usefully run more than 6 consumer instances in one group — a 7th instance sits idle, receiving no partitions, no matter how much spare processing capacity it has.

topic: freshcart.orders, 6 partitions
consumer group: fulfillment-group

# 4 consumer instances running:
# fulfillment-1 → partitions 0, 1
# fulfillment-2 → partitions 2, 3
# fulfillment-3 → partitions 4
# fulfillment-4 → partitions 5

# Scale up to 6 instances — perfect 1:1 assignment:
# fulfillment-1 → partition 0
# fulfillment-2 → partition 1
# fulfillment-3 → partition 2
# fulfillment-4 → partition 3
# fulfillment-5 → partition 4
# fulfillment-6 → partition 5

# Scale up to 8 instances — 2 sit completely idle:
# fulfillment-1..6 → one partition each, same as above
# fulfillment-7 → (no partitions assigned) — idle
# fulfillment-8 → (no partitions assigned) — idle

# This ceiling is set by partition count, independent of how good the key
# choice is. A perfectly chosen key with too few partitions still limits
# how far this consumer group can ever scale.

This is the practical reason partition count planning (Part 06) and key choice (Part 04) have to be decided together, not separately. Partition count needs enough headroom for peak expected consumer parallelism, not just peak expected producer throughput — and because increasing it later breaks key-based ordering for existing data, both the throughput requirement and the scaling requirement need to be estimated at design time, together.

A Step-by-Step Cutover When Partition Count Must Change

Part 06 established the rule: don't resize an ordering-sensitive topic's partition count in place. But eventually a topic really does outgrow its original partition count — traffic grew far past what was planned, or a genuinely new consumer parallelism requirement appeared. When that happens, the safe path is a deliberate migration, not a live --alter --partitions command. It's worth walking through what that migration actually looks like end to end.

# Goal: move freshcart.orders from 8 partitions to 24 partitions
# without breaking per-order-id ordering for orders already in flight

# Step 1 — Create the new topic with the target partition count
kafka-topics --create --topic freshcart.orders.v2 --partitions 24 --replication-factor 3

# Step 2 — Dual-write: update producers to write to BOTH freshcart.orders
#           and freshcart.orders.v2 for some transition window.
#           (Or use a bridging consumer that reads the old topic and
#           republishes to the new one, if producer changes are harder to land.)

# Step 3 — Update consumers one at a time to read from freshcart.orders.v2
#           instead of freshcart.orders. Because dual-writes are happening,
#           each order's NEW events are visible on the new topic in correct
#           per-key order (hash(order_id) % 24 is now stable and consistent
#           for every event written after the cutover began).

# Step 4 — For orders that were already in flight when the migration started,
#           reconcile: any order with pre-cutover events on the OLD topic and
#           post-cutover events on the NEW topic needs application-level logic
#           to treat both as one logical sequence during the transition window.
#           This is the one piece of real complexity in the whole migration —
#           budget time for it rather than treating cutover as instantaneous.

# Step 5 — Once all consumers are confirmed reading from freshcart.orders.v2
#           and the transition window has fully drained (no more in-flight
#           orders that started before cutover), stop dual-writing and
#           decommission freshcart.orders after its retention window passes
#           (so it remains available briefly for any final reconciliation).

Notice that none of this happens inside Kafka's partitioning mechanism — the mechanism itself (hash(key) % partition_count) offers no migration path, because a migration path isn't what it's designed to do. It's a pure, stateless routing function. Moving safely to a new partition count is entirely an application-level exercise in careful dual-writing, phased consumer cutover, and explicit handling of the in-flight transition window — which is exactly why avoiding the need for this in the first place, by sizing partition count generously up front (Part 06), is so much cheaper than doing it.

Beyond a Single Field: Composite Keys and Deliberate Salting

Not every good partition key is a single existing field copied straight from your data model. Sometimes the right key is a composite — a deliberately constructed string combining two pieces of information — built specifically to balance the distribution and ordering trade-offs from Part 04.

A composite key is useful when the natural entity identifier alone would be too coarse (too few distinct values, causing skew) or too fine (splitting related records that actually need to stay together). For a multi-tenant SaaS platform, keying purely by tenant_id risks a whale tenant dominating one partition, per Part 04's table; keying purely by a finer field likeuser_id might scatter one tenant's events across every partition when the business actually needs per-tenant event ordering for its audit log. A composite key —tenant_id + "-" + shard_bucket, where shard_bucket is a small deterministic hash of the user ID — can thread that needle.

# Problem: tenant_id alone is too skewed (one huge enterprise tenant dominates)
# Requirement: still need ordering WITHIN a tenant's events, just spread across
#              more than one partition for the largest tenants

def composite_key(tenant_id: str, user_id: str, num_buckets: int = 4) -> str:
    # deterministic sub-bucket within the tenant, based on user_id
    bucket = hash(user_id) % num_buckets
    return f"{tenant_id}#{bucket}"

# Every event for tenant "acme_corp" now lands on one of 4 sub-buckets
# instead of a single partition:
composite_key("acme_corp", "user_501")   # → "acme_corp#2"
composite_key("acme_corp", "user_502")   # → "acme_corp#0"
composite_key("acme_corp", "user_503")   # → "acme_corp#2"  (same bucket as user_501)

# Trade-off being made explicitly:
# - Strict ordering is preserved PER (tenant, bucket) pair, not per tenant overall
# - acme_corp's total load now spreads across up to 4 partitions instead of 1
# - A consumer needing tenant-wide ordering must merge across those 4 buckets
#   explicitly — this key choice moved complexity from "hot partition" to
#   "consumer-side merge", which is the right trade when strict global
#   per-tenant order isn't actually required, just rough per-user order

This is the same salting idea introduced as a hot-partition mitigation in Part 04, made concrete as a designed-in key structure rather than a reactive fix. The general lesson: a partition key doesn't have to be a single raw field. It can — and for genuinely skewed, high-volume entities, often should — be an engineered value that deliberately balances distribution against the specific ordering guarantee the use case actually needs, rather than the strongest possible guarantee by default.

Why Stream Processing Joins Depend Entirely on Key and Partition Choices

Everything covered so far treats one topic at a time. A particularly high-stakes application of keying and partitioning shows up the moment you try to join two topics together in a stream processor like Kafka Streams or ksqlDB — for example, joining freshcart.orders withfreshcart.customers to enrich every order event with the customer's current tier. A stream-stream or stream-table join like this only works correctly, and only works without expensive cross-network shuffling, if the two topics are co-partitioned.

Co-partitioned means: both topics use the same key (the join key — customer_id in this example), and both topics have the same partition count. When both conditions hold, partition 0 of the orders topic and partition 0 of the customers topic are guaranteed to contain exactly the same set of customer IDs (because both were routed by the same hash(customer_id) % partition_countformula with the same partition count), so a stream processor can join them locally, partition by partition, on the same machine, with no network shuffle required. Break either condition and the join either requires an expensive repartitioning step or silently produces wrong results.

# freshcart.orders    — keyed by customer_id, 12 partitions
# freshcart.customers  — keyed by customer_id, 12 partitions
# SAME key, SAME partition count → co-partitioned

# hash("cust_88213") % 12 = 5   (evaluated identically for both topics)
# → orders for cust_88213 are in orders-partition-5
# → the customer record for cust_88213 is in customers-partition-5
# A stream processor task assigned partition 5 of BOTH topics can join them
# entirely locally — every order it sees has its matching customer record
# sitting in the same local partition, no network call needed.

# Now suppose freshcart.customers only has 6 partitions instead of 12:
# hash("cust_88213") % 6 = 2   ← DIFFERENT partition number than orders' partition 5
# The stream processor task handling orders-partition-5 does NOT have
# cust_88213's customer record locally. The join either fails to find a
# match, or (if the framework handles it) triggers an internal repartitioning
# step — re-keying and rewriting one of the topics through an intermediate
# topic just to make the join possible. That's real infrastructure cost that
# co-partitioning from the start would have avoided entirely.

In practice, teams that lean heavily on Kafka Streams or ksqlDB joins often standardize on a single partition count (or a small number of standard counts) across all topics that participate in joins, precisely to keep co-partitioning simple to reason about — rather than tuning each topic's partition count independently and then discovering a join needs repartitioning.

Where the Key Actually Comes From When Data Enters Kafka Automatically

Everything so far has assumed a producer application deliberately chooses a key. In practice, a large share of the records that end up in a Kafka topic never pass through hand-written producer code at all — they arrive via Kafka Connect source connectors, most commonly CDC (change data capture) connectors like Debezium reading a database's write-ahead log. It's worth being explicit about how keying decisions get made in that path, because it's a different mechanism from theproducer.send(key=..., value=...) calls used throughout this module, even though the underlying partitioning formula from Part 01 still applies once a key exists.

A CDC connector reading a database table almost always keys each record by that table's primary key — which is exactly the right default, because it means every change to a given row (insert, update, delete) lands on the same partition, giving you per-row ordering for free, matching the per-entity ordering pattern from Part 04. This is also precisely the setup that makes a CDC topic a natural fit for cleanup.policy=compact, covered in the companion module on retention and compaction — a compacted, primary-key-keyed CDC topic converges to a live mirror of the source table's current state.

# Source table: customers (primary key: customer_id)

# UPDATE customers SET tier = 'gold' WHERE customer_id = 'cust_88213';

# Connector produces, onto freshcart.customers.cdc:
{
  "key": { "customer_id": "cust_88213" },   # primary key becomes the Kafka record key
  "value": {
    "before": { "customer_id": "cust_88213", "tier": "silver", ... },
    "after":  { "customer_id": "cust_88213", "tier": "gold", ... },
    "op": "u",   # u=update, c=create, d=delete, r=read (snapshot)
    "ts_ms": 1732012345000
  }
}

# hash(customer_id="cust_88213") % partition_count → same partition every time,
# same as any other keyed record from Part 01 — the routing mechanism doesn't
# care whether the key came from application code or a CDC connector.

# A DELETE from the source table typically produces a tombstone-adjacent pair:
# one record with op="d" (so consumers see the deletion event itself), followed
# by a genuine null-value tombstone record for that key — this is what lets a
# CDC topic correctly participate in log compaction's key-removal mechanism.

Proving the Routing Mechanism Rather Than Trusting It Blindly

Everything in this module describes a deterministic mechanism, which means it's directly testable — you don't have to take it on faith that two records with the same key land on the same partition, or that a null key spreads load. Before relying on a partitioning strategy in production, it's worth actually observing the assignment on a local or test cluster, the same way you'd verify retention and compaction behavior rather than only trusting the configuration.

# Create a small test topic
kafka-topics --create --topic test.partitioning --partitions 4 --replication-factor 1

# Produce a few records with explicit keys, printing the partition each lands on
kafka-console-producer --topic test.partitioning --bootstrap-server localhost:9092 \
  --property "parse.key=true" --property "key.separator=:"
# type: user_9271:order-A
# type: user_4410:order-B
# type: user_9271:order-C
# (Ctrl+D to exit)

# Consume with partition info visible to confirm the mapping:
kafka-console-consumer --topic test.partitioning --bootstrap-server localhost:9092 \
  --from-beginning --property print.key=true --property print.partition=true

# Expected output confirms Part 01's mechanism directly:
# Partition:2 user_9271  order-A
# Partition:0 user_4410  order-B
# Partition:2 user_9271  order-C     ← same partition as order-A, same key

# Now repeat with partitions=8 on a NEW topic using the same keys and observe
# that user_9271 very likely lands on a DIFFERENT partition number than it
# did with 4 partitions — a hands-on demonstration of Part 06's partition-
# count trap, safely observed on disposable test data instead of production.

This kind of direct verification is especially worth doing before a partition-count change on any topic you're not sure is safe to resize, and before trusting a new custom partitioner's logic (Part 05) — printing the actual partition assignment for a representative sample of real keys is a fast, concrete way to catch a routing bug before it reaches production, rather than discovering it weeks later as an ordering anomaly nobody can immediately explain.

What "representative" keys means in practice

A quick spot-check with two or three convenient test keys is not the same thing as verifying distribution. Because a good key's whole value proposition is even spread across the real, production-shaped population of key values, a meaningful local test samples real (or realistically shaped) key values at real volume — pulling a sample of actual customer IDs or order IDs from a staging dataset, if one exists, rather than typing user1, user2,user3 by hand. A handful of sequential test keys can easily hash to a suspiciously even spread purely by coincidence, giving false confidence that a skewed production key population would not actually deliver.

Five Misconceptions About Keys and Partitioning

What This Looks Like on Day One

5 Interview Questions — With Complete Answers

Mistakes Beginners Make Constantly

Errors and Symptoms You Will Hit — And Exactly Why