Consumer Groups and Offsets

A Consumer Group Is a Named Claim on a Set of Partitions

A consumer group is not a queue, a thread pool, or a load balancer in the traditional sense. It is a set of consumer processes that share one group.id string. Kafka's job is to hand out the partitions of every topic the group subscribes to across the consumers currently in that group, so that each partition is owned by exactly one consumer within the group at any moment. That single sentence is the entire mental model, and almost every consumer-group bug traces back to forgetting one clause of it — "within the group," "at any moment," or "exactly one."

The "within the group" clause is what makes Kafka behave like a queue and a pub-sub system simultaneously, the same duality covered for brokers generally in the Data Engineering track. Two consumers in the same group compete — Kafka gives each one a disjoint slice of the partitions, so records are spread across them like workers pulling from a shared queue. Two consumers in different groups reading the same topic do not compete at all. Each group gets its own independent read position over the same log. A payments-processing group and a fraud-analytics group can both subscribe to the orders topic, each seeing every record, each moving through the log at its own pace, with zero coordination between the two groups.

orders topic — 6 partitions

group: payments-service (3 consumer instances)
  payments-1  owns partitions 0, 1
  payments-2  owns partitions 2, 3
  payments-3  owns partitions 4, 5

group: fraud-analytics (1 consumer instance)
  fraud-1     owns partitions 0, 1, 2, 3, 4, 5

payments-service has processed through offset 940,201 on partition 0.
fraud-analytics has processed through offset 812,004 on the same partition.
Neither group affects the other's position. Neither "removes" records for
the other. Both are reading the same durable log independently.

Every Group Has a Coordinator Broker That Runs the Protocol

For every consumer group, one broker in the cluster is elected as that group's group coordinator. Which broker it is depends on hashing the group id to a partition of the internal __consumer_offsets topic — the coordinator is whichever broker currently leads that partition. Consumers discover their coordinator the same way producers discover partition leaders: they ask any broker via bootstrap.servers, and the cluster tells them.

The coordinator's job is to run the membership protocol: track which consumer instances are currently alive in the group, decide when a rebalance is needed, choose a partition assignment (or delegate the assignment computation to a chosen "group leader" consumer, depending on the assignor), and distribute that assignment back out. It also owns the group's offset storage — commits and fetches for that group's offsets go through the coordinator.

Heartbeats are how the coordinator knows you're alive

Each consumer runs a background heartbeat thread that pings the coordinator on an interval (heartbeat.interval.ms, default 3 seconds). If the coordinator does not hear a heartbeat from a member within session.timeout.ms (default 45 seconds in modern clients), it considers that consumer dead and evicts it from the group — triggering a rebalance so its partitions can be reassigned to a still-living consumer. This is deliberately decoupled from the main polling loop: heartbeats run on their own thread precisely so that a consumer doing slow record processing between poll() calls is not mistaken for a dead one.

There is a second, related timeout that catches a different failure mode: max.poll. interval.ms (default 5 minutes). This bounds how long a consumer is allowed to go between calls to poll() itself. A consumer that is alive at the heartbeat-thread level but stuck processing a batch — deadlocked, or just genuinely too slow — will still get evicted once it blows past this interval, because the coordinator's real concern is whether the consumer is making forward progress, not merely whether its process is running.

A Rebalance Is Kafka Recomputing Who Owns What

A rebalance is the process of reassigning partitions among the members of a consumer group. It is not an error condition by itself — it is the mechanism that lets a group adapt to a changing set of consumers or a changing set of partitions. But a rebalance is disruptive: assigned partitions are revoked and reassigned, which means, depending on the protocol, active fetching from those partitions can pause. Understanding what triggers one is the difference between an expected, brief blip and a mysterious recurring outage.

• ✓A new consumer instance joins the group (a deploy adding a pod, a scale-out event).
• ✓An existing consumer leaves the group cleanly (a graceful shutdown that calls consumer.close()).
• ✓An existing consumer is considered dead by the coordinator — missed heartbeats past session.timeout.ms, or a stalled poll loop past max.poll.interval.ms.
• ✓The set of partitions a subscribed topic has changes (a topic is repartitioned to add partitions).
• ✓A consumer changes its subscription — for example calling subscribe() with a different topic pattern.
• ✓The group coordinator itself changes broker (rare, but forces members to rediscover it and can coincide with a rebalance).

Every rebalance is tagged with a generation id — a monotonically increasing integer the coordinator increments each time it recomputes group membership. This number exists to reject stale requests: if a consumer's heartbeat or offset commit arrives carrying an old generation id, the coordinator rejects it, because that consumer is operating on an assignment that has already been superseded. This is what prevents a consumer that was evicted and is unaware of it — a "zombie" — from continuing to commit offsets for partitions it no longer owns.

orders-consumer group, generation 41, 4 partitions, 4 consumers (1 each)

deploy starts: pod orders-consumer-2 is terminated
  -> coordinator misses its heartbeats (or sees a clean leave request)
  -> generation increments to 42
  -> rebalance: partitions 0,1,2,3 are reassigned among the 3 remaining consumers
  -> some consumers now own partitions they didn't own a moment ago

new pod orders-consumer-2 (replacement) starts and joins
  -> generation increments to 43
  -> rebalance again: partitions reshuffled among 4 consumers

net effect of one pod replacement: two rebalances, each one briefly pausing
fetches on affected partitions while the new assignment is computed and applied

Eager Rebalancing Stops the World; Cooperative Rebalancing Doesn't

How disruptive a rebalance is depends on which partition assignment strategy the group is configured with. This is a production distinction that matters far more than most introductory material suggests, because the historical default behaved in a way that surprises people the first time they watch it happen at scale.

Eager rebalancing — revoke everything, then reassign everything

With the classic eager protocol (assignors like RangeAssignor andRoundRobinAssignor), a rebalance works in two hard phases. First, everyconsumer in the group gives up all of its currently assigned partitions — including the ones it would end up keeping anyway. Only after every member has revoked everything does the coordinator compute the new assignment and hand partitions back out. For the entire window between revoke and reassign, the whole group stops consuming. This is the "stop-the-world" behavior: one new consumer joining a 50-consumer group pauses all 50, not just the ones whose assignment actually changes.

group has 5 consumers, 10 partitions, 2 partitions each. consumer-6 joins.

phase 1 (revoke): ALL 5 existing consumers give up ALL of their partitions
  -> for a moment, nobody in the group owns anything
  -> nothing is being consumed from any of the 10 partitions

phase 2 (reassign): coordinator computes a fresh assignment across 6 consumers
  -> each of the 6 gets partitions back (most get the same 1-2 they had before,
     purely by coincidence of the assignment algorithm, not because it was preserved)

net effect: the entire group was idle during the revoke/reassign window,
even though only 1 out of 6 consumers' assignment was actually new

Cooperative sticky rebalancing — only revoke what actually has to move

CooperativeStickyAssignor changes this fundamentally. A rebalance still happens, but it runs in incremental rounds: the coordinator computes the new assignment, and each consumer only revokes the specific partitions it is actually losing — partitions it keeps stay assigned and keep being fetched the entire time. "Sticky" means the assignor also biases toward keeping a consumer's existing partitions when computing the new assignment in the first place, minimizing how much actually needs to move. The practical effect: a single consumer joining or leaving a large group now disrupts only the small number of partitions that genuinely change owners, not the whole group.

An Offset Is a Position in a Log — Committed Offsets Live in Kafka Itself

An offset is simply the sequential position of a record within one partition — 0, 1, 2, 3, and so on, strictly increasing, never reused. It has meaning only relative to a specific topic-partition; "offset 500" on orders partition 0 has nothing to do with "offset 500" onorders partition 1. There are two offsets worth keeping distinct in your head: the log-end-offset (the position of the next record that will be written — how far the partition has grown) and the committed offset (the position a given consumer group has durably recorded as "everything before this has been safely processed").

A very common belief left over from older Kafka deployments is that offsets are stored in ZooKeeper. They are not, and have not been since Kafka 0.9. Committed offsets are stored as records in an internal Kafka topic named __consumer_offsets — a normal, replicated, partitioned Kafka topic, created automatically, with 50 partitions by default. Each commit is itself a Kafka write: a record keyed by (group.id, topic, partition) whose value encodes the committed offset, written to whichever partition of__consumer_offsets that key hashes to. The topic is compacted, so only the latest committed offset per key is retained long-term — exactly the same log-compaction mechanism used for changelog-style topics elsewhere in Kafka.

group.id = payments-service commits offset 940,202 for orders partition 0

this is written as a Kafka record to __consumer_offsets:
  key:   (group=payments-service, topic=orders, partition=0)
  value: (offset=940202, metadata=..., commit_timestamp=...)

which partition of __consumer_offsets does this land on?
  partition = hash("payments-service") % 50

so ALL offset commits for a given group.id land on the SAME
__consumer_offsets partition, and are handled by that partition's
leader broker — which is exactly the broker that acts as the
group coordinator for that group.id

Auto-Commit vs Manual Commit, and Exactly When You Commit

By default, most Kafka clients auto-commit offsets on a timer — enable.auto.commit=truewith auto.commit.interval.ms defaulting to 5 seconds. Every five seconds, the client library commits the offset of the latest record returned by poll(), regardless of whether your application has actually finished doing anything useful with it. This is convenient and, for workloads where an occasional reprocessed or skipped record is harmless, perfectly fine. For anything where correctness matters, it is a trap.

The trap: auto-commit fires on a wall-clock timer that has no relationship to your processing. If your application pulls a batch of records, starts processing them, and crashes halfway through — after the auto-commit timer already fired for offsets beyond where processing actually got to — those records are silently skipped on restart. The consumer resumes from the committed offset, which is now ahead of the last record actually processed.

The manual-commit alternative: commit only after real work is durable

Turning off auto-commit (enable.auto.commit=false) and committing explicitly, after your application has confirmed the record's effects are durable — written to a database, published downstream, whatever "done" means for that pipeline — moves the failure mode from "silently skipped" to "possibly reprocessed." That is a strictly better failure mode for most business logic, because reprocessing can be made safe with idempotency (a unique key, an upsert, a dedup check), while a silently skipped record usually cannot be recovered at all without a full reprocess from an earlier offset.

while running:
    records = consumer.poll(timeout=1.0)

    for record in records:
        result = process(record)          # e.g. compute a derived value
        db.upsert(record.key, result)     # durable, idempotent write FIRST

    consumer.commit()   # only now — after every record in the batch is durably written

Manual commits come in two flavors. commitSync() blocks until the coordinator acknowledges the commit, retrying on retriable failures — it is slower per call but you know for certain the commit landed (or got an exception telling you it didn't) before you move on. commitAsync() fires the commit and continues immediately, calling back on completion — higher throughput, but a failed async commit is easy to lose track of if you don't handle the callback, and async commits can complete out of order under retries. A common, well-tested pattern: use commitAsync() during the normal processing loop for throughput, and a final commitSync() on shutdown to guarantee the last commit is durable before the process exits.

What Happens When a Consumer Comes Back — and When There's Nothing to Come Back To

When an existing consumer group restarts, it asks the coordinator for its last committed offset per partition and resumes exactly there. This is the entire point of committing offsets in the first place — the group's progress lives in Kafka, not in the process, so a restart, a redeploy, or a total replacement of every consumer instance in the group loses no memory of where it was.

But there is a case with no committed offset to resume from: a brand-new group.id that has never committed anything for a partition, or an existing group whose committed offset has aged out of retention (offsets.retention.minutes, default 7 days — a group that has been offline longer than that loses its position entirely). In either case, Kafka has nothing to resume from, and falls back to the auto.offset.reset policy.

This is also exactly why standing up a brand-new consumer group against a topic that already has months of history is a decision, not a default. If that new group id has never committed anything and auto.offset.reset=earliest, it will start consuming from the oldest retained record — potentially reprocessing a huge volume of historical data the moment it starts, which can overwhelm a downstream system that wasn't expecting a flood.

Consumer Lag Is the Health Signal; Static Membership Avoids Needless Rebalances

Consumer lag is the gap between the log-end-offset of a partition and the offset a consumer group has committed for it: lag = log-end-offset − committed-offset, measured in number of records. Lag of zero means the group is fully caught up. A small, stable, or oscillating lag means the group is keeping pace with bursts and draining them. A lag that trends upward over a sustained window means the group is structurally slower than the rate records are being produced, and — left alone — it only gets worse, since nothing about Kafka causes a falling-behind consumer to automatically catch up.

Lag should always be checked per partition, not just summed across the group. A healthy-looking total can hide one badly lagging partition — a hot key concentrating disproportionate volume onto a single partition, which only one consumer instance can ever own regardless of how many consumers exist in the group. Tools like kafka-consumer-groups.sh --describe --group <id>report lag per partition specifically because the aggregate number is not enough to diagnose a skew problem.

kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --describe --group payments-service

GROUP             TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
payments-service  orders  0          940310          940310          0
payments-service  orders  1          812004          812004          0
payments-service  orders  2          551200          812440          261240   <- badly lagging
payments-service  orders  3          940190          940310          120

Static group membership — surviving a restart without a rebalance

By default, every consumer process is a fresh, anonymous member of its group — when it restarts, the coordinator sees it as a brand-new member joining, which triggers a rebalance, exactly as described in Part 03. For a rolling restart of many instances, this produces exactly the rebalance storm called out earlier, even though from a human's point of view nothing really "changed" — the same set of instances just cycled one at a time.

Setting group.instance.id to a stable, unique string per consumer instance turns that instance into a static member. When a static member disconnects and reconnects within session.timeout.ms, the coordinator recognizes it as the same member returning — not a departure followed by an arrival — and simply hands its previous assignment back without triggering a rebalance at all. This converts a rolling restart from "one or two rebalances per pod cycled" into "zero rebalances, as long as each pod comes back reasonably quickly."

# each consumer pod gets a stable, unique instance id (e.g. derived from pod name)
group.id=payments-service
group.instance.id=payments-service-pod-3
session.timeout.ms=45000   # window in which a static member can reconnect without a rebalance

Assignment Strategy Also Decides How Partitions Split Across Topics

Everything in Part 04 about eager versus cooperative rebalancing describes how a rebalance plays out mechanically. A separate question is which partitions each consumer ends up with when a group subscribes to more than one topic at once — a very common real-world shape, since many services consume several related topics with one group id rather than running a separate group per topic.

RangeAssignor assigns partitions topic by topic, independently, trying to give consecutive consumers consecutive partition ranges within each topic. This sounds harmless until you notice the failure mode: because each topic is assigned independently using the same ordering of consumers, the same consumer instance tends to land at the "front of the line" for every topic, and can end up with a disproportionate share of the total partitions across all subscribed topics while another consumer gets comparatively few.

group subscribes to: orders (3 partitions), payments (3 partitions)
3 consumers: c1, c2, c3

RangeAssignor computes each topic's assignment independently, same consumer order:
  orders:   c1 -> [0,1]   c2 -> [2]     c3 -> []
  payments: c1 -> [0,1]   c2 -> [2]     c3 -> []

totals:
  c1 owns 4 partitions
  c2 owns 2 partitions
  c3 owns 0 partitions   <- sitting idle despite being a paid-for running instance

RoundRobinAssignor and CooperativeStickyAssignor both spread partitions across
ALL subscribed topics as one combined pool, avoiding this per-topic repetition:
  c1 -> orders[0], payments[1]
  c2 -> orders[1], payments[2]
  c3 -> orders[2], payments[0]
  every consumer gets 2 — evenly split

It is also worth being precise about subscribe() versus assign(). Callingsubscribe() with a topic name or pattern hands partition assignment over to the group coordinator entirely — this is the consumer-group model described throughout this module, with rebalancing, heartbeats, and generation ids all in play. Calling assign() instead lets an application manually pin itself to specific partitions, bypassing the group protocol altogether. Manual assignment has real uses — a Kafka Streams-style application doing custom partition-to- instance mapping, or a tool that genuinely needs to read one specific partition regardless of group membership — but it forfeits every benefit covered in this module: no automatic rebalancing on scale-out, no automatic failover if that instance dies, and no group-tracked offset management unless you build it yourself.

What to Actually Watch, and What Good Looks Like

Everything covered so far is mechanism. In production, the mechanism only matters insofar as it shows up in a small set of metrics that tell you whether a consumer group is healthy — and, more usefully, whether it is about to become unhealthy before anyone notices from user-facing symptoms.

A healthy consumer group, viewed on a dashboard, looks almost boring: per-partition lag hovering near zero with brief, self-correcting spikes during load bursts; rebalances only around expected deploy windows and quickly settling; commit latency flat and low. The interesting failures in production are rarely a total outage — they are one of these signals quietly drifting in the wrong direction for hours before anyone notices, which is exactly why they are worth alerting on directly rather than waiting for a downstream symptom like a customer-facing delay.

# Per-partition lag, not aggregate — catches hot-key skew (Part 08)
alert: max(kafka_consumer_lag) by (group, topic, partition) > 100000
       AND deriv(kafka_consumer_lag[10m]) > 0

# Rebalance frequency outside deploy windows — catches assignor / static
# membership misconfiguration (Parts 03-04, 08)
alert: increase(kafka_consumer_group_rebalances_total[15m]) > 3
       AND NOT deploy_in_progress

# Active member count below expected replica count — catches consumers
# stuck evicted in a crash loop (Part 02)
alert: kafka_consumer_group_active_members < expected_replica_count
       for: 5m

isolation.level Decides Whether a Consumer Sees In-Flight Transactions

Everything in this module assumes a consumer reads whatever is physically appended to a partition's log. That assumption gets one important qualifier when the producer side is using Kafka transactions — the read-process-write pattern where a producer writes to an output topic and commits a consumer offset atomically, as one unit. A transactional producer's writes land in the log the moment they're sent, before the transaction actually commits or aborts — which means a naive consumer reading that log could see data from a transaction that later gets aborted and rolled back, or read a record before the rest of its transaction's records have landed.

The consumer-side control for this is isolation.level. The default, read_uncommitted, hands back every record the moment it's in the log, transactional or not, committed or not — this is fine for a consumer group whose upstream producers never use transactions, but wrong for a consumer of a topic written to by a transactional producer, because it can observe records from a transaction that is later aborted and never actually should have existed from the application's point of view. read_committed instead buffers and withholds records belonging to an open transaction until that transaction's outcome is known, delivering them only if the transaction commits, and silently skipping them if it aborts.

producer begins a transaction, writes 3 records to orders.enriched,
then the transaction aborts (an exception during processing, e.g.)

isolation.level=read_uncommitted (the default):
  consumer sees all 3 records immediately as they're appended
  -> consumer processes records from a transaction that was later rolled back
  -> the application-level effect of an aborted operation leaks downstream anyway

isolation.level=read_committed:
  consumer's fetch withholds those 3 records while the transaction is open
  transaction aborts -> those 3 records are never delivered to this consumer at all
  -> exactly matches the producer's intent: an aborted transaction produced nothing

This setting is scoped per consumer group, not per producer, which means the decision belongs to whoever owns the consuming application: a group reading a topic it knows is written transactionally should set read_committed explicitly rather than relying on a default that happens to be permissive. Getting this wrong doesn't throw an error or fail loudly — the consumer just silently sees records it shouldn't, which is exactly the kind of bug that surfaces much later as an unexplained downstream inconsistency rather than as an obvious failure at the point where the mistake was actually made.

Putting It Together — A Consumer Loop That Survives Real Production Conditions

Every individual mechanism in this module — commits, rebalances, lag, static membership, isolation level — is straightforward on its own. What makes consumer design genuinely hard in practice is that production conditions combine several of them at once: a rebalance happens mid-batch, a downstream write times out right before a commit, a deploy restarts half the group while the other half is still catching up on lag from an earlier spike. A consumer designed only against the happy path tends to work perfectly in every demo and staging environment, then surface a subtle correctness bug the first time two of these conditions overlap in production.

Handling a revoked partition mid-batch

A cooperative rebalance (Part 04) can revoke a specific partition out from under a consumer while it is in the middle of processing a batch fetched from that partition. Client libraries expose this through a rebalance listener — onPartitionsRevoked fires before partitions are taken away, giving the application a chance to finish and commit any in-flight work for exactly those partitions before ownership actually changes. Ignoring this callback and committing blindly on the normal loop schedule risks committing an offset for a partition the consumer no longer owns, which the coordinator will reject once the generation id has moved on, per Part 03.

class SafeRebalanceListener(ConsumerRebalanceListener):
    def __init__(self, consumer, pending_offsets):
        self.consumer = consumer
        self.pending_offsets = pending_offsets  # offsets processed but not yet committed

    def on_partitions_revoked(self, revoked_partitions):
        # Commit only the offsets for partitions actually being taken away —
        # finishing in-flight work for THOSE partitions before losing ownership
        to_commit = {
            tp: offset for tp, offset in self.pending_offsets.items()
            if tp in revoked_partitions
        }
        if to_commit:
            self.consumer.commit(offsets=to_commit)

    def on_partitions_assigned(self, assigned_partitions):
        # Newly assigned partitions start from their last committed offset automatically —
        # nothing to do here beyond logging, unless warming local state is needed

Bounding retries so a stuck downstream write doesn't stall the whole partition

A slow or failing downstream dependency — a database under load, a flaky third-party API — inside the processing loop has the same effect on a consumer group as a poison message covered for message brokers generally: unbounded retries on one record block every record behind it in that partition, and eventually trip max.poll.interval.ms from Part 02, causing an eviction and rebalance on top of the original problem. Bounding retries with a real limit, and routing records that exhaust their retries to a separate topic for later inspection, keeps one bad record from taking an entire partition's throughput down with it — the same dead-letter-queue pattern used for message brokers generally, applied specifically at the consumer-group layer.

Choosing where lag is allowed to hide

Finally, a consumer's design should make an explicit choice about where temporary slowness is allowed to accumulate. Buffering a large batch in application memory before committing anything trades commit overhead for a bigger reprocessing window if the process crashes mid-batch; committing very frequently trades some throughput for a much smaller window of potential reprocessing. Neither is universally correct — a payments consumer usually wants the smaller, more frequent commits even at some throughput cost, while a high-volume analytics consumer usually wants the opposite. The point is that this should be a deliberate choice made with Part 06's commit-timing tradeoffs in mind, not an accident of whatever batch size a library defaults to.

Five Misconceptions About Consumer Groups and Offsets

What This Looks Like on Day One

5 Interview Questions — With Complete Answers

Mistakes Beginners Make Constantly

Errors You Will Hit — And Exactly Why They Happen