Replication, Leaders, and ISR

A Partition on One Disk Is One Failure Away From Gone

Every partition of every Kafka topic ultimately lives on the local disks of some set of broker machines. Disks fail. Machines get decommissioned, lose power, or run out of memory and get killed by the OS. Whole availability zones occasionally go dark. If a partition existed as a single copy on a single broker, any one of those ordinary, expected failures would permanently destroy every record in it — not delay access to the data, destroy it, with no way to recover.

Replication is Kafka's answer: instead of one copy of a partition's log, Kafka maintains multiple copies, spread across different broker machines (and, if you configure rack awareness, across different racks or availability zones), so that the loss of any single broker still leaves the data intact on the others. This is not an advanced feature you opt into for special topics — it is the basic durability model every production Kafka deployment relies on, and it is worth understanding precisely, because the exact guarantees it provides depend on several settings working together correctly, not on any one of them alone.

Replication Factor Is a Count of Copies, Placed on Different Brokers

The replication.factor of a topic (settable per topic, with a cluster-wide default) is the number of copies Kafka maintains of every partition in that topic — one leader replica and replication.factor − 1 follower replicas, always placed on different broker machines from each other. A replication factor of 3 on a topic with 4 partitions does not mean 3 brokers total; it means each of the 4 partitions individually has 3 copies, and Kafka spreads those copies (and which broker leads which partition) across the cluster so that no single broker is overloaded and no single broker's failure takes out every partition's only copy.

orders topic, replication.factor=3, 4 partitions, cluster of 5 brokers

partition 0: leader=broker-1  replicas=[broker-1, broker-2, broker-3]
partition 1: leader=broker-2  replicas=[broker-2, broker-3, broker-4]
partition 2: leader=broker-3  replicas=[broker-3, broker-4, broker-5]
partition 3: leader=broker-4  replicas=[broker-4, broker-5, broker-1]

notice: leadership is spread across brokers 1-4, not concentrated on one broker
notice: every partition has exactly 3 total copies, on 3 DIFFERENT brokers
notice: broker-5 holds replicas but currently leads nothing — still doing useful work

A replication factor of 1 means no replication at all — a single copy, the scenario Part 01 warns against. A replication factor of 2 tolerates exactly one broker failure at a time, but leaves zero margin if a second broker fails before the first is replaced and re-replicated. Replication factor 3 is the conventional production default for a reason: it tolerates one broker failure while still maintaining a full 2 remaining copies, giving you a safety margin during the (sometimes lengthy) window it takes to replace a failed broker and let it fully re-replicate.

Only the Leader Serves Clients — Followers Exist Purely to Replicate

For a given partition, exactly one of its replicas is the leader at any moment, and every producer write and every consumer read for that partition goes through that leader broker exclusively. Followers do not serve client reads or writes at all under normal operation — their entire job is to continuously fetch new records from the leader's log and append them to their own local copy, keeping pace with the leader as closely as possible.

This is a deliberate simplification compared to systems that allow reads from any replica. Kafka's leader-only model avoids an entire class of consistency problems that arise when a client can read from a replica that happens to be slightly behind — there is never a question of "which replica did I read from and how stale was it," because there is only one replica clients ever talk to for that partition. The tradeoff is that a follower's CPU and network capacity for serving reads sits unused by client traffic; that capacity exists purely as replication insurance and (in clusters with follower-fetching enabled for reduced cross-zone traffic) for other followers to read from instead of always adding load to the leader.

producer wants to write to orders partition 2
  -> looks up metadata: partition 2 leader = broker-3
  -> sends write directly to broker-3
  -> broker-3 (leader) appends to its local log
  -> broker-3's followers (say broker-4, broker-5) fetch the new record
     and append it to their own local copies, independently, on their own schedule

consumer wants to read orders partition 2
  -> looks up metadata: partition 2 leader = broker-3
  -> sends fetch request directly to broker-3
  -> broker-4 and broker-5, even though they have the same data, are never contacted
     for this read under normal client fetches

The ISR Is the Live List of Replicas Actually Caught Up

Not every replica listed for a partition is necessarily caught up at every moment. A follower can fall behind — a slow disk, network congestion, a garbage collection pause, or simply being overloaded relative to the leader's write rate. The in-sync replica set (ISR) is the leader's live-tracked list of which replicas — including itself — are currently keeping up closely enough to be considered "in sync," and it is this list, not the static replication factor, that determines what a durable acknowledgement actually means at any given moment.

A follower is removed from the ISR when it falls behind the leader's log by more than replica.lag.time.max.ms (default 30 seconds in modern Kafka) — specifically, when the follower has not fetched up to the leader's log-end-offset within that time window. It is added back to the ISR once it catches up. This is a continuously recalculated set, not a one-time configuration; a healthy cluster's ISR for a partition is normally all of its replicas, and it shrinks only when something is actually going wrong with a specific follower.

orders partition 0, replication.factor=3
  replicas: [broker-1 (leader), broker-2, broker-3]
  ISR (t=0): [broker-1, broker-2, broker-3]   <- all caught up

broker-3 hits a long GC pause and stops fetching for 45 seconds
  (replica.lag.time.max.ms = 30000)

  ISR (t=45s): [broker-1, broker-2]           <- broker-3 dropped from ISR
  broker-3 is still a replica, still holds most of the data,
  but is no longer counted for acks=all durability until it catches up

broker-3 recovers, fetches rapidly, catches up to the leader's log-end-offset

  ISR (t=60s): [broker-1, broker-2, broker-3] <- broker-3 rejoins the ISR

When a Leader Broker Dies, a New One Is Elected From the ISR

When the broker currently leading a partition fails — crashes, is network-partitioned, or is taken down for maintenance — that partition needs a new leader immediately, or it becomes unavailable for both reads and writes. The cluster's controller (covered in Part 07) detects the failure and elects a new leader for every partition the failed broker was leading. Under normal, clean leader election, the new leader is chosen from the partition's current ISR — a replica that was, by definition, fully caught up with the old leader's log at the moment of failure.

This is the entire point of maintaining an ISR rather than just a replica list: electing a new leader from a replica that was actually in sync guarantees the new leader's log contains every record that was ever fully acknowledged under acks=all. Nothing acknowledged is lost, because the replica taking over already had it.

orders partition 0: leader=broker-1, ISR=[broker-1, broker-2, broker-3]
broker-1 crashes

controller elects a new leader from the current ISR: broker-2 (or broker-3)
  -> broker-2 becomes the new leader
  -> broker-2's log already contained every record that had been
     acknowledged to producers under acks=all, because it was in the ISR
  -> no acknowledged data is lost
  -> broker-3 continues following the new leader, broker-2
  -> once broker-1 recovers, it rejoins as a follower and catches back up

Unclean leader election — the dangerous escape hatch

What if every in-sync replica is also unavailable at the moment of failure — say, bothbroker-1 (the leader) and broker-2 are down, and only broker-3is alive, but broker-3 had already fallen out of the ISR before the failure because it was lagging? The setting unclean.leader.election.enable decides what happens next. If it is true, Kafka is allowed to elect that out-of-sync replica as the new leader anyway, in the name of availability — the partition comes back online, but its new leader's log is missing whatever records the lagging replica hadn't caught up to yet, including some that may have already been acknowledged to producers. Those records are not merely delayed. They are gone, silently, from the client's point of view.

orders partition 0: leader=broker-1, replicas=[broker-1, broker-2, broker-3]
broker-3 had already fallen out of the ISR (lagging 200 records behind)
ISR at time of failure: [broker-1, broker-2]

broker-1 AND broker-2 both go down simultaneously (a correlated failure —
rack outage, bad deploy, etc). Only broker-3 (out of sync) is left alive.

unclean.leader.election.enable=true:
  -> broker-3 is elected leader anyway, missing its last 200 records
  -> those 200 records, some of which may have been acked to producers
     under acks=all when broker-1 was still leader, are gone
  -> the topic is available again, but has silently lost data

unclean.leader.election.enable=false (the safer default in modern Kafka):
  -> the partition simply stays unavailable — no leader is elected
  -> no data is lost, but the partition cannot serve reads or writes
     until broker-1 or broker-2 comes back and can resume as leader

acks and min.insync.replicas Together Define What "Acknowledged" Actually Means

This is the single most commonly mis-taught relationship in Kafka, so it is worth being exact. Three separate settings interact to define your real durability guarantee, and no one of them alone tells you the whole story: the producer's acks, the topic's min.insync.replicas, and the current size of the ISR at the moment of the write.

acks — how many replicas the producer waits for

min.insync.replicas — the floor that makes acks=all actually mean something

Here is the detail that trips people up: acks=all by itself means "wait for every replica currently in the ISR" — not "wait for replication.factor replicas." If the ISR has shrunk to just the leader (every follower has fallen behind or is down),acks=all is satisfied by the leader alone confirming the write. That write is now exactly as durable as acks=1 would have been — a single point of failure — even though the producer configured acks=all in good faith believing it was safe.

min.insync.replicas is the topic-level setting that closes this gap. It sets a minimum ISR size the leader will accept an acks=all write against at all — if the current ISR is smaller than min.insync.replicas, the leader rejects the write outright with aNotEnoughReplicasException, rather than accepting it and quietly providing weaker durability than the producer expects. This is what actually turns acks=all from "wait for whatever happens to be in sync right now" into "wait for a real, guaranteed number of replicas, or refuse the write."

healthy state: ISR = [broker-1 (leader), broker-2, broker-3]  (size 3)
  acks=all write -> leader waits for confirmation from all 3 ISR members
  write succeeds -> durable against the loss of any 1 or even 2 of these 3 brokers

one follower falls out of the ISR (still 2 members left): ISR = [broker-1, broker-2]
  acks=all write -> leader waits for confirmation from both remaining ISR members
  write succeeds because ISR size (2) still meets min.insync.replicas (2)
  durable against the loss of 1 more broker, no more margin left

a second follower falls out of the ISR: ISR = [broker-1]  (just the leader)
  acks=all write -> ISR size (1) is BELOW min.insync.replicas (2)
  the leader REJECTS the write: NotEnoughReplicasException
  the producer sees a clear failure and can retry, alert, or fail the request —
  instead of getting a false "success" backed by only one broker's disk

Someone Has to Decide Leadership — That's the Controller's Job

All of the leader-election behavior in Part 05 has to be decided by something — some component of the cluster has to detect a broker failure, know which replicas are currently in each partition's ISR, and actually record "this broker is now the leader of this partition" in a way every other broker and every client can find out about. In modern Kafka, that component is the KRaft controller — a small quorum of controller-eligible brokers that use the Raft consensus protocol among themselves to agree on cluster metadata, including partition leadership, without depending on an external system.

You do not need deep Raft internals to work productively with Kafka day to day, and this module deliberately does not go there — the point to take away is narrower: there is always exactly one active controller for the cluster at a time, it is the component that notices a broker has stopped sending heartbeats, it is the component that consults the ISR to pick a new leader per Part 05's rules, and it is the component that then propagates "here is the new leader for partition X" as metadata every broker and every client refreshes and relies on. When a client sees aNotLeaderOrFollower error and refreshes its metadata, it is asking the cluster — ultimately backed by the controller's metadata — who the current leader actually is now.

broker-1 (leader of several partitions) stops responding

active controller notices broker-1 has missed its expected heartbeats
  -> for each partition broker-1 was leading:
       consult that partition's current ISR
       elect a new leader from the ISR (or apply unclean election policy
         if configured and the ISR is empty of live replicas)
       record the new leader assignment in cluster metadata
  -> the new leadership metadata propagates to every broker

clients that try to reach broker-1 get connection failures or stale-metadata
errors, refresh their metadata from any reachable broker, learn the new
leader, and resume sending requests to the correct broker — usually within
a few seconds of the failure being detected

Leadership Doesn't Automatically Move Back When a Broker Recovers

When a leader broker fails and a new leader is elected from the ISR per Part 05, the cluster does not automatically move leadership back once the original broker recovers and rejoins as a follower. It simply keeps following the current leader, fully caught up, indefinitely. This is intentional — moving leadership back automatically the instant a broker returns would itself be disruptive, and a broker that just recovered from a failure is not necessarily the broker you'd want serving live traffic again immediately.

But left unaddressed across many partitions and many recoveries over time, this produces leadership skew: leadership for a disproportionate number of partitions drifts onto whichever brokers happened to survive the most failures, while brokers that failed and recovered end up leading very little, sitting mostly idle as followers despite being fully healthy. Since only the leader serves client traffic for a partition (Part 03), this translates directly into uneven load — some brokers running hot serving reads and writes for many partitions, others under-utilized.

Kafka tracks each partition's preferred leader — the replica that was the leader when the partition was originally created or last explicitly reassigned, typically the first entry in the replica list. A preferred leader election, triggered manually or automatically on a schedule (auto.leader.rebalance.enable=true, checked againstleader.imbalance.check.interval.seconds), moves leadership back to the preferred leader whenever it is healthy and in the ISR, restoring the cluster's intended even distribution.

initial state, evenly distributed:
  partition 0: preferred=broker-1  current leader=broker-1
  partition 1: preferred=broker-2  current leader=broker-2
  partition 2: preferred=broker-3  current leader=broker-3

broker-1 fails; partition 0's leader fails over to broker-2 (from its ISR)
broker-1 recovers, rejoins as a healthy in-sync follower of partition 0
  -> but broker-2 is still leading BOTH partition 0 and partition 1 now
  -> broker-2 is doing 2x the leader work it was designed for
  -> broker-1, fully healthy, leads nothing

preferred leader election runs (manually via kafka-leader-election.sh,
or automatically if auto.leader.rebalance.enable=true):
  -> partition 0's preferred leader (broker-1) is in the ISR and healthy
  -> leadership is moved back to broker-1
  -> distribution is restored to the original, balanced state

Replica Placement Should Assume Correlated Failures, Not Just Independent Ones

Part 02's replication-factor examples spread replicas across different brokers, which protects against a single broker failing independently. In a real deployment, broker failures are not always independent — an entire rack can lose power, an entire availability zone can have a networking incident, and every broker physically located there fails at once, together, regardless of how carefully replicas were spread across broker IDs.

broker.rack lets each broker declare which rack or availability zone it physically lives in, and Kafka's replica-placement algorithm uses that information to avoid putting all of a partition's replicas in the same rack whenever the cluster topology allows it. Without rack awareness configured, Kafka only guarantees replicas land on different brokers — it has no idea two of those brokers happen to share a power supply or a network switch, and a correlated failure there can take out every replica of a partition simultaneously, defeating the entire purpose of replication factor 3.

without broker.rack configured, replicas spread only across broker IDs:
  orders partition 0: replicas=[broker-1, broker-2, broker-3]
  broker-1 and broker-2 happen to both be physically in rack-A
  a rack-A power incident takes out 2 of the partition's 3 replicas at once
  -> ISR drops to just broker-3 -> min.insync.replicas=2 rejects writes
  -> much closer to full data loss than replication.factor=3 implied

with broker.rack configured (rack-A, rack-B, rack-C):
  orders partition 0: replicas=[broker-1 (rack-A), broker-4 (rack-B), broker-7 (rack-C)]
  a single rack-A incident takes out only 1 of the 3 replicas
  -> ISR drops to 2, still meets min.insync.replicas=2, writes continue
  -> the replication.factor=3 durability promise is actually intact

Replica placement is not permanently fixed at topic-creation time either. Partition reassignment (kafka-reassign-partitions.sh) lets an operator move replicas between brokers — used when decommissioning a broker, rebalancing disk usage across the cluster, or correcting a placement that turned out not to be rack-aware. A reassignment is itself a replication operation: the new target replica catches up by fetching the full partition history from the current leader before it is considered in sync, which is why large reassignments are throttled deliberately — an unthrottled reassignment can saturate broker network bandwidth and degrade normal replication traffic for every other partition sharing that network path.

Reading kafka-topics --describe Correctly, Column by Column

Every concept in this module — replicas, leader, ISR — shows up directly in the output of kafka-topics.sh --describe, which is the first thing worth checking during any replication-related incident. Reading it correctly, and knowing exactly what a discrepancy between columns means, turns this from a wall of text into a precise diagnostic tool.

kafka-topics.sh --bootstrap-server broker:9092 --describe --topic orders

Topic: orders   PartitionCount: 4   ReplicationFactor: 3
  Partition: 0  Leader: 3   Replicas: 3,1,2   Isr: 3,1,2
  Partition: 1  Leader: 1   Replicas: 1,2,3   Isr: 1,2
  Partition: 2  Leader: 2   Replicas: 2,3,1   Isr: 2,3,1
  Partition: 3  Leader: -1  Replicas: 1,2,3   Isr: 1

Partition 0 is fully healthy: Replicas and Isr list the same three brokers in the same set, meaning every assigned replica is caught up, per Part 04. Partition 1 has a real problem worth investigating: Replicas lists three brokers (1, 2, 3) but Isr lists only two (1, 2) — broker 3 has fallen behind and dropped out of the in-sync set, exactly the condition UnderReplicatedPartitions is built to catch. Partition 3 is the most serious of the three: Leader: -1 means there is currently no leader at all — the partition is fully unavailable for both reads and writes, the scenario from Part 05 where every in-sync replica became unreachable and, with unclean.leader.election.enable=false, the controller correctly refused to promote the one out-of-sync replica still listed in Isr.

The consumer-groups equivalent tool, kafka-consumer-groups.sh --describe, is unrelated to replication but is worth mentioning here because the two are commonly confused: replication health (this module) is about whether a partition's data is durably copied across brokers, while consumer group health (the previous module) is about how far behind a group's readers are. A partition can have a perfectly healthy ISR while a consumer group reading it has enormous lag, and vice versa — they are orthogonal failure modes that happen to both show up as "something is wrong with this topic" from a distance.

The Same Cluster Should Rarely Have One Replication Policy for Every Topic

Every setting covered in this module — replication factor, min.insync.replicas, unclean.leader.election.enable, acks — can be set per topic (the first three) or per producer (acks), not just as a single cluster-wide default. Treating them as a single cluster-wide policy is a common simplification that either over-pays for durability on data that doesn't need it, or under-protects data that does, because a real production cluster almost always hosts topics with genuinely different loss tolerance sitting side by side.

A useful exercise, walked through concretely here, is to take a handful of realistic topics on one shared cluster and reason about what each one's settings should actually be, rather than assuming a single answer applies everywhere.

Notice that the compacted changelog topic gets the same strict settings as payments, even though it isn't financial data in the traditional sense — the reasoning is different but the conclusion is the same. Because log compaction (covered generally for message brokers elsewhere in this track) retains only the latest value per key, losing an acknowledged write to a compacted topic doesn't just lose one historical event — it can silently corrupt the "current state" view every downstream consumer reconstructs from that topic, which is a much larger blast radius than a single lost event in an append-only, non-compacted topic.

Five Misconceptions About Replication and Durability

What This Looks Like on Day One

5 Interview Questions — With Complete Answers

Mistakes Beginners Make Constantly

Errors You Will Hit — And Exactly Why They Happen