Kafka Without ZooKeeper — KRaft Mode
Learn how Kafka's KRaft metadata quorum replaces ZooKeeper, how controller and broker roles work, and what the Kafka 3.3 and 4.0 milestones mean for operators.
Chapter 01
Why Kafka Replaced ZooKeeper
Prerequisites: brokers, partitions, replication, leaders, and the controller's role in cluster metadata.
Learning objective: explain the KRaft controller quorum, start a disposable local cluster, and identify the migration boundary between ZooKeeper-based Kafka and Kafka 4.0.
Kafka stores user records in partition logs, but it also needs an authoritative view of cluster metadata: which brokers exist, which broker leads each partition, what topic configuration applies, and how the cluster should react when a broker fails. Historically Kafka delegated this coordination and metadata storage to ZooKeeper. That meant operators ran and secured two distributed systems: Kafka for event data and ZooKeeper for coordination.
KRaft is Kafka's built-in consensus-based metadata mode. Controllers keep the metadata log and elect an active controller through a quorum; brokers use that metadata to serve partitions. The metadata log is separate from application-topic data logs. KRaft reduces the dependency and coordination boundary, but it does not eliminate cluster operations, replication design, monitoring, or careful quorum sizing.
Version facts that matter
Chapter 02
The KRaft Roles and Metadata Quorum
Controllers maintain the cluster's metadata history
A KRaft controller is a Kafka process role responsible for the metadata quorum. The active controller handles metadata changes and leadership decisions; other controllers replicate the metadata log and can take over if the active controller fails. A broker serves client requests and partition replicas. In a combined deployment a process can have both roles; in a larger production environment, dedicated controller nodes isolate metadata duties from broker data traffic.
Quorum consensus requires a majority of controller voters to make progress. For three voters, two must be available; for five, three must be available. Adding voters improves fault tolerance only when they are independent and correctly configured. A network partition that leaves no majority prevents metadata changes, even if some brokers still have local data.
| Role | Primary responsibility | Typical placement |
|---|---|---|
| Broker | Serve client reads and writes; host partition replicas. | All Kafka clusters. |
| Controller | Replicate metadata log; coordinate cluster metadata and partition leadership. | Combined for development; often dedicated in larger clusters. |
| Combined node | Run broker and controller roles in one process. | Local development and appropriately sized small clusters. |
| Dedicated controller | Keep quorum work isolated from broker workload. | Production clusters with separate role configuration. |
Do not count brokers as quorum voters
Chapter 03
Start a Local KRaft Broker
Use the configuration and command syntax shipped with the Kafka distribution you install.
Kafka quickstarts are useful for a disposable local cluster, but a single combined broker/controller is not a production topology. Start from the distribution's KRaft-ready server configuration, which defines node identity, process roles, listeners, and quorum settings. Format the empty metadata log once with a generated cluster ID, then start the server.
The format command initializes storage metadata; it is not a command to run on every restart. Keep the generated cluster identity and storage directory together. Never point a production node at an existing data directory and reformat it to “fix” startup errors—first identify the cluster ID, node ID, quorum membership, and log directory state.
One replica is only for local practice
Chapter 04
Membership, Listeners, and Storage Are Explicit
KRaft configuration describes each node's identity, its roles, client/inter-broker listeners, controller listener, and how a node discovers the quorum. Kafka has supported static and dynamic controller quorum configurations across the KRaft era; the appropriate properties and bootstrap procedure depend on the Kafka release and quorum mode. Do not copy a configuration from a different release without checking its versioned documentation.
For a dedicated-controller layout, controller nodes are configured as controllers and brokers as brokers; both sides must agree on listener names and quorum membership. A controller listener should not be advertised as the public client endpoint. Bind or advertise client endpoints to addresses reachable from the intended client network, and secure them with the chosen TLS/SASL mechanism.
Before adding a node, confirm whether the cluster uses a static voter list or dynamic quorum membership, and follow the supported add/remove procedure for that mode. Treat node IDs, cluster ID, metadata directories, and quorum membership as durable operational state. A controller replacement is not just a DNS change.
Chapter 05
Migration and Upgrade Are Planned Operations
Kafka 3.3 made KRaft generally available, but a ZooKeeper-to-KRaft migration still has to follow the exact supported procedure for the source and target releases. Migration involves metadata state, controller quorum configuration, broker versions, and a carefully managed transition. Check the versioned upgrade documentation and vendor guidance before operating on real data. Do not skip intermediate releases or infer that a successful binary upgrade means metadata migration completed.
Kafka 4.0 removes ZooKeeper mode. That is a compatibility boundary: a ZooKeeper-based cluster must be migrated through a supported KRaft path before moving to 4.0. Keep backups, verify the migration plan in a representative test cluster, check client and connector compatibility, and define a rollback point before changing production metadata.
| Milestone | What it means | Operator action |
|---|---|---|
| Kafka before 3.3 | KRaft existed as an evolving mode, not yet GA. | Use release-specific support guidance; do not assume production readiness. |
| Kafka 3.3 | KRaft became generally available. | Plan a supported migration if the cluster is on ZooKeeper. |
| Kafka 4.0 | ZooKeeper mode was removed. | KRaft is required; confirm migration and client compatibility first. |
Versioned documentation beats a generic recipe
Chapter 06
Recovery, Monitoring, and Common Mistakes
- ✓Formatting an existing metadata directory as if it were a fresh node; inspect logs and cluster identity before taking action.
- ✓Running too few controller voters for the required failure tolerance; size the quorum and distribute members across failure domains.
- ✓Making every process combined by default; separate controller and broker roles when workload isolation is useful.
- ✓Exposing the controller listener to application clients; keep controller traffic on a controlled network.
- ✓Assuming KRaft removes the need for backups, upgrade tests, or broker data replication; it changes metadata management, not every reliability concern.
Observe controller health and quorum progress alongside broker metrics. Alert on controller unavailability, metadata replication lag, repeated elections, offline partitions, under-replicated partitions, and storage pressure. During an incident, gather logs and configuration before changing membership; arbitrary quorum edits can make recovery harder.
Chapter 07
Practice and Takeaways
- ✓Run the local single-node setup in a disposable environment. Create a topic and explain why replication factor 1 proves no failure tolerance.
- ✓Draw a three-controller, five-broker topology. Mark which nodes vote in the metadata quorum and what happens if one controller is unavailable.
- ✓Compare combined and dedicated roles for a small team and a high-throughput production cluster.
- ✓Write an upgrade checklist for a ZooKeeper-backed cluster targeting Kafka 4.0. Identify versioned documentation, backups, compatibility checks, and rollback criteria.
- ✓Review a sample listener configuration and identify which endpoint is for clients versus controller communication.
🎯 Key Takeaways
- ✓KRaft stores Kafka cluster metadata in Kafka’s own replicated metadata log and quorum.
- ✓Controllers participate in the metadata quorum; brokers serve data partitions.
- ✓KRaft has been GA since Kafka 3.3, and Kafka 4.0 removed ZooKeeper mode.
- ✓A local one-node cluster demonstrates startup, not production availability.
- ✓Migration and upgrades must follow the exact versioned procedure for the current cluster.
HELP US IMPROVE
How was this lesson?
Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.
Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.
