Local Setup and Kafka CLI

You Cannot Learn Kafka Without Running It

Every concept in the previous two modules — partitions, leaders, producers, consumers, offsets — is easy to nod along to and easy to misunderstand without actually watching it happen. Reading about consumer lag is not the same as producing 50 messages, killing your consumer halfway through, restarting it, and watching it resume from exactly where it left off. This module gets a real, working Kafka broker running on your own machine in a few minutes, and then teaches the command-line tools you will use constantly, both while learning and in real production debugging.

Historically, running Kafka locally meant running two separate systems: ZooKeeper, an external coordination service, and the Kafka broker itself, which depended on it. That is no longer true. Kafka 3.x and later fully support KRaft mode, where the broker manages its own cluster metadata internally using the Raft consensus protocol, with no separate service to install, configure, or keep alive. This module uses KRaft mode exclusively — you should never need to set up ZooKeeper to learn or run Kafka today.

This module focuses entirely on the first approach, because it is the fastest path to a real, fully-functional broker with zero cloud dependency, zero cost, and a completely disposable environment you can tear down and recreate in seconds whenever you want a clean slate.

A Real, Working docker-compose.yml

The configuration below runs one Kafka broker that also acts as its own controller — a common, fully supported topology for local development and small deployments. In a real production cluster you would run multiple brokers and typically separate the controller role onto its own nodes, but for learning, one process doing both jobs is exactly what you want: simpler to reason about, and it starts in seconds.

services:
  kafka:
    image: apache/kafka:3.8.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      # This one process is both the broker and the controller
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller

      # Where controller-role traffic and broker-role traffic each listen
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER

      # Since this single node is the entire controller quorum,
      # it votes for itself: node id 1 is reachable at kafka:9093
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093

      # Fine for local learning -- never use replication factor 1 in production
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1

      # A KRaft cluster needs a cluster ID -- this is a fixed, valid
      # pre-generated one purely for local development convenience
      CLUSTER_ID: "4L6g3nShT-eMCtK--X86sw"
    volumes:
      - kafka-data:/var/lib/kafka/data

volumes:
  kafka-data:

Bring it up with a single command, and Kafka will be listening on localhost:9092 a few seconds later.

docker compose up -d

[+] Running 2/2
 ✔ Network local-setup-cli_default  Created
 ✔ Container kafka                  Started

Why a fixed CLUSTER_ID is fine here

In a real deployment, you generate a fresh, random cluster ID once per cluster usingkafka-storage.sh random-uuid, then format each broker's storage directory with that ID before it ever starts, using kafka-storage.sh format. The official Docker image used above handles this bootstrap step automatically on first startup when aCLUSTER_ID environment variable is provided, which is why the compose file above works without you running those commands by hand — that machinery still happens, just inside the container's startup script.

What each environment variable is actually doing

The split between KAFKA_LISTENERS and KAFKA_ADVERTISED_LISTENERS trips up a lot of first-time Docker Compose Kafka setups. The broker binds to 0.0.0.0:9092inside the container (that's what KAFKA_LISTENERS says), but it needs to tell connecting clients an address those clients can actually reach — from your host machine, that address is localhost:9092, which is exactly what KAFKA_ADVERTISED_LISTENERSspecifies. Get this pair wrong, and clients can often connect for the very first metadata request but then fail mysteriously on subsequent requests, because the broker handed back an advertised address the client cannot actually reach.

Confirm the Broker Is Actually Ready Before Doing Anything Else

A container reporting "running" is not the same as Kafka being ready to accept connections — the broker process needs a few seconds to initialize its log directories and start listening. Check logs first, then confirm with the CLI tools themselves, since a successful metadata request is the most reliable proof the broker is genuinely ready.

docker compose logs -f kafka

kafka  | [KafkaServer id=1] started (kafka.server.KafkaServer)

Once you see a line like that, the broker is accepting client connections. All CLI commands from here on run inside the container, using docker compose exec, because the container image ships the CLI scripts under /opt/kafka/bin/ — you do not need to install the Kafka CLI tools on your host machine at all for local learning.

What "ready" actually means under the hood

Broker startup, even for this single-node setup, goes through a specific, observable sequence: the process first formats or validates its KRaft storage directory against the configured cluster ID, then starts the controller role and establishes itself as the single-member controller quorum, then starts the broker role and registers itself with that controller, and finally opens its client-facing listener socket. Only after that last step can a producer, consumer, or CLI tool successfully connect. If you run a CLI command too early, before that final step, you will see a connection-refused or timeout error rather than a Kafka-specific error — the process for these is simply not listening on the port yet.

kafka  | [MetadataLoader] initialized (using CLUSTER_ID 4L6g3nShT-eMCtK--X86sw)
kafka  | [QuorumController] Becoming the active controller
kafka  | [BrokerServer id=1] Transitioning from STARTING to RECOVERY
kafka  | [BrokerServer id=1] Transitioning from RECOVERY to RUNNING
kafka  | [SocketServer] Started socket server acceptors on PLAINTEXT://0.0.0.0:9092
kafka  | [KafkaServer id=1] started (kafka.server.KafkaServer)

docker compose exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --list

__consumer_offsets

Seeing the internal __consumer_offsets topic listed (even with zero topics of your own created yet) confirms the broker is fully initialized — that topic is created automatically the first time it is needed, and its presence means the broker successfully handled a metadata request end to end.

Creating, Listing, and Describing Topics

kafka-topics.sh is the tool for all topic administration: creating topics, listing what exists, inspecting a topic's configuration and partition layout, and deleting topics. Every subcommand needs --bootstrap-server to know which broker to talk to.

Creating a topic

kafka-topics.sh --bootstrap-server localhost:9092 \
  --create \
  --topic orders \
  --partitions 3 \
  --replication-factor 1

Created topic orders.

Listing topics

kafka-topics.sh --bootstrap-server localhost:9092 --list

__consumer_offsets
orders

Describing a topic

--describe is the command you will run most often once a cluster has real topics on it — it shows partition count, replication factor, which broker leads each partition, and which replicas are currently in-sync.

kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe \
  --topic orders

Topic: orders  TopicId: 5f3d8a...  PartitionCount: 3  ReplicationFactor: 1  Configs: segment.bytes=1073741824
  Topic: orders  Partition: 0  Leader: 1  Replicas: 1  Isr: 1
  Topic: orders  Partition: 1  Leader: 1  Replicas: 1  Isr: 1
  Topic: orders  Partition: 2  Leader: 1  Replicas: 1  Isr: 1

On a single-broker cluster, every partition's leader and only replica is broker 1 — there is nowhere else for a replica to live. In a real multi-broker cluster this same output is exactly how you would confirm, for example, that a replication-factor-3 topic actually has 3 healthy entries in each partition's Isr list, not just 3 in its Replicas list (a replica can exist but have fallen out of the in-sync set, which this command makes immediately visible).

Changing partition count and topic configuration with --alter

--alter lets you increase a topic's partition count after creation (Kafka does not support decreasing partition count, since that would require deciding which existing partition's data to discard or merge). It is also how you change individual topic-level configs, like retention, without recreating the topic.

kafka-topics.sh --bootstrap-server localhost:9092 \
  --alter \
  --topic orders \
  --partitions 6

WARNING: If partitions are increased for a topic that has a key,
the partition logic or ordering of the messages will be affected
Adding partitions succeeded!

Deleting a topic

kafka-topics.sh --bootstrap-server localhost:9092 \
  --delete \
  --topic orders

Deletion is asynchronous — the command returns immediately, but the broker removes the underlying log segments from disk in the background. Running --list again immediately afterward may still briefly show the topic until that background cleanup finishes.

Sending and Reading Test Messages Without Writing Any Code

Before writing a single line of producer or consumer application code, you can exercise a topic entirely from the command line. This is invaluable for two things: sanity-checking that a topic and broker are behaving correctly, and reproducing production issues locally by feeding known test data through the same topic shape.

kafka-console-producer.sh

This starts an interactive prompt. Each line you type and press enter on becomes one record sent to the topic. Exit with Ctrl+D (or Ctrl+C).

kafka-console-producer.sh --bootstrap-server localhost:9092 \
  --topic orders

>order-1001 placed
>order-1002 placed
>order-1003 cancelled
>

Those three lines are now three separate records in the orders topic, distributed across its partitions (with no key specified, the producer spreads records across partitions using a round-robin-like strategy, as covered in Module 02's partitioning material).

kafka-console-consumer.sh

By default, the console consumer only shows messages produced after it starts — exactly like a fresh consumer group with no committed offset reading only new records. The flag that changes this, and the one you will reach for constantly while learning, is--from-beginning, which reads the entire retained history of the topic from the earliest available offset.

kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic orders \
  --from-beginning

order-1001 placed
order-1003 cancelled
order-1002 placed

Producing with an explicit key

To test key-based partitioning specifically, use --property parse.key=true with a key separator, then type keyTABvalue lines (or use the explicit separator property shown below for clarity in scripts).

kafka-console-producer.sh --bootstrap-server localhost:9092 \
  --topic orders \
  --property "parse.key=true" \
  --property "key.separator=:"

>customer-42:order-1001 placed
>customer-42:order-1004 placed
>customer-91:order-1002 placed
>

Every record keyed customer-42 will always land on the same partition as every other record with that exact key, for as long as the topic's partition count does not change — this is what lets a consumer reading that one partition see all of one customer's events in the exact order they were produced.

Producing from a file instead of typing interactively

For anything beyond a couple of quick test records, typing into the interactive prompt gets tedious fast, and it is not reproducible. Piping a file into the producer is the practical pattern for reproducing a bug with a known, fixed set of input records, or for quickly loading realistic test data into a local topic.

# test-orders.txt, one record per line
echo "order-2001 placed
order-2002 placed
order-2003 shipped
order-2004 cancelled" > test-orders.txt

kafka-console-producer.sh --bootstrap-server localhost:9092 \
  --topic orders < test-orders.txt

This produces all four lines as four separate records without any interactive typing, and the same input file can be replayed identically as many times as needed — invaluable when trying to reproduce a timing-sensitive bug reliably.

Inspecting raw records with formatting flags

By default the console consumer prints only the record's value. Two flags that come up constantly during debugging are --property print.key=true, which prepends each record's key, and--property print.timestamp=true, which prepends the broker-assigned or producer-assigned timestamp — both essential when you need to see not just what was sent, but which partition-determining key it carried and exactly when.

kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic orders \
  --from-beginning \
  --property print.key=true \
  --property print.timestamp=true \
  --property key.separator=" | "

CreateTime:1757600123456 | customer-42 | order-1001 placed
CreateTime:1757600123901 | customer-42 | order-1004 placed
CreateTime:1757600124210 | customer-91 | order-1002 placed

Reading as a Named Consumer Group, and Why It Changes Behavior

Running the console consumer without a group ID (as in Part 05) creates a random, throwaway group each time — nothing about its position is remembered between runs. Adding--group turns it into a real, named consumer group whose offsets are committed and persisted, exactly like an application consumer.

kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic orders \
  --group orders-cli-test \
  --from-beginning

Run that command, let it read everything, then stop it with Ctrl+C and run the exact same command again. The second run will print nothing new — the group's committed offset already covers every record that exists. This is the fastest way to physically see offset commit behavior instead of just reading about it.

Resetting a group's offset deliberately

Sometimes you want to intentionally rewind or skip a group's position — to replay a topic from scratch after fixing a bug, or to skip past a poison message a group is stuck on.kafka-consumer-groups.sh --reset-offsets does this, but only for a group with no currently active members, and only takes effect when you pass --execute — without it, the command runs in a dry-run mode that shows what it would do without actually changing anything.

# stop the consumer first -- reset-offsets refuses to run against
# a group with active members

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group orders-cli-test \
  --topic orders \
  --reset-offsets \
  --to-earliest \
  --execute

GROUP            TOPIC   PARTITION  NEW-OFFSET
orders-cli-test  orders  0          0
orders-cli-test  orders  1          0
orders-cli-test  orders  2          0

kafka-consumer-groups.sh — The Command You Will Run Most in Production

kafka-consumer-groups.sh is arguably the single most important operational CLI tool in Kafka, because consumer lag is the primary health signal for almost every streaming pipeline. It lets you list every consumer group, and, most usefully, describe one group's per-partition offset position and lag.

Listing consumer groups

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

orders-cli-test

Describing a group's offsets and lag

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe \
  --group orders-cli-test

GROUP            TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID  HOST  CLIENT-ID
orders-cli-test  orders  0          2               2               0    -            -     -
orders-cli-test  orders  1          1               1               0    -            -     -
orders-cli-test  orders  2          3               3               0    -            -     -

LAG is simply LOG-END-OFFSET minus CURRENT-OFFSET for each partition — how many records exist in the partition that this group has not yet committed past. The CONSUMER-ID, HOST, and CLIENT-ID columns show- here because the consumer that last committed these offsets is not currently running; when a live consumer is actively connected and assigned to the group, those columns populate with its identity.

GROUP            TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG   CONSUMER-ID
billing-service  orders  0          148200          151900          3700  billing-service-0-a1b2c3
billing-service  orders  1          150010          150300          290   billing-service-0-a1b2c3
billing-service  orders  2          149500          162100          12600 billing-service-1-d4e5f6

This is exactly the shape of output referenced in Module 03's discussion of per-partition lag hiding behind a healthy-looking average — partition 2 here has over 12,000 records of lag while partition 1 is nearly caught up. Always check the per-partition breakdown, not just a summed total, when diagnosing a slow consumer group.

Setting Per-Topic Configuration for Fast Local Iteration

Kafka's cluster-wide defaults are tuned for production durability and retention, which often fight against fast local iteration. A 7-day default retention means a topic you create, fill with test data, and forget about will happily keep that data around for a week, quietly consuming disk. Per-topic configuration overrides, set with kafka-configs.sh, let you tune individual topics without touching cluster-wide broker settings.

Shortening retention for a throwaway test topic

kafka-configs.sh --bootstrap-server localhost:9092 \
  --alter \
  --entity-type topics \
  --entity-name orders \
  --add-config retention.ms=600000

Completed updating config for topic orders.

That sets retention to 10 minutes (600,000 milliseconds) for just the orders topic — any data older than 10 minutes becomes eligible for deletion by the broker's background log cleaner, without affecting retention on any other topic in the cluster.

Viewing a topic's current configuration overrides

kafka-configs.sh --bootstrap-server localhost:9092 \
  --describe \
  --entity-type topics \
  --entity-name orders

Dynamic configs for topic orders are:
  retention.ms=600000 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:retention.ms=600000}

Only settings that have been explicitly overridden for this specific topic show up here — a topic with no overrides at all describes as having no dynamic configs, meaning it inherits every setting from the broker's cluster-wide defaults.

Removing an override, back to the cluster default

kafka-configs.sh --bootstrap-server localhost:9092 \
  --alter \
  --entity-type topics \
  --entity-name orders \
  --delete-config retention.ms

The Things That Trip Up a First Local Kafka Setup

Port conflicts

Port 9092 is Kafka's conventional default, and it is common for a previous Kafka container, a different local install, or another tool entirely to already be bound to it. Docker Compose will fail to start with a clear "address already in use" error in that case.

lsof -i :9092

COMMAND   PID   USER   FD   TYPE  NODE NAME
java      4821  you    62u  IPv6  TCP *:9092 (LISTEN)

Either stop that process, or change the host-side port mapping in docker-compose.yml(for example "19092:9092") and adjust your client's bootstrap.servers to match. Note that the container-internal port stays 9092 regardless — only the host-side mapping changes.

Cluster ID mismatches after editing the compose file

KRaft's metadata log is formatted with a specific cluster ID the first time the broker starts, and that ID is written into the data directory. If you change CLUSTER_ID in your compose file later but the old, already-formatted volume is still attached, the broker refuses to start, because the ID it was told to use no longer matches the ID already stamped into its storage.

docker compose down -v
# -v removes the named volume too, forcing a clean re-format
# on the next docker compose up. Fine for local learning; this
# permanently deletes all locally stored topic data.
docker compose up -d

Replication factor limits on a single broker

If you try to create a topic with --replication-factor 3 against this single-broker cluster, the command fails outright — there are not 3 brokers to place 3 replicas on.

kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic payments --partitions 3 --replication-factor 3

Error while executing topic command : Unable to replicate the partition to 3 broker(s)
InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1

This is not a bug to work around locally — it is a correct, honest error. If you want to exercise multi-broker behavior (leader election, ISR shrinking, replication) locally, you need to extend the compose file to run 3 broker services instead of 1, each with a unique KAFKA_NODE_IDand listed in a shared KAFKA_CONTROLLER_QUORUM_VOTERS. A single broker is sufficient for everything in this module and the next several, but topic-design and replication modules later in this track will walk through a multi-broker compose file.

Commands hanging with no output at all

If a CLI command appears to hang indefinitely rather than returning an error, the most common cause is a mismatch between the address the client is told to connect to and the address the broker actually advertises. If you changed the host-side port mapping indocker-compose.yml but ran a command against the old port, or if you are running the CLI from your host machine instead of inside the container without adjustingKAFKA_ADVERTISED_LISTENERS accordingly, the TCP connection can appear to hang rather than failing cleanly, because the client is waiting on a socket that either isn't listening or is advertising a hostname unreachable from where the command is being run.

# add a short explicit timeout to fail fast instead of hanging indefinitely
kafka-topics.sh --bootstrap-server localhost:9092 \
  --list \
  --command-config <(echo "request.timeout.ms=5000")

# or, simpler first check: confirm the container is actually running
docker compose ps

NAME    IMAGE                 STATUS
kafka   apache/kafka:3.8.0    Up 2 minutes (healthy)

Running out of memory with several other containers active

Kafka broker processes default to a JVM heap sized for production hardware, which can be unnecessarily large for a throwaway local single-broker setup, especially if Docker Desktop's allocated memory is already shared with several other running containers. If the container is being killed unexpectedly (visible as an OOMKilled status in docker compose ps ordocker inspect), lowering the heap explicitly keeps local resource usage predictable.

# add to the kafka service's environment block in docker-compose.yml
KAFKA_HEAP_OPTS: "-Xmx512m -Xms512m"

512MB of heap is comfortably enough for the volume of test data covered in this module — a few topics, a handful of partitions, and a modest number of test records — while leaving headroom for whatever else is running on the same machine.

Five Misconceptions About Running Kafka Locally

What This Looks Like on Day One

5 Interview Questions — With Complete Answers

Mistakes Beginners Make Constantly

Errors You Will Hit — And Exactly Why They Happen