Testing and Debugging Kafka Systems

Why Testing Kafka-Dependent Code Is Harder Than Testing a Plain Function

A pure function is trivial to test: call it with inputs, assert on the output, done. Code that talks to Kafka breaks that model in three separate ways at once. It is asynchronous — a produced record is not guaranteed to be readable the instant produce() returns, so a naive test that produces then immediately asserts on consumer output is a race condition disguised as a test. It is distributed — the behavior you actually care about (partition assignment, rebalances, replication, offset commits) only exists when there is a real broker coordinating multiple parties, not inside a single process. And it needs a realistic broker to exercise correctly — mocking the wire protocol yourself reproduces bugs Kafka itself does not have, and misses ones it does.

This module treats testing as a spectrum rather than a single technique, because no single technique covers everything worth testing. At one end is fast, isolated unit testing of your own business logic with the Kafka client mocked out entirely — cheap, instant, and blind to anything Kafka-specific. At the other end is a full integration test against a real broker running in Docker — slower, but the only way to catch bugs in partitioning, serialization wire format, or consumer group rebalancing. In between sits topology testing for Kafka Streams, which gets you Kafka Streams' actual processing semantics without a broker at all, and contract testing, which catches schema-compatibility breakage before it ever reaches a shared topic.

Unit Testing Producer and Consumer Logic by Mocking the Kafka Client

The single most valuable testing habit for Kafka-dependent services is separating "code that talks to Kafka" from "code that decides what to do with the data." If your order-validation logic is buried inside a function that also calls consumer.poll() directly, you cannot test the validation logic without also standing up a broker. If you extract the validation logic into its own function that takes a plain Python dict or a deserialized object and returns a result, you can test it with zero Kafka dependency at all — and that is most of what actually needs testing on every commit.

Structuring code so the Kafka client is a thin, mockable boundary

The pattern is: a thin adapter layer owns the actual Producer/Consumerclient objects and nothing else. Everything with real logic — parsing, validating, enriching, deciding what to produce next — lives in plain functions or classes that take and return normal data structures, never a Kafka client object. The adapter layer is what you mock in unit tests; the logic layer is what you actually assert against.

# order_processor.py — pure logic, zero Kafka imports
def validate_and_price(order: dict) -> dict:
    if order.get("quantity", 0) <= 0:
        raise ValueError(f"invalid quantity: {order.get('quantity')}")
    unit_price = get_unit_price(order["sku"])          # some lookup, not Kafka
    return {**order, "total_cents": unit_price * order["quantity"]}

# consumer_service.py — thin adapter, owns the Kafka client only
from order_processor import validate_and_price

def run_consumer_loop(consumer, producer, output_topic):
    for msg in consumer:
        order = deserialize(msg.value())
        priced = validate_and_price(order)               # <- the part worth unit testing
        producer.produce(output_topic, value=serialize(priced))
        consumer.commit()

validate_and_price above needs no mock at all — it is a plain function, tested with plain dicts. The value of mocking shows up one layer up, when you want to verify thatrun_consumer_loop correctly wires deserialization, the logic call, serialization, production, and commit ordering together — without needing a real broker to do it.

from unittest.mock import MagicMock, call

def test_consumer_loop_produces_priced_order_and_commits():
    fake_message = MagicMock()
    fake_message.value.return_value = b'{"sku": "P100", "quantity": 3}'

    mock_consumer = MagicMock()
    mock_consumer.__iter__.return_value = iter([fake_message])

    mock_producer = MagicMock()

    run_consumer_loop(mock_consumer, mock_producer, "orders.priced")

    # Assert the producer was called with the right topic and a correctly priced payload
    args, kwargs = mock_producer.produce.call_args
    assert kwargs["topic"] == "orders.priced" if "topic" in kwargs else args[0] == "orders.priced"
    assert b'"total_cents"' in kwargs.get("value", args[1] if len(args) > 1 else b"")

    # Assert the offset was committed exactly once, after the produce call
    mock_consumer.commit.assert_called_once()

Notice what this test does not prove: it does not prove your consumer will actually be assigned the right partitions, that your serializer produces bytes a real Avro/JSON deserializer on the other end can read, or that your commit strategy survives a real rebalance. Those are integration-test concerns, covered in Part 03. What this test does prove, cheaply and repeatably, on every commit: the loop calls the logic function correctly, routes the result to the right topic, and commits only after producing — exactly the kind of wiring bug that is easy to introduce during a refactor and easy to miss in code review.

Integration Testing With an Ephemeral, Real Broker via Testcontainers

Mocked unit tests cannot catch bugs in partitioning behavior, actual serialization wire format, consumer group rebalancing, or anything that depends on Kafka's real semantics — because a mock, by definition, only does what you told it to do. To catch that entire category of bug, you need a real broker. The older approach to this was running an embedded Kafka broker inside the same JVM process as the test suite (EmbeddedKafka, in the Java ecosystem). It works, but it is heavyweight to set up, ties your test infrastructure to the JVM even for non-JVM services, and the embedded broker's behavior can drift from what a real, separately-deployed broker does.

The modern standard is Testcontainers — a library that spins up real Docker containers for the duration of a test run and tears them down afterward. For Kafka specifically, Testcontainers ships an official Kafka module that starts an actual Kafka broker (the real binary, running in KRaft mode) in a container, exposes its bootstrap address to your test code, and destroys the container when the test finishes. You get a genuinely real broker, isolated per test run, with no persistent infrastructure to maintain and no risk of test pollution between runs.

Why Testcontainers beat embedded-Kafka-in-JVM as the default

import json
import pytest
from testcontainers.kafka import KafkaContainer
from confluent_kafka import Producer, Consumer

@pytest.fixture(scope="module")
def kafka_bootstrap():
    with KafkaContainer("confluentinc/cp-kafka:7.6.0") as kafka:
        yield kafka.get_bootstrap_server()

def test_order_flows_through_real_broker(kafka_bootstrap):
    producer = Producer({"bootstrap.servers": kafka_bootstrap})
    consumer = Consumer({
        "bootstrap.servers": kafka_bootstrap,
        "group.id": "test-group",
        "auto.offset.reset": "earliest",
    })
    consumer.subscribe(["orders.raw"])

    producer.produce("orders.raw", key="C42", value=json.dumps({"sku": "P100", "quantity": 2}))
    producer.flush()

    msg = consumer.poll(timeout=10.0)
    assert msg is not None and msg.error() is None
    payload = json.loads(msg.value())
    assert payload["sku"] == "P100"

Because this test drives an actual broker, it exercises real partition assignment, real serialization bytes on the wire, and a real consumer group joining and receiving an assignment — none of which a mock can meaningfully simulate. The trade-off is speed: this test takes seconds, not milliseconds, largely dominated by container startup. The usual pattern is to scope the container fixture to the whole test module or session (as above, withscope="module") so the cost of starting the broker is paid once, not once per test function.

Testing Kafka Streams Topologies With TopologyTestDriver

Kafka Streams topologies — the KStream/KTable pipelines covered in the stream-processing module — present a distinct testing problem. A topology can contain joins, windowed aggregations, and stateful transformations, and you want to verify that logic is correct without paying the cost of a real broker and real wall-clock time for windows to close. TopologyTestDriversolves exactly this: it runs your actual topology definition against simulated input and output topics, entirely in-process, with full control over simulated time — no broker, no Docker, no waiting for a real window to elapse.

The key idea is that TopologyTestDriver does not fake Kafka Streams' processing logic — it runs the real topology code, the same Topology object you would deploy to production. What it fakes is the I/O boundary: instead of reading from and writing to real Kafka topics over the network, it reads from and writes to in-memory test topics you control directly, and instead of relying on real clock time for windowing, you advance a simulated clock explicitly in the test.

from confluent_kafka import TopicPartition
# (Java/Scala shown here — the canonical TopologyTestDriver API; the concept
#  is identical whatever language wraps Kafka Streams' JVM core)

StreamsBuilder builder = new StreamsBuilder();
builder.stream("orders.raw", Consumed.with(Serdes.String(), orderSerde))
    .groupByKey()
    .windowedBy(TimeWindows.of(Duration.ofMinutes(5)))
    .count()
    .toStream()
    .to("orders.count.5min", Produced.with(WindowedSerdes.timeWindowedSerdeFrom(String.class), Serdes.Long()));

Topology topology = builder.build();

try (TopologyTestDriver driver = new TopologyTestDriver(topology, props)) {
    TestInputTopic<String, Order> input = driver.createInputTopic(
        "orders.raw", new StringSerializer(), orderSerializer);
    TestOutputTopic<Windowed<String>, Long> output = driver.createOutputTopic(
        "orders.count.5min", new StringDeserializer(), new LongDeserializer());

    input.pipeInput("C42", order1, Instant.parse("2026-09-11T10:00:00Z"));
    input.pipeInput("C42", order2, Instant.parse("2026-09-11T10:02:00Z"));

    // Advance simulated time past the 5-minute window boundary explicitly —
    // no real waiting, the test controls the clock
    input.pipeInput("C42", order3, Instant.parse("2026-09-11T10:06:00Z"));

    KeyValue<Windowed<String>, Long> result = output.readKeyValue();
    assertEquals(2L, result.value);  // orders 1 and 2 fell in the same 5-minute window
}

This runs in milliseconds because nothing about it touches a network, a disk, or a real clock. You get deterministic control over exactly when a window closes, which is essential for testing windowing edge cases (a record arriving right at the boundary, a late-arriving record after grace period expiry) that would be painfully flaky to test against real wall-clock time.

Contract Testing — Validating Schema Changes Against the Registry Before Deploy

The schemas-and-serialization module covered how the Schema Registry enforces compatibility rules (backward, forward, or full) so a producer's schema change cannot silently break a consumer that has not been updated yet. Contract testing takes that enforcement and moves it earlier — from "the registry rejects a bad schema at deploy time in production" to "CI catches the same rejection on every pull request, before it ever reaches a shared environment."

The mechanism is the Schema Registry's own compatibility-check API, which you can call from CI without actually registering the new schema. You submit the candidate schema against a subject (a topic's registered schema name) and the registry evaluates it against the currently configured compatibility mode and returns whether it is compatible — the exact same check the registry would perform if you tried to register it for real, but without side effects.

#!/usr/bin/env bash
# ci/check-schema-compatibility.sh
# Fails the build if the candidate schema breaks compatibility for the subject

SUBJECT="orders.raw-value"
REGISTRY_URL="https://schema-registry.internal:8081"

curl -s -X POST \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data "{\"schema\": $(jq -Rs . < schemas/orders.raw.avsc)}" \
  "$REGISTRY_URL/compatibility/subjects/$SUBJECT/versions/latest" \
  | tee /tmp/compat-result.json

IS_COMPATIBLE=$(jq -r '.is_compatible' /tmp/compat-result.json)

if [ "$IS_COMPATIBLE" != "true" ]; then
  echo "Schema change is NOT compatible with subject $SUBJECT — failing build."
  exit 1
fi

echo "Schema change is compatible. Proceeding."

Wired into a pull-request CI pipeline, this turns a category of incident that used to surface as a production deserialization failure — a downstream consumer suddenly throwing exceptions on every message because a field it depends on was removed — into a failed CI check on the PR that introduced the change, with a clear, actionable error message, days before it would otherwise have shipped.

What contract testing does not catch

Compatibility-mode enforcement catches structural breakage — removing a required field, changing a field's type incompatibly, removing an enum value a consumer still switches on. It does not catch semantic breakage: a schema-compatible change that is nonetheless wrong, like renamingtotal_cents to total_amount_cents while keeping the same type (fully compatible from the registry's point of view if you also keep the old field as an alias) but silently changing what downstream consumers actually compute from it. Contract testing is a necessary layer, not a complete substitute for reviewing what a schema change actually means.

Diagnosing "My Consumer Isn't Receiving Any Messages"

This is the single most common Kafka support ticket, and it almost always has a small, mechanical root cause once you check things in the right order rather than guessing. Work through the checklist below in sequence — each step rules out an entire category of cause before you move to the next.

Step 1 — confirm messages are actually landing on the topic at all

Before suspecting the consumer, rule out the producer side entirely. Use the CLI consumer to check the topic directly, independent of your application's consumer code.

kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic orders.raw \
  --from-beginning \
  --max-messages 5

No output after 30+ seconds and no timeout error
-> messages genuinely are not landing on this topic; go check the producer, not the consumer

If the CLI consumer sees nothing, your consumer code was never the problem — go debug the producer (is it actually calling produce(), is it catching and swallowing an exception, is it pointed at the wrong topic name or the wrong cluster entirely). If the CLI consumer does see messages, the problem is specifically in your application's consumer, and you move to step 2.

Step 2 — check group.id and subscription

A consumer group's committed offset is scoped to the exact group.id string. A typo, an environment-variable that resolved differently than expected, or an accidental copy-paste from another service are extremely common causes — the consumer connects fine, joins a group fine, and simply never receives anything because it subscribed under agroup.id nobody is producing an assignment worth reading for, or because the topic name it subscribed to does not match the topic name being produced to (a trailing environment suffix like -staging is a classic culprit).

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group order-processing-group

GROUP                  TOPIC         PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
order-processing-group orders.raw    0          182391          182391          0
order-processing-group orders.raw    1          177204          177204          0

If this command shows no rows for your group at all, the group has never committed anything for that topic — either it has never successfully subscribed, or it is a brand-new group with nothing consumed yet. If it shows rows with a healthy assignment and zero lag, the consumer group mechanics are fine and the bug is almost certainly in what your application does with the records after receiving them (silently swallowing an exception before it logs anything, for example).

Step 3 — check partition assignment specifically

If --describe shows the group exists but a specific partition is missing from the assignment, or the group has more members than the topic has partitions, some consumers in the group are sitting idle by design — Kafka never assigns more than one consumer per partition within a group, so a group with 8 consumers on a 4-partition topic always has 4 idle members. If this is unexpected, it usually means the topic's partition count is lower than assumed, or a recent repartitioning was not communicated to whoever is running the consumer fleet.

Diagnosing "producer.send() Is Hanging"

A producer that appears to freeze on send() (or on flush(), or on the future/callback never resolving) almost always traces to one of three causes, and they are diagnosable in order of likelihood rather than by guesswork.

Cause 1 — metadata fetch is failing

Before a producer can send its first record to a topic, it must fetch metadata (which brokers lead which partitions) for that topic, as covered in the producers-and-brokers module. If the producer cannot reach any broker in bootstrap.servers at all, or the topic does not exist and auto-creation is disabled, the metadata fetch itself blocks — and depending on client configuration, that block can look identical to send() hanging, because the send is queued behind a metadata fetch that never completes.

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders.raw

Error while executing topic command: Topic 'orders.raw' does not exist as expected
[2026-09-11 14:02:11] ERROR org.apache.kafka.common.errors.UnknownTopicOrPartitionException

If the topic genuinely does not exist and your cluster has auto.create.topics.enableset to false (the correct production setting), the producer will sit retrying the metadata fetch for up to max.block.ms before finally raising a timeout — which, ifmax.block.ms is left at a high default, looks a great deal like an indefinite hang rather than a clear, fast error.

Cause 2 — max.block.ms is set too high, masking a real failure as a hang

max.block.ms bounds how long a call like send() will block waiting for metadata to become available or for buffer space to free up, before finally raising an exception. A high or default value (60 seconds in many clients) is often mistaken for "the producer is broken" when it is actually working as configured — it just has not yet reached the point where it gives up and tells you why. Lowering it in a debugging session (never permanently, without understanding the trade-off) turns a mysterious multi-minute hang into a fast, clear exception that names the actual underlying problem.

# Debugging config, not a production default:
producer = Producer({
    "bootstrap.servers": "broker:9092",
    "max.block.ms": 5000,   # fail fast in 5s instead of blocking up to 60s
})

# Now instead of a mysterious hang, you get within 5 seconds:
# KafkaException: Failed to update metadata after 5000 ms
# -> immediately actionable: go check broker reachability or topic existence

Cause 3 — network or broker reachability is the root cause underneath both of the above

Both of the causes above are usually symptoms of a simpler underlying problem: the producer genuinely cannot reach any broker, whether from a firewall rule, a DNS resolution failure inside a container network, a broker that is up but not accepting new connections because it is overloaded, or TLS/SASL credentials that are silently rejected rather than cleanly erroring. Confirming basic reachability first, before chasing client-config theories, saves the most time.

nc -zv broker-1.internal 9092
# Connection to broker-1.internal 9092 port [tcp/*] succeeded!
#   -> network path is fine; the problem is almost certainly auth, topic config, or client settings

nc -zv broker-1.internal 9092
# nc: connect to broker-1.internal port 9092 (tcp) failed: Connection timed out
#   -> network/firewall issue; stop debugging client config until this is fixed

Diagnosing "Duplicate Records Downstream"

Duplicate records reaching a downstream system are almost always explained by one of the exact mechanisms covered in the delivery-semantics module, and the fix depends entirely on identifying which one is actually at fault — treating every duplicate-records ticket the same way (usually "just add idempotence somewhere") wastes time chasing the wrong layer.

Check 1 — is producer-side idempotence actually enabled?

The first, cheapest check: confirm enable.idempotence=true is actually set on the producer that writes to the topic in question. Without it, a lost acknowledgement followed by a safe client-side retry writes the same logical record twice at the broker level, before a consumer is even involved. This is the easiest duplicate source to rule out and the easiest to fix — a one-line config change with no downstream code impact.

Check 2 — consumer commit timing relative to processing

If idempotence is already enabled and duplicates persist, the far more common real cause is consumer-side: the consumer processes a record (writes to a database, calls a downstream API, sends a notification) and then crashes or gets rebalanced away before committing the offset for that record. On restart or reassignment, the record is re-delivered from the last committed offset — and reprocessed, producing a genuine duplicate side effect, even though Kafka itself delivered nothing incorrectly.

t=0    poll() returns order event, offset 4021
t=1    consumer writes the order to the database        <- side effect happens here
t=2    consumer process crashes before commit() runs     <- offset 4021 was never committed
t=restart  consumer resumes from last committed offset, which is still 4020
t=restart  offset 4021 (the same order) is delivered and processed again
           -> the database write from t=1 happens a second time

This is not a bug in Kafka — at-least-once delivery, which is what a consumer gets by committing after processing, guarantees a record is never silently lost, at the cost of allowing exactly this kind of re-delivery after a crash between processing and committing. The fix, as the delivery-semantics module covers, is making the downstream side effect itself idempotent — a database write keyed on a unique, stable event ID with an upsert rather than an insert, so reprocessing the same record twice produces the same end state rather than two rows.

Low-Level Inspection — What Is Actually on Disk

Sometimes the client-level view (offsets, lag, consumer group state) is not enough, and you need to look at what is physically written to a partition's log segment on disk — the raw bytes Kafka's storage layer actually persisted, including headers, timestamps, compression, and batch metadata that never surface through a normal consumer API.

kafka-console-consumer.sh — the fast, everyday tool

For the vast majority of debugging, kafka-console-consumer.sh is the right tool: it reads real records through the normal consumer client and prints their values, which is enough to confirm data is landing, check its shape, or spot an obviously malformed payload.

kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic orders.raw \
  --partition 2 \
  --offset 981200 \
  --max-messages 3 \
  --property print.key=true \
  --property print.partition=true \
  --property print.offset=true \
  --property print.timestamp=true

kafka-dump-log.sh — the low-level tool for what the client API cannot show you

When the question is not "what is the record's value" but "what is actually in the segment file on disk" — is a batch corrupted, what compression codec was actually used, is a tombstone record present, what is the exact byte offset of a given logical offset — kafka-dump-log.shreads the raw .log segment file directly and prints its internal structure, below the level any consumer client ever exposes.

kafka-dump-log.sh \
  --files /data/kafka/orders.raw-2/00000000000000981200.log \
  --print-data-log \
  --deep-iteration

baseOffset: 981200 lastOffset: 981204 count: 5 baseSequence: 1180 lastSequence: 1184
  producerId: 5001 producerEpoch: 0 partitionLeaderEpoch: 12 isTransactional: false
  isControl: false position: 0 CreateTime: 1757606400123 size: 842 magic: 2
  compresscodec: LZ4 crc: 3821994102 isvalid: true
| offset: 981200 CreateTime: 1757606400123 keySize: 4 valueSize: 156
  key: C4291 payload: {"sku":"P100","quantity":2,"total_cents":3980}

The producerId and baseSequence fields visible here are exactly the idempotence bookkeeping described in the broker-internals module — seeing them directly on disk is the definitive way to confirm whether a specific batch was written under an idempotent producer session, which is otherwise invisible from a normal consumer's point of view.

Building a Local Reproduction Environment for a Production Bug

A production incident report rarely arrives with enough detail to fix blind — "consumers are lagging" or "some orders are missing fields" is a starting point, not a diagnosis. The fastest path to a real fix is almost always reproducing the exact failure locally, against real Kafka behavior, rather than reasoning about it purely from logs and metrics.

The local-setup-cli module's docker-compose pattern is the right foundation for this — a single-broker KRaft-mode Kafka container you can start in seconds — extended with seeded test data that reproduces the specific shape of the production problem.

# docker-compose.yml (from the local-setup-cli module, unchanged)
services:
  kafka:
    image: apache/kafka:3.7.0
    ports: ["9092:9092"]
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092

# seed_repro.py — reproduces the exact production shape of the bug
# (e.g. a malformed record that the on-call engineer pulled from the DLQ)
import json
from confluent_kafka import Producer

producer = Producer({"bootstrap.servers": "localhost:9092"})

# The actual malformed payload captured from the production DLQ topic,
# pasted in verbatim rather than guessed at — this is the key step
malformed_from_prod = {
    "sku": "P100",
    "quantity": None,     # <- production incident: null quantity crashed the consumer
}
producer.produce("orders.raw", key="C4291", value=json.dumps(malformed_from_prod))
producer.flush()
print("Seeded the exact production failure case locally.")

The critical discipline here is using the actual payload pulled from the production DLQ (Part 08 of the message-brokers-and-queues module covers how DLQ events preserve the original record for exactly this purpose), not a guessed-at approximation of what might have caused the failure. A guessed reproduction can pass locally while the real bug remains unfixed; the actual captured payload either reproduces the failure immediately or proves your working theory about the cause is wrong before you spend time on a fix for the wrong problem.

Once the failure reproduces locally against a real broker, you have a fast, disposable feedback loop: run the consumer against the local cluster, watch it fail the same way, fix the code, re-run, confirm it now handles the case correctly — all in seconds per iteration, entirely disconnected from production, and with the fix validated against the exact case that caused the incident before it ever ships.

Wiring All Four Testing Layers Into a Single CI Pipeline

Each testing technique covered above has a different cost, and a well-designed CI pipeline runs them in increasing order of cost, failing fast on the cheap layers before ever paying for the expensive ones. A pull request that breaks basic business logic should fail in seconds from the mocked unit tests in Part 02, not minutes later after a Testcontainers integration test has already spent time pulling and starting a broker container.

stages:
  - name: unit-tests
    run: pytest tests/unit/ -x          # mocked client, Part 02 — seconds
    fail_fast: true

  - name: schema-compatibility
    run: ./ci/check-schema-compatibility.sh   # Part 05 — a few seconds, HTTP only
    fail_fast: true
    only_if: schema files changed in this PR

  - name: streams-topology-tests
    run: ./gradlew test --tests "*TopologyTest"   # Part 04 — no broker, milliseconds
    fail_fast: true

  - name: integration-tests
    run: pytest tests/integration/ --testcontainers  # Part 03 — real broker, tens of seconds
    fail_fast: false   # run all of them even if one fails, for full visibility
    only_if: earlier stages passed

  - name: merge-gate
    requires: [unit-tests, schema-compatibility, streams-topology-tests, integration-tests]

The only_if gating on the schema-compatibility stage is worth calling out specifically: running it unconditionally on every PR regardless of whether a schema file changed wastes a small amount of CI time on every single PR in the repository, while gating it on "did this diff touch a schema file" keeps the fast-path fast for the vast majority of PRs that never touch a schema at all, without ever skipping the check on the PRs that actually need it.

What belongs in CI versus what stays a manual pre-release check

Not everything from this module belongs gated on every PR. Part 10's local-reproduction technique is explicitly a manual, incident-driven activity — you would not wire a specific captured production bug's reproduction script into the standard PR pipeline forever, though promoting a fixed bug's reproduction into the permanent Testcontainers suite, as Part 10's closing callout recommends, is exactly how it earns a permanent home in the automated pipeline going forward, rather than living as a one-off script nobody remembers to re-run.

Load Testing and Injecting Failure Deliberately, Before Production Does It For You

Everything covered so far verifies correctness under normal operation. A separate, complementary discipline is deliberately testing behavior under the conditions that actually cause Kafka incidents in production — sustained high load, a broker disappearing mid-write, or a consumer group rebalancing under pressure. None of the earlier techniques exercise these conditions by default, because a mocked test, a Testcontainers test running a handful of records, and aTopologyTestDriver run all execute far below the throughput and failure conditions that reveal these bugs.

Load testing with a real producer/consumer pair against Testcontainers

The same Testcontainers-backed broker from Part 03 can be driven at meaningfully higher throughput in a dedicated load test, separate from the fast correctness-focused integration suite that runs on every PR. The goal here is different: not "does this produce the right output" but "does throughput hold up, does consumer lag stay bounded, does the producer's buffer stay comfortably belowbuffer.memory" under a sustained, realistic load profile.

import time
from confluent_kafka import Producer

def test_producer_sustains_target_throughput_without_buffer_pressure(kafka_bootstrap):
    producer = Producer({
        "bootstrap.servers": kafka_bootstrap,
        "linger.ms": 10,
        "batch.size": 65536,
        "buffer.memory": 33554432,
    })

    target_rate = 20_000  # records/sec, a realistic peak for this service
    duration_sec = 10
    sent = 0
    start = time.monotonic()

    while time.monotonic() - start < duration_sec:
        producer.produce("load.test.topic", value=b"x" * 200)
        sent += 1
        if sent % 1000 == 0:
            producer.poll(0)   # let delivery callbacks fire, avoid local queue buildup

    producer.flush(timeout=10)
    actual_rate = sent / duration_sec
    assert actual_rate >= target_rate * 0.95, f"only sustained {actual_rate:.0f}/sec"

A load test like this run in a dedicated, less-frequent CI stage (nightly, or on-demand before a major release) rather than on every PR — per Part 11's cost-ordering principle, it is meaningfully more expensive than the correctness-focused suite and answers a different question, so gating every PR on it would slow down the common case for a check that only matters occasionally.

Chaos scenarios worth deliberately exercising

Testcontainers' ability to stop, pause, or restart a running container mid-test makes it possible to simulate the exact broker-failure scenarios the durability and ordering modules described conceptually, and actually observe your own client code's behavior under them rather than trusting it will behave as documented.

Keeping Test Data Realistic and Test Suites Maintainable Over Time

A testing strategy that is technically sound but relies on hand-typed, oversimplified test payloads tends to degrade over time — real production events accumulate fields, edge cases, and quirks that a test suite's inline JSON literals quietly stop reflecting. Two practices keep test data close enough to reality that the tests above keep catching real bugs rather than just passing forever.

Generating test fixtures from the actual registered schema

Rather than hand-writing example payloads that drift from the real schema over time, generate them directly from whatever the Schema Registry currently has registered for a subject. This guarantees test fixtures are always structurally valid against the schema contract from Part 05, and a schema change that adds a new required field is immediately reflected in newly generated fixtures rather than silently leaving old, now-incomplete hand-written fixtures in the suite.

from confluent_kafka.schema_registry import SchemaRegistryClient
import fastavro
import random

def generate_fixture(subject: str, registry_url: str) -> dict:
    client = SchemaRegistryClient({"url": registry_url})
    schema = client.get_latest_version(subject).schema.schema_str
    parsed = fastavro.schema.parse_schema(json.loads(schema))

    # Populate every field with a type-appropriate random/realistic value
    # rather than hand-typing each one -- stays correct as the schema evolves
    return {
        field["name"]: _realistic_value_for(field["type"])
        for field in parsed["fields"]
    }

Keeping a small library of real, sanitized production payloads for regression tests

Generated fixtures are good for broad structural coverage, but they rarely reproduce the specific, messy edge cases that caused real incidents — the payload from Part 10's local-reproduction pattern, for example. Maintaining a small, deliberately curated set of real (personally-identifiable information stripped or replaced with synthetic equivalents) production payloads that previously caused bugs, checked into the repository alongside the test suite, turns each past incident into a permanent, automatically-checked regression test rather than a one-time fire drill.

Five Misconceptions About Testing Kafka Systems

What This Looks Like on Day One

5 Interview Questions — With Complete Answers

The Mistakes That Make Kafka Testing Ineffective or Misleading

Errors You Will Hit While Testing and Debugging — And Exactly Why