Performance Tuning
Tuning Kafka for real throughput and latency targets: producer batching and compression trade-offs, broker page-cache and thread sizing, consumer fetch tuning, disk and filesystem choices, network settings, JVM heap sizing, and a real benchmarking methodology.
Quick answer
Kafka tuning trades latency against throughput toward a stated target: larger batches (batch.size, linger.ms) and compression (zstd, lz4) raise producer throughput; brokers rely on the OS page cache, so keep the JVM heap modest; consumers tune fetch sizes; and partition counts set the parallelism ceiling.
- Bigger batches compress better.
- Leave most RAM to the page cache; about 6 GB of heap is a common start.
- Measure against a target before and after each change.
linger.ms=20
batch.size=262144
compression.type=zstd
buffer.memory=134217728
max.in.flight.requests.per.connection=5 # safe with idempotenceEvery Kafka Tuning Knob Is Trading Latency for Throughput, or the Reverse
The monitoring module taught you how to see when something is wrong — under-replicated partitions, rising retry rate, growing consumer lag. This lesson is about what to actually change once you know a system is underperforming, and just as importantly, what target you are tuning toward in the first place. Almost every setting covered here sits on one axis: latency versus throughput. Nothing is free. A change that makes the system faster in aggregate almost always makes some individual record slower, and a change that makes an individual record faster almost always costs you aggregate throughput.
This is why performance tuning without a stated target is not really tuning — it is guessing. Before touching a single config, you need to answer: what does this pipeline actually need? A payment confirmation path that a user is waiting on cares about the 99th-percentile latency of a single record far more than it cares about aggregate megabytes per second. A batch ingestion pipeline loading a data warehouse overnight cares about aggregate throughput and essentially nothing about the latency of any individual record. These two workloads should not share the same producer configuration, the same consumer configuration, or the same mental model of "fast."
Latency-sensitive workload — an API request that blocks on a Kafka write before responding to a user, a fraud-check pipeline that must score a transaction within milliseconds. Tune for small batches, low linger.ms, cheap compression codecs, fast acks.
Throughput-sensitive workload — log aggregation, clickstream ingestion, bulk change-data-capture replication, nightly warehouse loads. Tune for large batches, higherlinger.ms, strong compression, larger fetch sizes — accepting more per-record delay in exchange for dramatically fewer requests and less CPU/network overhead per byte moved.
Every section below names which side of this trade-off a setting pushes you toward. Keep that frame in mind — the goal of this lesson is not to memorize "good" values, because there are no universally good values. The goal is to understand which lever moves which needle, so you can pick values that match your actual workload.
It is worth naming the cost of tuning carelessly up front, because it motivates the discipline the rest of this lesson asks for. A change to linger.ms or compression.typeapplied to the wrong producer can silently regress user-facing latency for weeks before anyone notices, because the symptom shows up in an unrelated API's own latency dashboard, not in any Kafka-specific metric. A JVM heap increased "to be safe" can quietly degrade an entire cluster's read performance and occasionally spike replication lag, without a single broker log line announcing the cause. Every recommendation in this lesson comes with a specific reason attached — read the reasoning, not just the recommended value, before applying it to a system you are responsible for.
batch.size does nothing if your producer is CPU-bound on compression. Raising fetch.min.bytes does nothing if your consumer's processing logic, not the fetch itself, is the slow part. Measure first, per Part 09.A quick map from symptom to likely cause
Before diving into the individual settings, it helps to have a rough map from an observed symptom to which part of this lesson is likely to contain the fix — this is not a substitute for the monitoring module's diagnostic process, but a starting point for where to look first.
| Observed symptom | Likely area to investigate | Covered in |
|---|---|---|
| High per-record producer latency, throughput is fine | linger.ms too high for this workload, or acks/compression choice adding avoidable delay | Part 02 |
| Producer CPU maxed out, throughput plateaued below expected | Compression codec choice, or batches too small to amortize per-batch compression overhead | Part 02, Part 09 |
| Broker request latency high, disk/CPU look idle | num.network.threads undersized for connection/request volume | Part 03 |
| Broker request latency high, disk/CPU genuinely busy | num.io.threads undersized, or a genuinely disk-bound broker needing more/faster disks | Part 03, Part 05 |
| Broker replication lag or GC-correlated slowdowns | JVM heap oversized, starving the OS page cache and lengthening GC pauses | Part 03, Part 07 |
| Consumer throughput plateaus with broker headroom to spare | fetch.min.bytes too low for the workload, generating excess small fetch requests | Part 04 |
| Cross-region link underperforming its advertised bandwidth | Socket buffer sizes smaller than the link's bandwidth-delay product | Part 06 |
batch.size, linger.ms, buffer.memory, and compression.type
The producer's batching mechanics were introduced in Lesson 03: records accumulate into a batch per partition, and the batch is sent when it hits batch.size bytes or linger.msmilliseconds elapse, whichever comes first. Tuning throughput on the producer side is mostly about deliberately widening that batch rather than accepting whatever falls out of default settings.
batch.size — the ceiling on how big one batch can get
batch.size (default 16 KB) is a per-partition limit, not a global one — a producer writing to 20 partitions can have 20 batches accumulating simultaneously, each up to this size. Raising it does not force bigger batches on its own; it only raises the ceiling. Under light load, batches rarely fill regardless of the ceiling because linger.ms or an idle network round trip flushes them first. Raising batch.size pays off specifically under sustained high load, where records are arriving fast enough that the ceiling, not the timer, becomes the limiting factor.
linger.ms — deliberately delaying a send to let a batch fill
Lesson 03 covered linger.ms as the throughput knob it is: a value of 0 sends immediately whenever the network is free, and a value of 10-50ms lets more records accumulate first. For this module's purposes, the operational guidance is to set linger.ms based on the workload classification from Part 01 — near 0 for latency-sensitive paths, 10-100ms for throughput-oriented bulk producers, and to always change it together with batch.size, since a higherlinger.ms with a small batch.size ceiling gains you very little — the batch fills and flushes before the linger timer would have mattered anyway.
buffer.memory — the pool all unsent batches draw from
buffer.memory (default 32 MB) is the total amount of memory the producer can use across all its in-flight, unsent batches for every partition combined. If your producer writes to many partitions with a large batch.size and a generous linger.ms, you can accumulate many partially-filled batches simultaneously, and this pool can run out before any individual batch's own limits are hit. When it does, send() blocks for up tomax.block.ms waiting for space to free up — a real, measurable latency spike that shows up as producer-side stalls with no broker-side symptom at all, which makes it easy to misdiagnose as a broker problem when the actual constraint is producer memory.
# A producer writing to a topic with 200 partitions
# batch.size = 32KB, linger.ms = 20 (throughput-tuned)
# Worst case: every partition has a nearly-full batch outstanding at once
worst_case_memory = 200 partitions * 32 KB = 6.4 MB
# Default buffer.memory (32 MB) comfortably covers this with headroom
# for retries and compression overhead. But if partition count grows
# to 2,000 (a 10x fan-out across more topics), the same math:
worst_case_memory = 2,000 * 32 KB = 64 MB
# Now buffer.memory=32MB (the default) is UNDERSIZED for the worst case
# -> send() calls start blocking under sustained peak load
# -> fix: raise buffer.memory, or reduce batch.size, or reduce partition fan-out per producercompression.type — trading CPU for network and disk
Lesson 03 introduced the four common codecs at a glance. Here is the trade-off in more operational depth, because compression is one of the highest-leverage settings for throughput-bound pipelines and one of the easiest to get wrong for latency-bound ones.
| Codec | CPU cost to compress | Typical ratio | Latency impact | Best fit |
|---|---|---|---|---|
| none | None | 1:1 | None | Payloads already compressed, or extremely low-volume topics where compression overhead is not worth it. |
| snappy | Low | Moderate | Negligible | Older default; still fine for high-throughput pipelines that need low CPU cost per byte. |
| lz4 | Low, fastest of the four | Moderate | Negligible | The common modern default — near-snappy CPU cost with slightly better ratio and faster decompression on the consumer side. |
| zstd | Moderate | High | Small, usually acceptable | Best ratio-per-CPU-cycle in most published benchmarks; a strong default when brokers have CPU headroom and network/disk cost matters. |
| gzip | High | Highest | Noticeable under sustained high volume | Only when network or storage cost dominates and CPU is genuinely cheap relative to bandwidth — rare as a first choice in 2026. |
The CPU cost is paid once, by the producer, when the batch is built — not by the broker, and not repeatedly. This is why compressing at the batch level (after records accumulate) rather than the record level matters so much: a bigger batch, built by a higher linger.ms/batch.size, compresses to a meaningfully better ratio than the same records compressed individually, because repeated JSON keys and similar field values across many records in one batch compress well together. This is the concrete mechanism behind the guidance from Lesson 03 that batching and compression settings reinforce each other and should be tuned together, not independently.
zstd is a reasonable default choice for most throughput-oriented pipelines — it typically beatslz4/snappy on compression ratio for only a modest additional CPU cost, which translates directly into less network traffic and less broker disk consumed for the same retention window (see Part 02 of the scaling module for the disk math this feeds into).When aggressive batching actively hurts a pipeline
It is worth being explicit about the failure mode of over-applying this section's advice. A team that reads "bigger batches are more efficient" and raises linger.ms and batch.sizeuniformly across every producer in their organization, including ones serving latency-sensitive request paths, trades away real, user-visible responsiveness for a throughput gain nobody asked for on that specific path. The added delay is not hidden — it shows up directly as slower API responses, because the producer's send() call is often sitting inside the critical path of a synchronous request handler waiting on a delivery confirmation.
# A checkout service producer writes a "payment authorized" event
# and the API handler waits for the delivery result before
# responding to the user (a common, reasonable pattern for an
# event that must be durably recorded before confirming success)
# Throughput-tuned settings applied here by mistake:
linger.ms = 30
batch.size = 131072
# Every checkout request now waits AT LEAST up to 30ms longer than
# necessary for its own event to even be sent, on top of normal
# network and broker processing time -- multiplied across every
# single checkout, this is a real, measurable latency regression
# that shows up in the API's own p50/p99 dashboards, not Kafka's
# Correct: this producer should use Profile A from Part 09,
# not Profile B, regardless of what a separate analytics producer
# writing to a different topic on the same cluster is doingThread Pools, the Page Cache, and Why a Huge JVM Heap Can Hurt
num.network.threads and num.io.threads
A broker splits its work into two thread pools. Network threads (num.network.threads, default 3) read requests off the wire, do minimal parsing, and hand them to a request queue — they do not touch disk. I/O threads (num.io.threads, default 8) pull requests off that queue and do the actual work: appending to the log, reading from disk, replicating to followers. If network threads are too few for the connection count and request rate a broker handles, requests queue up waiting just to be received, which shows up as elevated request latency with the disk itself barely busy. If I/O threads are too few, requests pile up waiting for actual log work, which shows up alongside real disk and CPU utilization.
| Symptom | Likely thread pool to increase | How to confirm before changing anything |
|---|---|---|
| High request latency, low disk/CPU utilization, many connections | num.network.threads | Check the network processor idle ratio metric from the monitoring module — a low idle ratio here with disk/CPU otherwise quiet points at the network pool being the bottleneck. |
| High request latency, high disk I/O or CPU utilization | num.io.threads | Check the request handler idle ratio — a low idle ratio alongside real disk activity points at the I/O pool, not the network pool. |
A common rough starting point is to size num.io.threads to roughly the number of disks (or, on SSD/NVMe hosts with fewer physical spindle constraints, a multiple of CPU core count) attached to the broker for log storage, since I/O threads are what actually perform the disk reads and writes.num.network.threads is usually left closer to its default unless a broker serves an unusually large number of concurrent client connections.
# Broker host: 4 dedicated NVMe disks for log.dirs, 16 CPU cores,
# serving roughly 3,000 concurrent producer/consumer connections
# num.io.threads starting point:
# rule of thumb -- roughly 1x to 2x the number of log disks,
# adjusted upward if the broker also does meaningful CPU work
# per request (e.g. SSL termination)
num.io.threads = 8 # 2x the 4 disks, room for concurrent disk ops
# num.network.threads starting point:
# default (3) is often fine up to a few hundred connections;
# a broker fielding thousands of connections benefits from more
num.network.threads = 6 # doubled from default given connection count
# Confirm the choice with the monitoring module's idle-ratio
# metrics after deploying -- request handler idle ratio and
# network processor idle ratio should both stay comfortably above
# zero under normal peak load; a ratio consistently near zero on
# either pool means that specific pool needs more threadsKafka relies on the OS page cache, not the JVM heap, for read performance
This is the single most counter-intuitive and most consequential broker-tuning fact, and it is the reason a naive instinct — "give the JVM a bigger heap, more memory should mean faster" — is often exactly backwards for Kafka. Kafka's storage engine writes to the filesystem directly (as covered in the retention and compaction module) and relies on the operating system's page cache, not JVM-managed memory, to serve reads efficiently. When a consumer near the head of the log reads recent data, that data is almost always still sitting in the OS page cache in RAM, so the broker serves it without touching physical disk at all — this is part of why Kafka can serve many consumers reading the same recent data with very little disk I/O.
# A broker machine with 64 GB of RAM
# BAD: JVM heap set to 32 GB ("give Kafka more memory")
# -> only 32 GB left for the OS page cache
# -> less recent log data stays cached in RAM
# -> more reads fall through to physical disk
# -> a 32 GB heap also means longer, more disruptive GC pauses
# (see the consumer session-timeout GC discussion in Lesson 03)
# GC pauses on a broker can delay replication and request handling
# GOOD: JVM heap set to 6 GB, the rest left to the OS
# -> ~58 GB available for the page cache
# -> most reads for recently-produced data are served from RAM
# -> shorter, less disruptive GC pauses on a small heap
# -> Kafka's own object footprint (mostly metadata, not message
# payloads -- payloads are handled via zero-copy sendfile,
# bypassing the JVM heap almost entirely for the read path)
# comfortably fits in a heap far smaller than the machine's total RAMChecking page cache effectiveness in practice
The claim that Kafka relies on the page cache is easy to state and easy to verify operationally. On a Linux broker host, standard OS tooling (free -h, or the cached/bufffigures it reports) shows how much RAM the kernel is currently using for the page cache versus how much is genuinely free or held by processes — on a well-tuned Kafka broker with a modest JVM heap, this figure should represent the large majority of the machine's RAM, growing to fill whatever is not claimed by the heap and other processes. A broker where the page cache figure looks unexpectedly small relative to total RAM is a strong signal that something — an oversized heap, another memory-hungry process sharing the host — is starving the cache the read path depends on.
free -h
# total used free shared buff/cache
# Mem: 64Gi 7.2Gi 1.1Gi 0.1Gi 56Gi
# ~56 GB of the 64 GB total is buff/cache -- the OS page cache and
# related buffers. Roughly what you'd expect from a broker running
# a ~6GB JVM heap on a 64GB host, with the remainder available to
# the page cache. This is a healthy-looking distribution.
# Compare against a broker running a 32GB heap on the same 64GB host:
# Mem: 64Gi 34.5Gi 1.8Gi 0.1Gi 27Gi
# Only ~27 GB left for buff/cache -- roughly half of what the
# well-tuned broker above has available for the same total RAM,
# directly illustrating Part 03's core finding in a single commandlog.segment.bytes — segment size trade-offs
log.segment.bytes (default 1 GB, from the retention and compaction module's storage model) controls how large each segment file grows before a new one is rolled. Smaller segments roll more often, which means more, smaller files to manage, more frequent index-file creation overhead, and — for compacted topics — compaction can act sooner on a recently-closed segment, reclaiming space from duplicate keys faster. Larger segments mean fewer files and less rolling overhead, but retention deletion (which operates at the segment level, per the retention module) becomes coarser-grained — an entire 1 GB+ segment must age out before any of its data is reclaimed, even if only its oldest message crossed the retention boundary.
| Segment size | Effect on compaction | Effect on retention granularity | Effect on file-handle overhead |
|---|---|---|---|
| Smaller (e.g. 256 MB) | Segments close and become compactable sooner — faster reclaim of space on high-churn compacted topics | Finer-grained deletion — less "wasted" retained data past the boundary | More open file handles and index files per partition |
| Larger (e.g. 2 GB+) | Segments stay open longer before compaction can act on them | Coarser deletion — a whole segment must fully age out | Fewer file handles, less filesystem metadata overhead |
fetch.min.bytes, fetch.max.wait.ms, and max.partition.fetch.bytes
Lesson 03 covered the poll loop's timing settings — max.poll.records,max.poll.interval.ms — which govern how much work one call to poll() hands your application and how long you have to process it. The settings in this section govern something earlier in the pipeline: how the consumer's underlying fetch requests to the broker are shaped, which is a separate lever from how the returned records are batched for your application code.
fetch.min.bytes and fetch.max.wait.ms — the same trade-off as linger.ms, mirrored on the read side
By default, a consumer's fetch request returns as soon as any data is available, even a single small record — this minimizes latency but means the broker handles many small fetch requests under light, steady load. fetch.min.bytes (default 1 byte, effectively "return immediately") tells the broker to hold the fetch request open until at least this many bytes are available to return, batching more data into each response. fetch.max.wait.ms (default 500ms) caps how long the broker will wait for that minimum before returning whatever it has anyway, so a genuinely idle topic does not leave the consumer waiting indefinitely.
fetch.min.bytes = 1 (default)
-> broker returns data the instant any is available
-> lowest possible per-record latency
-> many small fetch requests under light, steady load
fetch.min.bytes = 100000 (100 KB)
fetch.max.wait.ms = 500
-> broker waits until either 100KB has accumulated OR 500ms has
passed, whichever comes first, before responding
-> under high volume: fewer, larger fetch responses, much better
throughput per request
-> under low volume: consumer waits up to 500ms for data that
never reaches 100KB, adding up to 500ms of latency
-> this is functionally the read-side mirror of the producer's
batch.size / linger.ms trade-off from Part 02max.partition.fetch.bytes — the ceiling per partition, per fetch
max.partition.fetch.bytes (default 1 MB) caps how much data one fetch request can return from a single partition. This interacts with how many partitions one consumer is assigned: a consumer owning many partitions with this set high can receive a very large total response in one fetch, increasing both consumer-side memory pressure and the time spent deserializing before the nextpoll() — which, per Lesson 03's rebalancing discussion, risks trippingmax.poll.interval.ms if processing that large a batch takes too long. Raising this value trades higher per-fetch throughput for a larger memory footprint and a longer window between polls.
| Setting | Low value effect | High value effect |
|---|---|---|
| fetch.min.bytes | Lower latency, more frequent small fetch requests, more broker request overhead under load | Fewer, larger fetch responses; higher throughput per request; more latency on genuinely low-volume topics |
| fetch.max.wait.ms | Caps latency tightly even if fetch.min.bytes is high — data returns sooner on a quiet topic | Allows fetch.min.bytes more time to be satisfied, at the cost of higher worst-case latency on quiet topics |
| max.partition.fetch.bytes | Smaller responses, less consumer memory pressure, more fetch round trips for a high-volume partition | Larger single-fetch throughput per partition, more consumer memory used, longer per-poll processing time |
fetch.min.bytes=1 will still fetch efficiently, since the broker returns whatever is available (potentially a full large batch) immediately once available — but a latency-tuned producer sending small, frequent, uncompressed records into a consumer tuned with a highfetch.min.bytes will add real, avoidable latency, since the consumer now waits for volume that a latency-oriented producer was never going to generate quickly. Match both sides of the pipeline to the same workload classification from Part 01.A worked example: request-count reduction on the consumer side
The same request-count math Lesson 03 walked through for producer batching applies symmetrically on the fetch side, and concrete numbers again make the trade-off easier to evaluate than the settings alone.
Topic throughput: 40 MB/sec sustained, average record size 400 bytes
Consumer group: 10 consumers, each responsible for roughly 4 MB/sec
fetch.min.bytes = 1 (effectively immediate return):
each fetch returns whatever small amount has accumulated since
the last request -- under sustained load, this still batches
somewhat due to network round-trip time, but nowhere near the
broker's serving capacity per request
approx. 800 fetch requests/sec per consumer at this volume
fetch.min.bytes = 65536 (64 KB), fetch.max.wait.ms = 200:
each fetch waits for either 64KB to accumulate or 200ms to pass
approx. 65 fetch requests/sec per consumer at the same volume
Same data volume consumed. ~12x fewer fetch requests per consumer.
Broker-side request-handling overhead per byte moved drops
correspondingly -- meaningful at 10 consumers, more meaningful
still at 100.As with the producer-side version of this calculation, the added latency is bounded and predictable — at most fetch.max.wait.ms of extra wait on a quiet moment for the topic — while the reduction in request overhead compounds across every consumer in the group. This is why raisingfetch.min.bytes from its near-zero default is one of the higher-leverage, lower-risk consumer-side tuning changes for a throughput-oriented pipeline, mirroring Part 02's guidance aboutlinger.ms on the producer side.
Sequential I/O, XFS vs ext4, and Separating Logs From the OS Disk
Sequential I/O is the reason Kafka is fast at all
The retention and compaction module covered the commit log as an append-only structure. The performance implication is worth making explicit here: sequential disk writes — writing to the end of a file, never seeking backward to modify existing bytes — can approach the raw throughput limit of the storage device itself, even on spinning disks, because the disk head (or, on SSDs, the internal controller) never has to jump around. Random writes, by contrast, are dramatically slower, because every write requires relocating to a different part of the disk first. Kafka's entire storage design — append-only segments, no in-place updates, sequential reads from a consumer's offset forward — exists specifically to keep every disk operation sequential.
# It is tempting to assume "SSDs are fast at random I/O anyway, so this
# doesn't matter on modern hardware" -- this undersells two things:
# 1. Sequential I/O is still meaningfully faster than random I/O even
# on NVMe, especially for large sustained throughput, because larger
# contiguous writes reduce per-operation overhead and enable better
# write coalescing and read-ahead at the OS and controller level.
# 2. Kafka's real performance advantage over many alternative systems
# is that it was designed around sequential I/O from the start --
# this is what lets a single broker sustain very high throughput on
# commodity disks, not exotic hardware. The design assumption pays
# off regardless of which storage tier you run it on.XFS vs ext4
Both XFS and ext4 are common production choices for Kafka broker log directories. XFS is generally the more commonly recommended default in current Kafka deployment guides, largely because of better out-of-the-box performance characteristics for the large sequential file operations and large numbers of files (many segments across many partitions) that a broker's log directory involves, without requiring as much manual tuning as ext4 to reach comparable throughput. ext4 remains a perfectly workable choice, particularly on teams already standardized on it operationally, but if you are choosing a filesystem for a new Kafka deployment with no existing constraint, XFS is the safer default.
Separate disks for Kafka log data and the OS
Placing Kafka's log directories on a dedicated disk (or dedicated set of disks/volumes), separate from the disk the operating system, its logs, and swap live on, matters for two reasons. First, contention — OS-level disk activity (system logging, package updates, swap activity under memory pressure) competing with Kafka's own sequential write pattern for the same physical disk introduces exactly the kind of random-access interference that Part 05's sequential-I/O design is trying to avoid. Second, isolation — if the OS disk fills up or degrades, a shared disk risks taking the broker's log writes down with it; a dedicated log disk isolates that failure domain. On multi-disk broker hosts, Kafka also supports configuring multiple log directories (log.dirs, a comma-separated list) so partitions can be spread across several physical disks, increasing aggregate disk throughput available to one broker.
log.dirs and letting Kafka spread partitions across them, are both reasonable approaches; pick based on whether you want the filesystem or Kafka itself deciding the data placement.Mount options and a few smaller filesystem details worth checking
A handful of smaller filesystem-level decisions compound with the sequential-I/O design rather than replacing it. Mounting the log directory with the noatime option (skipping the write that would otherwise update each file's last-accessed timestamp on every read) removes a write operation that provides Kafka no benefit, since Kafka does not rely on file access-time metadata for anything — on a broker doing enormous volumes of sequential reads, this avoids a meaningful number of otherwise unnecessary metadata writes. Pre-allocating segment files at their configuredlog.segment.bytes size rather than growing them incrementally (Kafka does this by default) also avoids repeated filesystem metadata updates during the busiest part of a segment's life.
| Filesystem detail | What it avoids | Relevant setting |
|---|---|---|
| noatime mount option | An unnecessary metadata write on every single file read | Set at the OS mount level for the log.dirs volume, not a Kafka broker config. |
| Segment pre-allocation | Repeated incremental file-growth metadata updates while a segment is actively being written to | log.preallocate (broker config, off by default in some distributions — confirm the default for your Kafka version). |
| Filesystem choice (XFS vs ext4) | ext4's comparatively higher tuning burden to reach equivalent large-file, high-file-count throughput | An OS-level choice made when provisioning the broker, not a runtime Kafka setting. |
Socket Buffer Sizes for High-Throughput, High-Latency Links
Most Kafka deployments run within a single data center or availability zone, where network latency is low enough that default socket buffer sizes rarely become the bottleneck. This changes when a producer, consumer, or cross-region replication link (covered in more depth in the disaster recovery module) has to traverse a high-latency network path — cross-region traffic, or a client connecting from outside the cluster's own network. On a high-bandwidth, high-latency link, the default TCP socket buffer sizes can become the actual limiting factor on throughput, independent of how well-tuned every other Kafka setting is.
The underlying reason is the bandwidth-delay product: the amount of data that can be "in flight" on a network connection before an acknowledgment can possibly return is bandwidth multiplied by round-trip latency. If the socket's send/receive buffers are smaller than this product, the connection cannot fill the available bandwidth no matter how fast either endpoint is — it spends time waiting for acknowledgments instead of sending more data.
# Example: a producer in one AWS region writing to a Kafka cluster
# in another region, over a 1 Gbps link with 80ms round-trip latency
bandwidth_delay_product = (1,000,000,000 bits/sec / 8) * 0.080 sec
= 125,000,000 bytes/sec * 0.080 sec
= 10,000,000 bytes (~10 MB)
# Kafka's default socket.send.buffer.bytes / socket.receive.buffer.bytes
# is often 100KB (broker) or OS-default (client) -- far below 10 MB
# Result: the connection cannot use the full 1 Gbps of available
# bandwidth, regardless of batch.size, linger.ms, or compression
# tuning -- the socket buffer itself is the ceiling
# Fix: raise socket.send.buffer.bytes / socket.receive.buffer.bytes
# (and the OS-level net.core.rmem_max / net.core.wmem_max on Linux,
# since the OS must also be configured to allow buffers this large)
# to comfortably exceed the calculated bandwidth-delay productThis tuning is specific to genuinely high-latency links. On a low-latency local network, the bandwidth-delay product is small enough that default buffer sizes are almost never the constraint, and raising them has little to no effect — it is worth confirming which situation you are actually in before spending time here, per the "tune the bottleneck" guidance from Part 01.
| Network path | Typical round-trip latency | Is socket buffer tuning usually worth the effort? |
|---|---|---|
| Same availability zone | < 1ms | Rarely — bandwidth-delay product is tiny; default buffers are almost never the constraint. |
| Same region, cross-AZ | 1-5ms | Occasionally, only at very high sustained throughput — check the calculated bandwidth-delay product before spending time here. |
| Cross-region, same continent | 20-60ms | Usually yes for any meaningfully high-throughput link — this is the common case in Part 06's worked example. |
| Cross-region, intercontinental | 80-200ms+ | Almost always yes — the bandwidth-delay product at these latencies is large enough that default buffers routinely underperform, sometimes dramatically. |
G1GC and Avoiding the Instinct to Oversize the Heap
Part 03 already established the core broker-heap insight — keep it small, let the OS page cache have the rest of the machine's RAM. This section covers the garbage collector choice that makes a reasonably small heap perform well under sustained broker load.
G1GC (Garbage-First Garbage Collector) is the standard recommended collector for Kafka brokers and is the default in current JVM versions used with current Kafka releases. Its defining characteristic relevant to Kafka is that it targets a configurable maximum pause time (-XX:MaxGCPauseMillis) rather than optimizing purely for throughput at the cost of occasional long stop-the-world pauses. Since a long GC pause on a broker can delay replication heartbeats and request handling — the same mechanism covered in Lesson 03's discussion of a Java consumer's heartbeat thread freezing during a GC pause — a collector that keeps individual pauses short and predictable matters more for a broker than one that maximizes raw allocation throughput.
-Xms6g -Xmx6g
-XX:+UseG1GC
-XX:MaxGCPauseMillis=20
-XX:InitiatingHeapOccupancyPercent=35
# -Xms == -Xmx: fixes the heap size rather than letting it grow
# dynamically, avoiding resize pauses and giving predictable
# page-cache headroom on the rest of the machine
# MaxGCPauseMillis: a target, not a guarantee -- G1GC tries to keep
# individual pauses under this, trading it against throughput
# InitiatingHeapOccupancyPercent: starts a concurrent GC cycle
# earlier, reducing the chance of a larger, more disruptive
# full GC being needed later under sustained loadWhat actually lives on the broker heap
It helps to be concrete about what the broker heap holds, since "modest heap is fine" can otherwise feel like it contradicts the sheer volume of data a broker handles. Message payloads themselves are not the answer — the read path uses a zero-copy transfer mechanism (the OS moves bytes from the page cache directly to the network socket without passing through JVM-managed memory at all), and the write path similarly avoids copying record bytes into heap-managed objects unnecessarily. What the heap actually holds is bounded, comparatively small bookkeeping: request/response object overhead, replica and partition metadata, client connection state, and various in-memory indexes and caches the broker maintains for its own coordination — none of which scales with the volume of message data flowing through the broker, only with the number of partitions, connections, and in-flight requests.
| Heap size choice | Effect on GC pause length | Effect on page cache | Net effect on broker performance |
|---|---|---|---|
| Too small (e.g. 2 GB on a busy broker) | Frequent collections, though individually short | Maximizes page cache — good for reads | Can cause its own problems: frequent GC activity competing for CPU, occasional allocation failures under a metadata spike |
| Well-tested (≈6 GB on most brokers) | Infrequent, short, predictable pauses | Large page cache on any reasonably sized modern host | The balance point most production Kafka deployments converge on |
| Oversized (e.g. 32 GB "to be safe") | Rare but long, disruptive pauses | Starved — less RAM available for the page cache | Actively worse for both read latency and GC-related replication delays, per this Part's core finding |
Measuring Before and After With kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh
Every trade-off in this lesson is only worth making if you can measure whether it actually helped your workload. Kafka ships two command-line benchmarking tools specifically for this — they generate synthetic load against a real topic and report throughput and latency numbers, letting you compare a setting change against a baseline under controlled, repeatable conditions rather than guessing from production noise.
kafka-producer-perf-test.sh
kafka-producer-perf-test.sh \
--topic perf-test-topic \
--num-records 5000000 \
--record-size 512 \
--throughput -1 \
--producer-props bootstrap.servers=broker-1:9092 \
acks=all \
linger.ms=0 \
batch.size=16384 \
compression.type=none
# --throughput -1 means "as fast as possible" -- useful for finding
# the ceiling. Set a specific number (e.g. 50000) to simulate a
# realistic sustained load instead, which is usually more useful
# than a raw ceiling number for capacity-planning purposes.5000000 records sent, 812743.234567 records/sec (396.85 MB/sec),
1.42 ms avg latency, 88.00 ms max latency, 1 ms 50th, 3 ms 95th,
12 ms 99th, 45 ms 99.9th.kafka-producer-perf-test.sh \
--topic perf-test-topic \
--num-records 5000000 \
--record-size 512 \
--throughput -1 \
--producer-props bootstrap.servers=broker-1:9092 \
acks=all \
linger.ms=20 \
batch.size=131072 \
compression.type=zstd5000000 records sent, 2145890.112233 records/sec (1048.77 MB/sec),
9.85 ms avg latency, 210.00 ms max latency, 8 ms 50th, 22 ms 95th,
61 ms 99th, 140 ms 99.9th.Reading these two runs side by side is the whole point of the exercise: throughput went up roughly 2.6x, but average latency went from 1.42ms to 9.85ms and tail latency (99th percentile) went from 12ms to 61ms. This is Part 01's trade-off made concrete and numeric, on your own hardware, with your own record shape — not a rule of thumb borrowed from someone else's blog post. Whether this trade is worth it depends entirely on which workload classification from Part 01 you are tuning for.
kafka-consumer-perf-test.sh
kafka-consumer-perf-test.sh \
--topic perf-test-topic \
--messages 5000000 \
--bootstrap-server broker-1:9092 \
--consumer.config consumer-perf.properties
# consumer-perf.properties contains the settings being tested, e.g.:
# fetch.min.bytes=65536
# fetch.max.wait.ms=200
# max.partition.fetch.bytes=2097152start.time, end.time, data.consumed.in.MB, MB.sec,
data.consumed.in.nMsg, nMsg.sec
2026-09-04 10:12:01, 2026-09-04 10:12:07, 2441.40, 406.90,
5000000, 833333.33The methodology, not just the tools
A few operational details separate a trustworthy benchmark from a misleading one, beyond simply running the tool and reading the output number.
- ✓Benchmark against a dedicated topic, not a production topic — synthetic load and real traffic sharing a topic invalidates both the benchmark and the production metrics.
- ✓Change one setting at a time between runs. Changing linger.ms, compression.type, and batch.size together tells you the combined effect but not which one actually mattered.
- ✓Run each configuration multiple times and look at the spread, not a single number — a single run can be skewed by an unrelated, transient blip on the broker or network.
- ✓Match record size and record count to your real workload's shape — a benchmark using tiny records to test a pipeline that actually sends large, complex payloads will mislead you about compression ratio and batching behavior specifically.
- ✓Benchmark on hardware that matches production as closely as possible — disk type, network path, and broker sizing all affect the numbers meaningfully; a benchmark run from a laptop against a cloud cluster measures the laptop-to-cloud network path as much as it measures Kafka.
Why percentiles matter more than averages when reading benchmark output
Both perf-test tools report latency as a set of percentiles — 50th, 95th, 99th, 99.9th — rather than a single average, and reading the percentiles rather than just the average is not optional detail; it changes the conclusion you draw from a benchmark. An average can look perfectly acceptable while a meaningful fraction of requests experience latency far worse than that average suggests, and for many real systems it is specifically the tail — the 99th or 99.9th percentile — that determines whether users notice a problem, because a system serving thousands of requests per second hits its 99th percentile constantly, not rarely.
# Tuned settings result from earlier in this Part:
# avg latency: 9.85 ms
# 50th percentile: 8 ms
# 95th percentile: 22 ms
# 99th percentile: 61 ms
# 99.9th percentile: 140 ms
# If you only looked at the average (9.85ms), you might conclude
# this configuration is barely slower than the untuned baseline's
# 1.42ms average and call it an easy win with no real cost.
# The 99th percentile tells a different story: roughly 1 in 100
# requests under this configuration waits 61ms or more -- more
# than 40x the average. For a latency-sensitive path where even a
# small fraction of slow requests matters (a user-facing API,
# a real-time bidding system), this tail latency is the number
# that actually determines whether the tuning change is acceptable,
# not the average that looked comfortable at a glance.This is exactly why Part 01 frames the choice between the two tuning profiles in Part 09 as a genuine decision with real costs on both sides, not a free lunch — a throughput gain that looks good in an average-latency number can hide a tail-latency regression that matters a great deal to whichever workload is actually latency-sensitive. Always inspect at least the 99th percentile before accepting a throughput-oriented change for any pipeline where individual request latency has a real cost.
Two Complete Tuning Profiles, Side by Side
Every setting in this lesson has been presented individually, with its own trade-off. In practice, you rarely tune one setting in isolation — a real pipeline needs a coherent profile where producer, broker, and consumer settings all point the same direction on the latency-versus-throughput axis from Part 01. Presenting two complete, opposite profiles side by side makes the coherence requirement concrete, and gives you a starting point to adjust from rather than a blank page.
Profile A — latency-sensitive (a fraud-check API path a user is waiting on)
| Setting | Value | Why |
|---|---|---|
| linger.ms | 0 | Send immediately — no deliberate delay is acceptable on a path a user is blocked on. |
| batch.size | Default (16 KB) or lower | Small batches are expected and fine; the goal is not to force artificial batching that adds latency. |
| compression.type | lz4 or none | Lowest CPU cost per record; a few percent of compression ratio is not worth added per-record latency here. |
| acks | all, with min.insync.replicas=2 | A fraud decision has real business consequences if the underlying event is lost — durability still matters even on a latency-sensitive path. |
| fetch.min.bytes (downstream consumer) | 1 (default) | The downstream scoring service needs the record the instant it is available, not after a batching delay. |
Profile B — throughput-sensitive (nightly clickstream warehouse load)
| Setting | Value | Why |
|---|---|---|
| linger.ms | 20-50ms | Nobody is waiting on any individual record; maximize batch fill for fewer, larger requests. |
| batch.size | 128 KB - 256 KB | Large batches amortize per-request overhead and compress far better as a unit. |
| compression.type | zstd | Best ratio-per-CPU-cost for a workload where CPU is available and network/disk savings compound at scale. |
| acks | all, or acks=1 if some loss is acceptable for this specific analytical use case | Depends on whether this data feeds a system where occasional loss has real cost — decide deliberately per Lesson 03's producer design rule, not by copying Profile A. |
| fetch.min.bytes (downstream consumer) | 64 KB - 1 MB, with a bounded fetch.max.wait.ms | Maximize throughput per fetch request; the consumer is a batch job with no per-record latency requirement at all. |
linger.ms and compression.type values, because they are answering different questions about what "fast" means for their own workload.Profile C — a common middle ground (order-confirmation events with a downstream analytics fan-out)
Not every pipeline sits cleanly at one extreme. A topic like order-confirmation events is often read by two very different kinds of consumer at once: a notification service that wants the event with low latency, and an analytics pipeline reading the same topic that cares only about aggregate throughput. Since consumer groups are independent (Lesson 03), each consumer can be tuned separately even though they read the same topic — the notification-service consumer group keepsfetch.min.bytes low, while the analytics consumer group raises it, both reading the same underlying data with no coordination required between the two tunings.
| Component | Tuning choice | Why |
|---|---|---|
| Producer (order service) | linger.ms=5, acks=all, compression.type=lz4 | A modest linger.ms balances the fact that some downstream consumers care about latency without fully sacrificing batching efficiency; acks=all is non-negotiable for an order event. |
| Notification-service consumer group | fetch.min.bytes=1 (default) | This consumer needs the event as soon as possible to notify the customer promptly. |
| Analytics consumer group | fetch.min.bytes=131072, fetch.max.wait.ms=500 | This consumer only cares about eventual aggregate throughput and benefits from larger, less frequent fetches. |
This pattern — one producer with a moderate, shared-compromise configuration, and multiple independent consumer groups each tuned for their own use case — is common enough in practice that it is worth recognizing as a distinct third option, not just a failure to pick between Profile A and Profile B.
A short checklist before calling a tuning pass complete
- ✓Did you classify the workload as latency-sensitive or throughput-sensitive before picking any value, per Part 01?
- ✓Did you confirm, using the monitoring module's metrics, which part of the pipeline was actually the bottleneck before changing settings, per Part 01's closing guidance?
- ✓Did you check broker heap size against Part 03's page-cache guidance rather than assuming bigger is better?
- ✓Did you change producer batching/compression and consumer fetch settings together, per Part 04's callout, rather than tuning only one side of the pipeline?
- ✓Did you benchmark the change against a recorded baseline with kafka-producer-perf-test.sh / kafka-consumer-perf-test.sh, per Part 08, before rolling it to production?
- ✓Did you re-check the same monitoring metrics after rollout to confirm the benchmark's findings held under real production traffic, not just synthetic load?
Five Misconceptions About Kafka Performance Tuning
What This Looks Like on Day One
At Reddit: a team ingesting comment and vote events into Kafka for real-time ranking finds their producer is CPU-bound on a shared host during peak traffic hours. Following Part 02\'s codec table, they switch from gzip to zstd and raise linger.ms from 0 to 15ms for this specific, throughput-oriented pipeline — not for their separately tuned, latency-sensitive moderation-alert topic — and confirm the change with kafka-producer-perf-test.sh per Part 08 before rolling it to production, seeing producer CPU utilization drop meaningfully at the same message volume.
At Pinterest: a newly provisioned Kafka cluster is set up with brokers configured with a 24 GB JVM heap "to be safe," on 128 GB machines. Broker GC pause metrics show unusually long pauses correlating with occasional replication lag spikes. Following Part 03 and Part 07\'s guidance, the platform team drops the heap to 6 GB with G1GC and a tuned MaxGCPauseMillis, freeing the remaining RAM for the OS page cache — read latency for recently-produced data improves, and the correlated replication lag spikes stop appearing.
At Yelp: a review-ingestion pipeline replicating data to a Kafka cluster in a second region over a cross-region link starts hitting a throughput ceiling well below the link\'s advertised bandwidth. Using the bandwidth-delay product calculation from Part 06, an engineer confirms the default socket buffer sizes are far smaller than what the link\'s latency and bandwidth require, raises socket.send.buffer.bytes and the corresponding OS-level net.core.wmem_max, and the replication link\'s achieved throughput rises to close to its theoretical ceiling.
5 Interview Questions — With Complete Answers
The Mistakes That Undermine Kafka Performance Tuning
Symptoms You Will Hit While Tuning — And Exactly Why They Happen
🎯 Key Takeaways
- ✓Every performance setting in Kafka trades latency for throughput or the reverse — tune toward a stated workload target (latency-sensitive vs throughput-sensitive), not toward a universally "good" value.
- ✓batch.size and linger.ms are the producer's main throughput levers; compression.type compounds with them, since bigger batches compress better — zstd is usually the best ratio-per-CPU-cost default, gzip the highest-ratio but highest-CPU choice.
- ✓Kafka relies on the OS page cache, not the JVM heap, to serve reads efficiently — an oversized broker heap steals RAM from the page cache and lengthens GC pauses; ~6 GB is a well-tested starting point regardless of total machine RAM.
- ✓num.network.threads and num.io.threads are separate broker bottlenecks — network threads receive requests, I/O threads do the actual disk and replication work; diagnose which pool is constrained before resizing either.
- ✓fetch.min.bytes and fetch.max.wait.ms are the consumer-side mirror of batch.size and linger.ms — raising fetch.min.bytes trades latency for fewer, larger fetch responses under high volume.
- ✓Sequential disk I/O is why Kafka is fast; XFS is generally preferred over ext4 for Kafka log directories, and dedicated disks for Kafka logs (separate from the OS disk) avoid contention that undermines the sequential-write advantage.
- ✓Socket buffer sizes matter specifically for high-bandwidth, high-latency links (cross-region traffic) — size them against the calculated bandwidth-delay product, not default values built for local, low-latency networks.
- ✓Always benchmark with kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh against a dedicated topic, changing one setting at a time against a recorded baseline, before rolling a tuning change to production.
Quick check
What does raising linger.ms trade?
HELP US IMPROVE
How was this lesson?
Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.
Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.
