Monitoring and Observability

Kafka's Health Is a Distributed-Systems Question, Not a Host Question

CPU usage, memory usage, and disk usage are the metrics most engineers reach for first when deciding whether a server is healthy, and they are necessary for Kafka too. They are also dangerously insufficient on their own. A broker can show comfortable CPU headroom, plenty of free memory, and disk usage well under any alarm threshold, while the cluster is actively losing its durability guarantees, a consumer group is silently falling further behind every second, or the cluster's single controller has gone missing — none of which shows up as a spike on a generic host dashboard.

This is because Kafka's correctness depends on distributed-systems-specific state that has no analog on a single machine: how many replicas of a partition are actually caught up right now, how many messages a consumer group has accumulated but not yet processed, whether exactly one broker in the cluster currently believes it is the controller. A host-level dashboard has no concept of any of these things — they live inside Kafka's own internal bookkeeping, and Kafka exposes them through its own metrics, not through anything the operating system reports.

Under-Replicated Partitions: the Single Most Important Cluster-Health Signal

A partition is under-replicated when the number of replicas currently in its in-sync replica (ISR) set is smaller than its configured replication factor. If a topic has replication factor 3 and one follower has fallen behind or gone offline, that partition now has only 2 replicas in its ISR — it is under-replicated, even though the leader is still healthy and still serving reads and writes normally. Nothing about this is visible from the outside; clients keep working exactly as before.

This metric matters more than almost any other because it is a direct, real-time measurement of how much durability margin the cluster currently has. A partition with replication factor 3 that is fully in sync can lose any single broker without losing data. The same partition, already under-replicated by one, is now one more broker failure away from data loss or unavailability — the exact safety margin that acks=all and min.insync.replicas depend on has quietly shrunk, with no other symptom yet visible.

kafka-topics.sh --bootstrap-server broker1:9092 --describe --under-replicated-partitions

# A healthy cluster returns nothing here — empty output is the
# expected, correct state. Any line returned is a partition that
# currently has fewer in-sync replicas than its replication factor.

Topic: freshcart.orders  Partition: 2  Leader: 3  Replicas: 1,2,3  Isr: 1,3
  (broker 2 is missing from the ISR — this partition is under-replicated)

The JMX metric behind this iskafka.server:type=ReplicaManager,name=UnderReplicatedPartitions, exposed as a simple integer count per broker. A cluster-wide dashboard typically sums this across all brokers, because a single under-replicated partition anywhere in the cluster is worth investigating regardless of which broker reports it.

What causes a partition to become under-replicated

• ✓A follower broker crashes, is restarted for maintenance, or is network-partitioned from the leader — the most common and usually self-resolving cause.
• ✓A follower is alive and reachable but cannot keep up with the leader's write rate — often a sign of disk I/O saturation or an undersized broker relative to actual throughput.
• ✓A large, sudden burst of writes to a partition outpaces what followers can fetch and replicate in the configured replica.lag.time.max.ms window, temporarily dropping them out of the ISR.

Handler Idle Ratio: How Close a Broker Is to Saturation

Each broker runs a fixed-size pool of network threads (which read requests off the wire and write responses back) and a fixed-size pool of I/O request handler threads (which actually process those requests — appending to the log, serving a fetch, and so on). The idle ratio for each pool is the fraction of time, over a measurement window, that threads in that pool were sitting idle rather than actively working.

A ratio near 1.0 means the pool is mostly idle — plenty of spare capacity. A ratio approaching 0 means threads are almost always busy, which means new requests are starting to queue rather than being handled immediately, and client-observed latency is about to climb even though nothing has crashed. This metric tends to give earlier warning of an overloaded broker than raw CPU usage does, because a broker can be CPU-comfortable while its fixed-size handler thread pools are still the actual bottleneck.

Both ratios trending steadily downward over days or weeks, rather than only spiking briefly during a known traffic peak, is the pattern worth treating as a capacity-planning signal — it usually means organic growth in producer or consumer traffic has outpaced the broker's current sizing, and the fix is either scaling the cluster out (more brokers, more partitions) or tuningnum.io.threads and num.network.threads if the bottleneck is genuinely thread-pool sizing rather than raw broker capacity.

Active Controller Count: Should Always Be Exactly 1

As covered in the producers-and-brokers module, every Kafka cluster has exactly one active controller broker responsible for cluster-wide coordination — electing new partition leaders, propagating metadata changes, tracking ISR membership. The metrickafka.controller:type=KafkaController,name=ActiveControllerCount reports 1 on the broker that currently holds the controller role, and 0 on every other broker. Summed across the cluster, the total should always equal exactly 1.

broker-1: kafka.controller:...ActiveControllerCount = 0
broker-2: kafka.controller:...ActiveControllerCount = 1   <- current controller
broker-3: kafka.controller:...ActiveControllerCount = 0

sum across cluster = 1   <- healthy, expected state

A cluster-wide sum of 0 means no broker currently believes it is the controller — cluster coordination is stalled, and no leadership changes can be processed until a new controller is elected. A sum greater than 1, briefly, can happen during the short window of a controller failover as the old controller is stepping down and a new one is taking over; a sum greater than 1 that persists is a serious split-brain-adjacent condition where two brokers disagree about cluster leadership, which risks inconsistent metadata propagation across the cluster.

ISR Shrink/Expand Rate and Log Flush Latency

Under-replicated partitions (Part 02) is a point-in-time snapshot — it tells you the current state. ISR shrink and expand rates tell you how often that state is changing, which is a different and complementary signal. A partition can flicker in and out of full replication repeatedly without ever staying under-replicated long enough to be caught by an instantaneous check, and that flicker — called ISR churn — is itself a symptom worth catching.

kafka.server:type=ReplicaManager,name=IsrShrinksPerSec
kafka.server:type=ReplicaManager,name=IsrExpandsPerSec

# A healthy broker shows both near zero, most of the time.
# A broker with frequent shrink-then-expand cycles on the same
# partition indicates a follower that is intermittently falling
# behind and catching back up -- often a sign of network jitter,
# an overloaded follower disk, or replica.lag.time.max.ms being
# tuned too aggressively tight for the cluster's real network
# characteristics.

Log flush latency is a separate, disk-focused signal:kafka.log:type=LogFlushStats,name=LogFlushRateAndTimeMs measures how long it takes the broker to flush log segments to disk. Kafka relies heavily on the operating system's page cache rather than flushing synchronously on every write, so this metric is normally very low — a rising trend is a direct signal of disk I/O pressure on the broker, often before that pressure shows up clearly in generic disk-utilization dashboards, because it specifically isolates the write path Kafka itself depends on rather than all disk activity on the host.

Producer-Side Metrics: Is the Write Path Actually Healthy?

Broker metrics describe the server's internal state. Producer metrics, exposed by the client library itself, describe whether writes are actually succeeding from the application's point of view — a distinction that matters because a broker can look perfectly healthy while a specific producer is failing every send due to a client-side misconfiguration, an authorization problem, or a network path issue specific to that one client.

record-error-rate:  0.02 records/sec   (still low)
record-retry-rate:  340 records/sec    (climbing steadily over the last hour)
request-latency-avg: 480 ms            (was 40ms an hour ago)

# Interpretation: writes are still mostly succeeding (low error
# rate), but it is taking meaningfully more retries and more time
# per request to get there. This pattern -- rising retries and
# latency with error rate still low -- is a leading indicator, not
# a trailing one. Waiting for record-error-rate to rise before
# investigating means investigating after clients have already
# started actually failing, not before.

Consumer-Side Metrics: Consumer Lag Is the Headline Signal

Consumer lag — the difference between the latest offset written to a partition and the offset a consumer group has committed for that partition — is the single most important consumer-health metric, for the same reason under-replicated partitions is the single most important broker metric: it is a direct, quantitative measurement of exactly how far behind reality the consumer currently is, and it is the metric that most directly maps to a business-visible symptom ("why is this dashboard showing stale data?", "why hasn't this notification been sent yet?").

kafka-consumer-groups.sh --bootstrap-server broker1:9092 \
  --describe --group fraud-detection-consumer-group

GROUP                          TOPIC              PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
fraud-detection-consumer-group freshcart.orders   0          1,204,552       1,204,560       8
fraud-detection-consumer-group freshcart.orders   1          998,201         1,041,880       43,679
fraud-detection-consumer-group freshcart.orders   2          1,190,004       1,190,020       16

Notice that partition 1 in this output has dramatically higher lag than partitions 0 and 2 — an aggregate or average across the group would show a moderate-looking total lag, while the real story is that one partition is badly behind and the others are essentially caught up. Checking per- partition lag rather than only the group-level total is what catches this — the same averaging trap that shows up whenever data or work is unevenly distributed across partitions.

A useful framing: lag by itself, at a single point in time, tells you how far behind a consumer currently is. Lag combined with records-consumed-rate and the producer's write rate tells you whether that gap is closing, holding steady, or growing — and only that combination actually tells you whether the situation is self-resolving or needs intervention.

How These Metrics Actually Get From Kafka to a Dashboard

Every metric covered in Parts 02 through 07 is exposed by Kafka's JVM process through JMX (Java Management Extensions) — Kafka does not natively speak Prometheus's metrics format, and it does not ship its own dashboard. The standard, widely adopted toolchain bridges this gap with three pieces working together, each doing one job.

# kafka-server-start.sh (or the systemd unit's ExecStart) launches
# the broker JVM with the exporter agent attached:

KAFKA_OPTS="-javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent.jar=7071:/opt/jmx_exporter/kafka-broker.yml"

kafka-server-start.sh /etc/kafka/server.properties

# kafka-broker.yml (JMX Exporter config)
rules:
  - pattern: "kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value"
    name: kafka_server_replicamanager_underreplicatedpartitions
  - pattern: "kafka.controller<type=KafkaController, name=ActiveControllerCount><>Value"
    name: kafka_controller_activecontrollercount
  - pattern: "kafka.server<type=KafkaRequestHandlerPool, name=RequestHandlerAvgIdlePercent><>OneMinuteRate"
    name: kafka_server_requesthandlerpool_requesthandleravgidlepercent

scrape_configs:
  - job_name: 'kafka-brokers'
    static_configs:
      - targets:
          - 'broker1.internal:7071'
          - 'broker2.internal:7071'
          - 'broker3.internal:7071'
    scrape_interval: 15s

Consumer lag specifically is often collected through a slightly different path in addition to raw JMX — a dedicated exporter such as Kafka's own lag-exporting tooling, or a sidecar that periodically runs the equivalent of kafka-consumer-groups.sh --describe against every group and exposes the result as Prometheus metrics — because consumer lag depends on the relationship between two offsets (log-end and committed) that is most reliably computed from the broker's own consumer-group-offset bookkeeping rather than from any single client's local JMX metrics, which only reflect that one client instance's view.

Alerting: Thresholds That Reflect What Actually Matters

A metric without a sensible alert threshold is a dashboard nobody looks at until something is already broken. But a badly chosen threshold is worse than no alert at all — it either fires constantly on noise until the team learns to ignore it, or it stays silent through a real, developing incident because the condition was defined too loosely to catch the actual failure pattern.

# Pseudocode for the alerting rule's intent:
# lag_now > 100,000 records
# AND lag_15_minutes_ago > 0
# AND lag_now > lag_15_minutes_ago * 1.2   (lag grew by at least 20% over 15 min)
#
# This deliberately ignores a lag that spiked once and is already
# recovering -- it only fires when the trend over a real window
# confirms the consumer is structurally behind, not just momentarily
# busy.

Worked Incident: Diagnosing a Consumer Lag Spike, Step by Step

An alert fires: fraud-detection-consumer-group's lag on freshcart.ordershas grown past the sustained-slope threshold from Part 09. Here is the order of checks that actually isolates the root cause, rather than guessing.

Step 1 — confirm it is real and see the shape of it, per-partition

The first check is kafka-consumer-groups.sh --describe for the group, exactly as in Part 07, broken down by partition. This immediately answers a critical branching question: is lag growing evenly across every partition the group owns, or is it concentrated in one or two partitions while the rest sit near zero?

GROUP                          TOPIC              PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
fraud-detection-consumer-group freshcart.orders   0          1,204,552       1,204,560       8
fraud-detection-consumer-group freshcart.orders   1          822,004       1,041,880       219,876
fraud-detection-consumer-group freshcart.orders   2          1,190,004       1,190,020       16

This shape — one partition badly lagging while its siblings are essentially caught up — points strongly toward either a hot key concentrating unusual volume on that specific partition, or a single consumer instance (the one assigned to that partition) being individually unhealthy, rather than a cluster-wide or group-wide capacity problem.

Step 2 — check whether the consumer group is rebalancing repeatedly

Next, the rebalance-rate metric from Part 07. Frequent rebalances would explain lag growth on their own — every rebalance pauses processing for the partitions being reassigned — and would redirect the investigation entirely toward session.timeout.ms ormax.poll.interval.ms tuning rather than a data or throughput problem.

fraud-detection-consumer-group rebalance rate: 0 rebalances in the last hour
# Rebalancing is ruled out as the cause -- assignment has been stable.

Step 3 — check the producer's write rate on that specific partition

With rebalancing ruled out, the next question is whether partition 1 is simply receiving more volume than usual. Comparing the partition's log-end-offset growth rate against its historical baseline answers this directly.

freshcart.orders partition 1 write rate:
  normal baseline: ~800 records/sec
  last 30 minutes: ~2,400 records/sec  (3x normal)

freshcart.orders partitions 0 and 2 write rate: unchanged from baseline

This is the finding: partition 1 specifically is receiving three times its normal write volume, while the other partitions are unaffected — consistent with the earlier per-partition lag shape, and consistent with a hot key (a single high-volume entity whose events all hash to the same partition) rather than a general traffic increase, which would have shown up evenly across all three partitions.

Step 4 — check the consumer's own processing rate on that partition

The final check confirms whether the consumer's processing speed on that partition has also degraded, or whether it is simply outmatched by 3x the normal volume it was already running close to capacity against.

records-consumed-rate for the consumer instance owning partition 1:
  before incident: ~850 records/sec (slightly above the ~800/sec baseline write rate)
  during incident: ~830 records/sec (essentially unchanged)

# The consumer's own throughput did not degrade. It was already
# running close to the partition's normal write rate with little
# headroom, and a 3x volume spike on that one partition instantly
# outpaced it.

Root cause: a hot key drove partition 1's write volume to roughly 3x normal, and the single consumer instance that owns that partition — which was already running with little spare capacity even at baseline volume — could not absorb the spike. This is not a cluster-wide capacity problem (the other two partitions and their consumers are fine), and it is not a rebalance problem (ruled out in step 2), so scaling the whole consumer group would not help beyond what partition count already allows.

The fix, and the longer-term follow-up

Immediate mitigation: since the consumer group is already at one consumer per partition, the immediate option is optimizing the per-record processing path for that specific consumer to buy headroom, or temporarily accepting the lag and letting it drain once the spike subsides, provided it stays within retention. The longer-term fix is investigating why one key is receiving 3x the volume of its peers and whether the partition key choice needs to change to spread that specific entity's traffic more evenly — the same hot-partition problem covered in the producers-and-brokers module's partitioning discussion, here caught through monitoring rather than through a post-incident review.

From Metrics to a Dashboard That Actually Gets Used

Collecting the right metrics and setting sensible alert thresholds, covered in Parts 02 through 09, solves the reactive half of monitoring — noticing when something is already wrong. The other half is a dashboard layout that makes the cluster's state legible at a glance during a live incident, and a habit of reviewing longer-term trends for capacity planning before a slow-moving problem becomes an urgent one.

What belongs on the first screen

A common mistake is building one large dashboard with every available metric spread evenly across the screen, which means the two or three signals that matter most during an actual incident are no more visually prominent than dozens of secondary ones. The metrics from Part 02 and Part 04 — under-replicated partitions and active controller count — earn a permanently visible, oversized position specifically because they have no legitimate bad state at all; anything else on the same screen requires more judgment to interpret.

Using trend data for capacity planning, not just incident response

The same metrics that page someone during an acute incident are also the right inputs for deciding, well ahead of any page, when a cluster needs more brokers, more partitions, or larger disks. A request handler idle ratio (Part 03) that has been trending downward for two months, even though it has never crossed the alert threshold, is telling you the same thing an alert would eventually tell you — just earlier, and with enough lead time to plan the change instead of reacting to it.

request-handler-idle-ratio, weekly average:
  8 weeks ago: 62%
  6 weeks ago: 58%
  4 weeks ago: 51%
  2 weeks ago: 44%
  this week:   38%

# No single week crossed an alert threshold. The trend line is the
# signal: at the current rate of decline, this broker pool is on a
# path toward saturation within roughly another month. This is a
# capacity-planning finding, surfaced from exactly the same metric
# used for incident alerting -- read differently, on a longer window.

This is also where under-replicated-partition history (not just its current value) is worth retaining and reviewing — a partition that briefly goes under-replicated during every single rolling restart is arguably expected and not worth an individual page each time, but if that same pattern is taking longer to recover with each restart, that is itself a capacity signal, distinct from the instantaneous alert covered in Part 09.

Metric retention — how long Prometheus should keep this data

Tier 3's trend view only works if the underlying time series is actually retained long enough to show a multi-week or multi-month trend. A Prometheus instance configured with a short retention window (a common default for a fast-moving development environment) will happily serve Tier 1 and Tier 2 dashboards but silently cannot answer "has this been getting worse over the last two months" — the data to answer that question was already deleted.

Monitoring the Wider Ecosystem: Connect, Schema Registry, and End-to-End Latency

Parts 02 through 07 cover brokers, producers, and consumers — the three roles at the center of every Kafka deployment. In practice, most real pipelines also depend on Kafka Connect workers moving data in and out of external systems, and a schema registry every producer and consumer relies on. Each of these has its own health signals, and each can silently degrade a pipeline in a way that looks, from the broker's point of view, like nothing is wrong at all.

Kafka Connect metrics

A Connect worker exposes its own JMX metrics, scraped through the same JMX Exporter toolchain from Part 08. The most important of these mirror consumer and producer health in spirit, but at the level of an individual connector task rather than a generic client.

fraud-connect-sink-task-0 status: RUNNING
sink-record-send-rate: 0 records/sec (was ~500/sec an hour ago)
put-batch-avg-time-ms: 42,000 ms (was ~80ms an hour ago)

# The consumer group backing this connector shows growing lag --
# indistinguishable, from the broker's point of view, from a
# generically slow consumer. The Connect-level metrics reveal the
# actual cause: the external database this sink writes to has
# become extremely slow to accept writes. Scaling the consumer
# group or investigating Kafka broker health would not have found
# this -- the bottleneck is downstream of Kafka entirely.

Schema registry metrics

A schema registry sits in the critical path of both producing and consuming, even though it is not a Kafka broker itself — a producer or consumer that cannot reach the registry to fetch or register a schema typically cannot proceed at all. Its own request latency and error rate are worth monitoring with the same seriousness as broker request latency, since a registry outage can stall every producer and consumer across the entire cluster simultaneously, even while every broker reports perfectly healthy.

End-to-end latency — the metric no single component reports on its own

Every metric so far, on brokers, producers, consumers, Connect, and the schema registry, measures one component in isolation. None of them directly answers the question a business stakeholder actually cares about: how long does it take, right now, from the moment an event happens to the moment a downstream system has acted on it? That end-to-end latency has to be constructed deliberately, usually by embedding a timestamp in the event at production time and comparing it against a timestamp at the point the consumer finishes processing.

# At produce time, the event already carries its creation timestamp
event = {"order_id": "o-8834", "event_time": "2026-09-11T14:02:11.204Z", ...}

# At the point a consumer finishes processing, compute:
end_to_end_latency_ms = now() - parse(event["event_time"])

# Emitted as its own histogram metric, e.g.
# order_processing_end_to_end_latency_ms

# This single number reflects the combined effect of producer
# batching (Part 06 of the producers/brokers module), broker
# request handling (Part 03), consumer lag (Part 07), and any
# Connect or downstream processing time -- all of it, in one
# metric that maps directly to what a stakeholder actually
# experiences.

Turning Metrics Into SLOs: Deciding What "Good Enough" Actually Means

Every metric and alert threshold covered so far answers "is something currently broken." A service-level objective (SLO) asks a related but distinct question: over a meaningful window of time, is the pipeline meeting the reliability bar its consumers actually need — and how much room is left before it doesn't? Framing Kafka monitoring in terms of SLOs turns a pile of individually useful metrics into a single, business-relevant number that both engineers and non-engineers can reason about together.

Choosing the right SLI for a Kafka pipeline

A service-level indicator (SLI) is the specific measurement an SLO is built on. For a Kafka-backed pipeline, the end-to-end latency metric from Part 12 is usually the strongest SLI candidate, because it is the one number that reflects the combined health of every component — broker, producer, consumer, and any Connect or downstream processing — in a way a stakeholder outside the data platform team can actually understand and care about.

SLI: order_processing_end_to_end_latency_ms (from event creation
     to fraud-detection-service finishing its check)

SLO: 99% of orders processed within 5,000 ms, measured over a
     rolling 30-day window

Error budget: 1% of orders over 30 days may exceed 5,000 ms
     before the SLO is considered breached

The error budget framing changes how alerting and incident response are prioritized. A single lag spike that resolves within a few minutes (the kind of event Part 09 explicitly designs alert thresholds to ignore) consumes a small, known amount of error budget and does not, by itself, require an incident response. A sustained degradation that is visibly on pace to exhaust the month's error budget — even before the SLO is technically breached yet — is exactly the kind of early warning that justifies proactive investigation, ahead of a customer-visible failure.

Mapping component metrics to SLO risk

None of the component-level metrics from Parts 02 through 07 need their own separate SLO — instead, they function as the diagnostic tools for understanding why the end-to-end SLI is trending the way it is, exactly as the Part 10 worked incident demonstrated for a single lag spike. The SLO is the business-facing summary; the component metrics are how an engineer explains and fixes a bad reading on that summary.

Before a Cluster Goes Live: A Concrete Monitoring Checklist

As with security, the gap between "monitoring is set up" and "monitoring actually catches real incidents" is usually a handful of specific, checkable items that are easy to defer. A cluster with a Grafana dashboard already open in a browser tab can still be missing the one alert that would have caught the incident that eventually happens. This checklist pulls together the concrete, verifiable items from this module into a single pre-production pass.

From Alert to Fix: Runbooks, and What Is Safe to Automate

An alert firing is the start of a response, not the end of one. Whether that response takes five minutes or forty-five often comes down to a single factor unrelated to the underlying technical skill of whoever is on call: does a written runbook already exist for this specific alert, or does the responder have to reconstruct the diagnostic process from scratch, under time pressure, the way Part 10 walked through once as a worked example.

Turning a worked incident into a reusable runbook

Part 10's consumer lag investigation is exactly the shape a runbook should take: a specific, ordered sequence of checks, each with a concrete command and an explicit "if this, then check that next" branch, rather than a vague description of the general area to investigate. The value of writing this down is that the next person who hits a similar alert — who may not have built the original mental model of why the checks are ordered the way they are — can still follow the same effective path.

ALERT: consumer-group-lag-sustained-growth

1. Run kafka-consumer-groups.sh --describe for the alerting group.
   -> Is lag concentrated in one/few partitions, or spread evenly?
      Concentrated -> go to step 2.
      Evenly spread -> likely a group-wide capacity issue; check
        whether consumer count already equals partition count
        before considering any other cause.

2. Check the group's rebalance rate over the last hour.
   -> Rebalancing frequently? Investigate session.timeout.ms and
      max.poll.interval.ms tuning; this likely explains the lag on
      its own.
   -> Stable assignment? Go to step 3.

3. Compare the affected partition's current write rate to its
   historical baseline.
   -> Significantly elevated? Likely a hot key or traffic spike;
      check whether the partition key choice is the underlying
      cause.
   -> Normal? Go to step 4.

4. Compare the consumer's own records-consumed-rate, before and
   during the incident, for the partition(s) involved.
   -> Degraded? Investigate the consumer's own processing path
      (slow downstream call, GC pause, resource contention).
   -> Unchanged? The consumer was already near capacity and got
      outpaced by upstream volume -- see step 3's finding.

What is reasonable to automate, and what still needs a human

Some responses in this module are safe to automate because they are low-risk and clearly correct regardless of root cause; others genuinely need a human to interpret the situation before acting, because the "fix" depends entirely on which branch of the runbook the investigation lands on.

Five Misconceptions About Kafka Monitoring

What This Looks Like on Day One

5 Interview Questions — With Complete Answers

The Mistakes That Make Kafka Monitoring Ineffective

Errors and Alert States You Will Hit — And Exactly Why They Happen