Disaster Recovery and Multi-Region Kafka

Why a Single Cluster in a Single Region Is a Single Point of Failure

Every module up to this point has treated "the Kafka cluster" as a fixed, given thing — a set of brokers in one place that you scale, secure, and tune. That framing quietly hides an assumption: that the region hosting those brokers is always available. It is not. Cloud regions have full outages — a shared networking fabric, a power event, a control-plane failure in the provider's own infrastructure. When that happens, every broker in that region becomes unreachable at once, regardless of how carefully you configured replication factor and min.insync.replicasinside the cluster.

This is the blast radius problem: replication within a cluster protects you against losing a broker, a disk, or a rack. It does nothing against losing the region the entire cluster lives in. A three-broker cluster with replication factor 3 is fully durable against any single broker failure — and fully unavailable the moment the region itself goes dark, because all three replicas of every partition were in that same region.

The same blast radius applies to operator error, not just infrastructure failure. A botched rolling config change pushed to every broker in the cluster, a bad ACL update that locks out every producer, a runaway retention-policy change that deletes data cluster-wide — none of these are "region" failures, but they still take down the entire event backbone, because there is only one cluster and every application depends on it. Disaster recovery for Kafka is really about answering one question: when this one cluster becomes unusable — for any reason — how does the business keep moving?

Rack Awareness — Surviving an Availability Zone Failure Inside One Cluster

Before reaching for multiple clusters and multiple regions, there is a cheaper, simpler layer of resilience available inside a single cluster: spreading replicas of the same partition across different availability zones (AZs) within one region. This is called rack awareness, and it is controlled by a single broker-level configuration, broker.rack.

Without rack awareness, Kafka's default replica placement strategy only guarantees that replicas of a partition land on different brokers — it has no idea that broker-1, broker-2, and broker-3 might all happen to sit in the same physical availability zone, in which case a replication factor of 3 gives you zero protection against that one AZ failing. broker.rack lets each broker declare which physical or logical "rack" it belongs to — in a cloud deployment this is almost always set to the cloud AZ identifier — and the partition-assignment logic then spreads replicas across distinct racks whenever possible.

# broker-1/server.properties
broker.id=1
broker.rack=us-east-1a

# broker-2/server.properties
broker.id=2
broker.rack=us-east-1b

# broker-3/server.properties
broker.id=3
broker.rack=us-east-1c

# broker-4/server.properties (added later, same AZ as broker-1)
broker.id=4
broker.rack=us-east-1a

# With rack awareness enabled, a partition with replication.factor=3
# on this 4-broker cluster will be assigned to three DIFFERENT racks
# whenever three distinct racks are available -- never all three
# replicas landing in us-east-1a alone, even though two brokers live there

What rack awareness actually buys you

If one AZ fails entirely — a real, common failure mode in every major cloud provider, far more common than a full region failure — a rack-aware cluster still has at least one in-sync replica of every partition alive in a surviving AZ. The controller promotes that replica to leader, and the cluster keeps serving reads and writes without operator intervention. Without rack awareness, an AZ failure has some real chance of taking out every replica of some partitions simultaneously, because nothing prevented all of them from landing in the same AZ.

Rack awareness is not multi-region DR — it does nothing if the entire region fails, since every AZ in that region goes down together. But it is the first, cheapest, and most important layer of resilience, and it should be configured on every production cluster regardless of whether multi-region DR is also in place. It has essentially no downside: replication traffic between AZs typically has low, predictable latency (single-digit milliseconds within a region), soacks=all write latency is barely affected compared to same-AZ replication.

The Real Trade-Off of a Cross-Region Stretched Cluster

If spreading replicas across AZs within a region protects against an AZ failure, the obvious next idea is: why not spread replicas across regions too? Put broker-1 in us-east-1, broker-2 in us-west-2, broker-3 in eu-west-1, and now a whole region failing still leaves two regions with a live replica. This is technically possible to configure. It is very rarely what production teams actually run, and the reason is latency, not theory.

Recall from earlier modules that acks=all means the leader waits for every replica currently in the ISR to confirm the write before acknowledging the producer. If those replicas are in different regions, that acknowledgement now has to wait for a full round trip across whatever network path connects the regions — typically 30-90ms between two US regions on the same continent, and 100ms or more between continents. Compare that to same-AZ replication (under 1ms) or cross-AZ-same-region replication (1-5ms). A cross-region stretched cluster turns everyacks=all write into a multi-hundred-times-slower operation.

# Same-AZ replication (leader and follower in the same data center):
# round trip: under 1ms
# acks=all write latency: roughly the disk fsync time, a few ms

# Cross-AZ, same-region replication (us-east-1a to us-east-1c):
# round trip: 1-5ms
# acks=all write latency: single-digit ms -- barely noticeable

# Cross-region replication (us-east-1 to us-west-2):
# round trip: ~60-70ms
# acks=all write latency: dominated by that 60-70ms round trip
# -- a write that used to take 3ms now takes 60-70ms
# -- for a payment API waiting on that write before responding to
#    a user, this is the difference between a snappy checkout and
#    a customer-visible slowdown on every single order

# Cross-continent replication (us-east-1 to eu-west-1):
# round trip: 90-120ms
# acks=all write latency: effectively unusable for
#   latency-sensitive, synchronous write paths

For latency-insensitive batch workloads, a stretched cluster's latency hit might be tolerable. But most production Kafka topics carry a mix of traffic, and the slowest path sets the tone for the whole cluster's behavior under acks=all. This is why, in practice, almost no production team runs one Kafka cluster with brokers physically spread across regions for latency-sensitive workloads. Instead, the standard pattern is multiple independent clusters — one per region — kept in sync by a separate replication tool running between clusters, not inside one. That tool is MirrorMaker 2, covered next.

MirrorMaker 2 — Replicating Between Independent Clusters

MirrorMaker 2 (MM2) is Kafka's standard tool for replicating data between two independent clusters — it is not a broker feature, it is a separate application, built on the Kafka Connect framework, that runs as its own set of processes reading from a source cluster and writing to a target cluster. Because it is built on Connect, it inherits Connect's scalability and fault-tolerance model: MM2 workers can be scaled horizontally, and failed tasks are automatically reassigned to healthy workers.

MM2 replicates three distinct things, and it is worth naming all three because teams often assume it only copies message data:

• ✓Topic data — the actual records, preserving partition-to-partition mapping where possible, so partition 3 on the source generally maps to partition 3 on the target.
• ✓Consumer group offsets — via a mechanism called offset translation, so a consumer group failing over to the target cluster can resume from roughly the right position instead of starting from the beginning or the end.
• ✓Configuration and ACLs — topic configs (retention, partition count, compaction settings) and access-control entries can be kept in sync so the target cluster is not a bare, unconfigured clone.

# mm2.properties -- replicating us-east (primary) to us-west (DR)

clusters = us-east, us-west
us-east.bootstrap.servers = broker-a1:9092,broker-a2:9092,broker-a3:9092
us-west.bootstrap.servers = broker-b1:9092,broker-b2:9092,broker-b3:9092

# Enable replication in one direction only: us-east -> us-west
us-east->us-west.enabled = true
us-west->us-east.enabled = false

# Which topics to mirror
us-east->us-west.topics = orders.*, payments.*, inventory.*

# Replicate consumer group offsets too, not just topic data
us-east->us-west.emit.checkpoints.enabled = true
us-east->us-west.sync.group.offsets.enabled = true
us-east->us-west.emit.checkpoints.interval.seconds = 30

# Replicate topic configs (retention, partitions, compaction) too
us-east->us-west.sync.topic.configs.enabled = true

# Replication factor for the mirrored topics on the target cluster
replication.factor = 3

Topic naming — how MM2 avoids infinite replication loops

A subtle but important detail: when MM2 replicates a topic named orders from us-east to us-west, it does not write to a topic also named orders on the target. By default it prefixes the topic name with the source cluster's alias — the replicated topic on us-west is named us-east.orders. This naming convention is what makes bidirectional replication (used in active-active designs, covered next) safe: it prevents MM2 from picking up the records it just replicated and re-replicating them back to the source in an infinite loop, because us-east.orders is a distinct topic from orders, and the replication topic filter only matches the original topic names, not the mirrored ones.

# On us-west, after MM2 replication is running, topics visible include:
us-west> kafka-topics --bootstrap-server broker-b1:9092 --list

orders                     <- native topics created directly on us-west, if any
us-east.orders              <- mirrored copy of us-east's "orders" topic
us-east.payments            <- mirrored copy of us-east's "payments" topic
us-east.inventory            <- mirrored copy of us-east's "inventory" topic
us-east.checkpoints.internal  <- MM2's internal offset-translation bookkeeping topic

Active-Passive — One Region Serves Traffic, the Other Waits

Active-passive is the simpler and by far the more common multi-region Kafka DR pattern. One cluster — the primary — serves all production traffic: every producer writes there, every consumer reads from there. A second cluster — the secondary, or DR cluster — sits in a different region, continuously receiving a mirrored copy of the primary's data via MM2, but no application writes to it or reads from it during normal operation. It exists purely to be promoted if the primary becomes unusable.

                     normal operation
                     ─────────────────

  producers ──write──► [ us-east cluster ] ──MM2──► [ us-west cluster ]
                              │                            │
                        (primary, live)              (passive, DR-only)
                              │
  consumers ◄──read────────────┘

  # us-west receives a continuous mirrored copy but serves
  # no production traffic under normal conditions

The appeal of active-passive is operational simplicity: applications only ever talk to one cluster at a time, so there is no need to reason about conflicting writes from two regions touching the same data — the primary is the single source of truth until the moment of failover. The cost is that the secondary region's capacity sits mostly idle, paid for but unused, until the day it is needed.

The real gotcha — offsets do not automatically match between clusters

This is the detail that catches teams off guard the first time they actually fail over. A consumer group reading from the primary cluster has committed offsets like "partition 3, offset 48,201." When that consumer group is redirected to the DR cluster after a failover, offset 48,201 on the DR cluster's copy of that partition is not necessarily the same record as offset 48,201 on the primary. MM2 writes replicated records to the target cluster as new records, at whatever offsets the target partition happens to be at when they arrive — the offsets are not preserved identically across clusters.

If a consumer just resumes from its old, primary-cluster offset number on the DR cluster, it will either skip records (if the DR cluster's offset numbering is ahead of where that data actually starts) or reprocess a huge amount of already-processed data (if it's behind). This is exactly why MM2's offset translation feature exists: it maintains a mapping, per consumer group, between "offset X on the source cluster" and "the corresponding offset on the target cluster," recorded continuously as replication happens, specifically so that a failed-over consumer group can resume from the translated equivalent position rather than a raw, meaningless offset number.

# MM2's internal checkpoint topic records translations like:

# consumer_group=order-processing-service, source=us-east,
#   source_topic=orders, partition=3, source_offset=48201
#   -> target_offset=48198 (on us-west's us-east.orders topic)

# On failover, the DR runbook (or an automated tool built on top of
# MM2's RemoteClusterUtils / checkpoint API) looks up this mapping
# and explicitly seeks the consumer group to offset 48198 on
# us-west's us-east.orders partition 3 -- NOT to raw offset 48201,
# and NOT just to "latest" or "earliest," either of which would
# silently skip or reprocess a large amount of data

# Without offset translation, a team's only fallback is an
# application-level reconciliation: seeking consumers by TIMESTAMP
# instead of offset (using the timeindex covered in earlier modules),
# accepting some imprecision, and relying on idempotent downstream
# processing to absorb any resulting reprocessing

Active-Active — Both Regions Serve Writes, and Conflicts Become Your Problem

Active-active takes the next step: both regional clusters serve live production traffic simultaneously, each with local producers and consumers, and MM2 (or an equivalent tool) mirrors data bidirectionally between them, so each region eventually has a copy of the other's data. The appeal is obvious — no idle secondary region, and lower write latency for users in each region since they write to their local cluster rather than a distant primary.

  us-east producers ──write──► [ us-east cluster ] ◄──MM2, both ways──► [ us-west cluster ] ◄──write── us-west producers
                                      │                                       │
                                 us-east consumers                     us-west consumers
                                 (read local + mirrored                (read local + mirrored
                                  us-west.* topics)                     us-east.* topics)

  # both regions accept writes AND serve reads to local applications
  # each region eventually sees the other's data, with replication lag

The real complexity active-active introduces is not replication mechanics — MM2 handles the bidirectional mirroring the same way it handles one-directional replication, using the source-cluster-prefixed topic naming from Part 04 to avoid replication loops. The real complexity is that Kafka itself does not dedupe or merge writes across independently-written clusters. If an entity's state can be modified from either region — the same customer's shopping cart, the same inventory count for the same product SKU — and both regions write an update to that entity at close to the same time, you now have two conflicting versions of that entity's state, replicated to both clusters, with no built-in mechanism to reconcile them.

A concrete conflict scenario

# SKU-4471 has 10 units in stock, tracked as a compacted-topic
# changelog entry with key=SKU-4471

# t=0    us-east cluster: 10 units in stock (both regions agree so far)
# t=1    a customer in New York buys 3 units
#        -> us-east producer writes: SKU-4471, stock=7
# t=1.05 (50ms later) a customer in Los Angeles, whose request is
#        routed to us-west, buys 4 units, based on us-west's LOCAL
#        view of stock which STILL SAYS 10, because MM2's replication
#        of the t=1 update hasn't arrived at us-west yet
#        -> us-west producer writes: SKU-4471, stock=6

# Now both writes replicate to both clusters:
# us-east ends up with: stock=7, then later overwritten by stock=6 (replicated from us-west)
# us-west ends up with: stock=6, then later overwritten by stock=7 (replicated from us-east)

# Whichever write arrives LAST at each cluster "wins" -- last-write-wins
# by arrival time, not by correctness. The true stock level after two
# independent purchases of 3 and 4 units from a starting 10 should be 3.
# Neither region's version (6 or 7) is correct -- both undercounted
# how much was actually sold, because neither wrote against the other's update

This is not a Kafka bug — Kafka faithfully replicated exactly what each producer wrote, in the order each cluster received it. The conflict exists because the business logic allowed the same mutable entity to be modified independently from two places without coordination. Solving it requires application-level design, not a Kafka setting: options include partitioning entities by region so each one has a single writing region (a customer's cart is only ever written from the region they're logged into), using conflict-free replicated data types (CRDTs) for counters that need to merge correctly regardless of order, or routing all writes for a given entity through one region while still serving reads from both (a hybrid that is really active-passive at the entity level, wrapped in an active-active-looking topology).

RPO and RTO — Putting Numbers on "How Bad Is a Failover"

Disaster recovery conversations need concrete, measurable targets, not just "we have a DR cluster." The two standard metrics, borrowed from general DR practice and directly applicable to Kafka, are Recovery Point Objective (RPO) and Recovery Time Objective (RTO).

RPO — how much data could be lost

Recovery Point Objective answers: if the primary cluster disappears right now, how much recently written data has not yet made it to the DR cluster, and is therefore lost? For MM2-based replication, RPO is directly determined by replication lag at the moment of failure — the gap between what the primary has committed and what the DR cluster has actually received and durably written.

RPO is not a fixed number; it is a moving target that depends on network conditions between regions, MM2's own throughput and tuning, and how close to real time the specific topic's replication has been keeping up. A healthy MM2 pipeline under normal load might run seconds behind. A pipeline under strain — a network degradation between regions, or MM2 workers under-provisioned for the topic's volume — could be minutes behind, and that lag is exactly how much data an instantaneous primary-region failure would lose.

RTO — how fast you can cut over

Recovery Time Objective answers: from the moment the primary is declared unusable, how long until traffic is fully served by the DR cluster again? This includes detecting the failure, deciding to fail over (often a deliberate human decision, not an automatic one, precisely because falsely failing over has its own cost), reconfiguring or redeploying producers and consumers to point at the DR cluster, and validating that offset translation landed consumer groups in a sane place before declaring the failover complete.

# Observed MM2 replication lag under normal conditions: 8 seconds average
# Observed lag during a network degradation event last quarter: 4 minutes peak

# RPO, normal conditions: ~8 seconds of data at risk in a sudden failure
# RPO, degraded conditions: up to 4 minutes of data at risk

# Failover runbook (rehearsed quarterly), timed end to end:
#   detect primary unavailable (alerting):        ~2 minutes
#   decide to fail over (human confirmation):      ~3 minutes
#   redirect producers/consumers to DR cluster:    ~5 minutes
#   validate offset translation landed correctly:  ~5 minutes
#   ────────────────────────────────────────────────────────
#   RTO (rehearsed, under normal conditions):     ~15 minutes

# These numbers only mean anything if measured from real drills,
# not estimated from how the runbook reads on paper

RPO and RTO targets should come from the business, not from infrastructure convenience — a payments topic might demand an RPO of seconds and an RTO of minutes, justifying tightly monitored MM2 lag and a heavily automated failover process, while an internal analytics-events topic might tolerate an RPO of tens of minutes and an RTO measured in hours. Applying the same DR rigor to every topic regardless of its actual business cost of loss is how DR programs become expensive without becoming more effective.

Topic-to-S3 Snapshotting — The Last-Resort Restore Path

MM2 replication is designed for fast failover — a DR cluster that is ready to serve live traffic within minutes. It is not designed as a long-term archival backup, and it shares a real weakness with the primary cluster: it is still a live, running Kafka cluster, subject to its own operator error, its own storage failures, and its own potential for a bad config change to be mirrored right alongside the good data it's replicating.

For genuine last-resort recovery — a scenario severe enough that both the primary and DR clusters are compromised, or where you need to reconstruct historical data outside of Kafka's own retention window entirely — teams commonly run a Kafka Connect sink connector that continuously exports topic data to object storage (commonly S3), independent of both live clusters. This is not a replacement for MM2-based DR; it is a separate, slower, cheaper safety net underneath it.

{
  "name": "orders-s3-backup-sink",
  "config": {
    "connector.class": "io.confluent.connect.s3.S3SinkConnector",
    "tasks.max": "4",
    "topics": "orders,payments,inventory",
    "s3.bucket.name": "kafka-backups-orders-payments",
    "s3.region": "us-east-1",
    "flush.size": "10000",
    "rotate.interval.ms": "300000",
    "storage.class": "io.confluent.connect.s3.storage.S3Storage",
    "format.class": "io.confluent.connect.s3.format.parquet.ParquetFormat",
    "partitioner.class": "io.confluent.connect.storage.partitioner.TimeBasedPartitioner",
    "path.format": "'year'=YYYY/'month'=MM/'day'=dd/'hour'=HH",
    "locale": "en-US",
    "timezone": "UTC"
  }
}

Restoring from an S3 snapshot is deliberately a manual, slow-path operation: a batch job reads the Parquet (or Avro/JSON) files back out of S3 and produces them into a freshly created topic on whichever cluster is being rebuilt. This is orders of magnitude slower than MM2 failover and loses the original partition-level offsets entirely, which is exactly why it is a last resort rather than the primary DR mechanism — it exists for the scenario where MM2-based replication itself has failed or was never keeping up, not as the everyday recovery path.

A Regional Failover Runbook, Worked End to End

Bringing the whole module together, here is what an active-passive regional failover actually looks like in practice, for a company running an order-processing platform with a primary cluster in us-east-1 and a passive DR cluster in us-west-2, mirrored continuously by MM2.

# ── STAGE 1: Detection ─────────────────────────────────────────────
# Alerting fires: us-east-1 broker health checks failing cluster-wide,
# not just one broker. Confirmed via cloud provider's own status page
# showing a us-east-1 networking incident. This is NOT a single-broker
# issue that self-heals via normal replication -- it is a region-level
# event, which is the trigger condition for this runbook.

# ── STAGE 2: Declare the failover ────────────────────────────────
# On-call incident commander confirms with a second engineer.
# Failover is a DELIBERATE decision, not automatic -- a false failover
# has real cost too (split traffic, confused consumers, wasted DR
# capacity), so this step is intentionally a human checkpoint.

# ── STAGE 3: Check MM2 replication lag AT time of failure ────────
us-west> kafka-consumer-groups --bootstrap-server broker-b1:9092 \
  --describe --group mm2-orders-replication

# LAG column shows: partition 0: 1,204 records, partition 1: 890 records
# This tells the team roughly how much data is about to be "lost"
# for RPO reporting purposes -- these records were written to
# us-east but had not yet replicated to us-west when it went down

# ── STAGE 4: Promote consumer groups using offset translation ────
us-west> kafka-mirror-maker-offset-tool --translate-offsets \
  --group order-processing-service \
  --source-cluster us-east --target-cluster us-west

# This seeks every consumer in the order-processing-service group
# to the translated-equivalent offset on us-west's mirrored topics,
# not a raw copy of the us-east offset numbers (Part 05)

# ── STAGE 5: Redirect producers and consumers ─────────────────────
# Application config / service discovery updated:
#   bootstrap.servers: broker-a1,broker-a2,broker-a3 (us-east)
#     -> broker-b1,broker-b2,broker-b3 (us-west)
# Rolled out via existing config-management pipeline, same one used
# for routine deploys -- deliberately NOT a special one-off mechanism,
# so it is well-tested by ordinary use rather than only during a crisis

# ── STAGE 6: Validate ──────────────────────────────────────────────
# Confirm: producers writing successfully to us-west
# Confirm: consumer lag on us-west's mirrored topics is draining,
#   not growing -- proves consumers picked up from a sane position
# Confirm: downstream systems (payment processor, notification
#   service) are receiving events again

# ── STAGE 7: Stop the clock, record RTO/RPO ───────────────────────
# Time from Stage 1 detection to Stage 6 validated: 17 minutes (RTO)
# Records lost per Stage 3's lag reading: ~2,094 records across two
#   partitions (RPO) -- logged for post-incident review and for
#   deciding whether MM2 capacity needs to increase to tighten RPO

The runbook's value comes entirely from having been rehearsed, not from existing on paper. A failover drilled quarterly, with the team actually executing Stages 1 through 7 against a real (non-production) cluster pair, surfaces the gaps that a document review never will — a config management pipeline that turns out to require manual approval steps nobody remembered, an offset translation tool that needs a flag nobody had used before, a monitoring dashboard that does not actually show MM2 lag broken out by topic. Every one of those gaps is cheap to fix in a drill and expensive to discover during an actual regional outage.

Monitoring DR Readiness — Catching a Silent DR Failure Before You Need It

A DR cluster that has quietly stopped receiving replication traffic looks, from a distance, identical to a healthy one — until the day the primary fails and the team discovers the secondary has been weeks stale. This is the single most dangerous failure mode in any DR program: not the primary failing, but the DR mechanism itself failing silently, unnoticed, until it's needed and found wanting. Monitoring DR readiness deserves the same rigor as monitoring the primary cluster.

The last row is easy to dismiss as not really "monitoring," but it belongs on the same dashboard conceptually as the technical signals. A DR cluster with perfect replication lag numbers and a runbook nobody has executed in eight months carries real, unmeasured risk — precisely the kind Part 09 emphasizes drills exist to surface before an actual incident does.

Paying for Resilience — What Multi-Region DR Actually Costs, and How to Scope It

Every pattern in this module has a real dollar cost, and pretending otherwise is how DR proposals stall in budget review. A passive DR cluster in a second region means paying for a second set of brokers, sized to handle full production load even though it serves none of it day to day — plus cross-region data transfer costs for the continuous MM2 replication stream, which scale directly with how much data you're mirroring and how many topics you've chosen to replicate.

The framing that tends to get DR investment approved is not "resilience is important" in the abstract — it's translating Part 07's RPO/RTO targets into a concrete cost of downtime and comparing it directly against the cost of the DR infrastructure. A regional outage that costs the business $40,000 per hour in lost orders, held for two hours because no DR plan existed, is a $80,000 incident that a DR cluster costing a few thousand dollars a month would have prevented or dramatically shortened — that comparison, with real numbers specific to the business, is what makes the investment case land.

How "Ready" Should the DR Cluster Be? Three Levels of Standby

"Active-passive" is not one single configuration — there is a real spectrum in how ready the passive cluster is kept, and where a team lands on that spectrum is itself a cost/RTO trade-off worth making deliberately, not by default.

Most of this module's guidance, including the Part 09 runbook, describes warm standby — it is the most common production choice because it gets RTO down to a reasonable, business-justifiable number without paying for a fully duplicated, always-scaled second production environment. Cold standby is sometimes acceptable for genuinely low-priority topics where a multi-hour RTO is tolerable and the cost savings matter more. Hot standby is reserved for the small set of systems where minutes of RTO genuinely isn't good enough and the cost is justified — most commonly regulated financial systems with explicit, contractual recovery-time requirements.

Drilling Failover Safely — Chaos-Style Testing for a DR Pipeline

A recurring objection to frequent DR drills is real and worth addressing directly: nobody wants to risk an actual production outage just to test the failover process. The good news is that most of the DR pipeline can be validated without ever touching production traffic, using the same layered approach chaos-engineering practice applies elsewhere.

A mature DR program uses all four layers on different cadences: the non-production full-failover drill and the shadow-consumer-group validation can run monthly or even continuously, cheaply, because they carry essentially no production risk. The game-day exercise and the full production drill are reserved for a slower cadence — quarterly is common — specifically because they carry real, if carefully scoped, risk and require more coordination to execute safely.

Don't Forget the Supporting Systems — Schema Registry and Security Config Also Need a DR Story

Everything in this module has focused on the Kafka clusters themselves, but a production Kafka deployment almost always runs alongside a schema registry (covered in earlier modules for its role in enforcing compatibility) and a set of access-control configurations. A DR plan that covers the Kafka clusters but forgets these supporting systems still leaves a real gap: a failed-over cluster that producers and consumers cannot actually authenticate against, or a DR cluster whose schema registry doesn't have the schemas needed to deserialize the very data MM2 has been faithfully replicating into it.

Five Misconceptions About Kafka Disaster Recovery

What This Looks Like on Day One

5 Interview Questions — With Complete Answers

The Mistakes That Undermine Kafka DR Plans

Errors You Will Hit — And Exactly Why They Happen