Change Data Capture with Debezium
What Change Data Capture actually is, why polling for changes is fragile, how Debezium reads a database's own transaction log instead of querying tables, the Debezium event envelope, initial snapshots, the outbox pattern, schema evolution, and the operational pitfalls of running CDC in production.
Change Data Capture — Every Row Change, As It Happens, Without Asking the Database Twice
Change Data Capture (CDC) is the practice of capturing every row-level insert, update, and delete made to a database, in the order they happened, as a stream of events — rather than periodically asking the database "what changed since I last checked." The distinction sounds small. It is not. A CDC system produces one event per write. A polling system produces a batch summary of net effects observed at whatever interval it happens to run, which is a fundamentally lossier signal.
Most teams' first encounter with the need for CDC looks the same: a service owns a Postgres table — orders, customers, inventory — and three other teams each want to know when a row changes, so they can react. The instinct is to add a scheduled job: SELECT * FROM orders WHERE updated_at > last_run_time, run every five minutes, publish whatever comes back. This works, briefly, and then breaks in ways that are hard to notice until they have already cost someone data.
What CDC gives you that polling structurally cannot:
Every change, not just the latest state — if a row is updated three times between polls, polling sees only the final value. CDC sees, and can emit, all three updates, each with its own before/after state.
Deletes, which polling cannot see at all — a WHERE updated_at > xquery finds rows that changed. A deleted row is gone; there is nothing left to select. Polling has no way to know a delete happened unless the application is disciplined enough to soft-delete every single row, forever, which most schemas do not do.
Near-zero load on the source database — CDC, done the way this module covers, reads the database's own internal change log rather than running repeated SELECTqueries against live tables, so it does not compete with production traffic for query capacity or lock contention.
This module builds directly on Kafka Connect (Module 13). Debezium — the dominant open-source CDC project, and the one this module focuses on — does not run as a separate system you deploy and operate independently. It runs as a Kafka Connect source connector, in distributed mode, configured through the same REST API, using the same converters and Single Message Transforms you already learned. Everything Module 13 taught about tasks, offsets, distributed-mode rebalancing, and DLQ handling for source connectors applies to Debezium directly — this module does not re-teach that ground, it builds on it.
The "Poll for Changes" Pattern Is Fragile in Three Specific, Predictable Ways
It is worth being precise about exactly how the polling approach fails, because each failure mode shows up independently in production, and each one is the kind of bug that passes code review and works fine in testing before quietly losing data at scale.
Failure 1 — deletes are invisible
A polling query built around a last_modified or updated_at column finds rows whose value in that column is newer than the last checkpoint. A row that was deleted no longer exists to be found by any query. Unless every table in the system is disciplined about soft-deletes (an is_deleted flag set instead of a real DELETE), and every consumer of the polling job knows to check that flag, deletions simply vanish from the event stream. Downstream systems — a search index, a cache, an analytics table — drift out of sync with the source of truth and nobody notices until a customer asks why a cancelled order is still showing up somewhere.
Failure 2 — intra-poll-window updates are collapsed
If a row is updated twice between two poll runs, the polling query sees the row once, with only its final value. Both intermediate states are gone. This matters more than it first appears: an inventory system that needs to know a product went from 40 units to 0 units to 15 units (a stockout and restock within one polling window) only sees "15 units" — the stockout, and any business logic that should have reacted to it, never happened as far as the event stream is concerned.
Failure 3 — polling load competes with production traffic
A polling job that scans a large table on a schedule adds real query load to a database that is also serving live application traffic. As tables grow, either the poll interval has to widen (making Failures 1 and 2 worse) or the query has to be more carefully indexed and rate-limited to avoid contending with production queries — and even a well-indexed poll is still periodic load the database's own internal write path does not otherwise need to pay.
# Timeline: three things happen to order_id=4821 in a 5-minute polling window
t=0:10 INSERT order_id=4821, status='pending'
t=1:40 UPDATE order_id=4821 SET status='paid'
t=3:05 DELETE FROM orders WHERE order_id=4821 -- customer cancelled, refunded
# Polling job runs at t=5:00, query: WHERE updated_at > last_checkpoint
# Result: ZERO rows returned for order_id=4821.
# The row was inserted, updated, and deleted, all inside the window --
# and the final state (deleted) means there is nothing left to select.
# Every one of these three events is invisible to the polling consumer.
# CDC reading the transaction log instead:
# offset 9001 op=c (create) order_id=4821 status=pending
# offset 9002 op=u (update) order_id=4821 status: pending -> paid
# offset 9003 op=d (delete) order_id=4821 status: paid -> (row gone)
# All three events exist on the Kafka topic, in order, individually.
# A downstream fraud-detection consumer that cares about rapid
# pay-then-cancel patterns can actually see this sequence happen.Debezium Reads the Database's Own Write-Ahead Log — It Never Runs a SELECT Against Your Tables
Every production relational database already maintains an internal, sequential log of every write it performs, for its own crash-recovery and replication purposes. Postgres calls this the write-ahead log (WAL). MySQL calls it the binary log (binlog). SQL Server has Change Data Capture or Change Tracking built directly into the engine. These logs exist independent of Debezium — every database that supports replication already writes one, because it is how the database replicates itself to standby replicas.
Debezium's core mechanism is to tap into this existing log rather than issuing queries. It effectively registers itself as a replication consumer — from the database's point of view, a Debezium connector looks similar to a standby replica asking to stream changes, not a client running repeated SELECT statements. This single design decision is what makes Debezium fundamentally different from a polling-based tool, and it is the reason it can capture deletes and every intermediate update with near-zero added load on the source database.
| Database | Log Debezium reads | Mechanism |
|---|---|---|
| PostgreSQL | Write-ahead log (WAL) | Logical replication slot, decoded via the pgoutput plugin (built into Postgres 10+) or the older decoderbufs plugin |
| MySQL | Binary log (binlog) | Debezium registers as a MySQL replication client, reading the binlog in ROW format |
| SQL Server | Change Data Capture (CDC) or Change Tracking tables | SQL Server's own built-in CDC feature must be enabled per-table; Debezium reads the resulting change tables |
| MongoDB | Oplog (operation log) | Debezium reads MongoDB's replica set oplog, the same log used for replica set replication |
Why this matters operationally — near-zero read load on the source
Because Debezium is consuming a log the database is already writing for its own purposes, it does not add meaningful query load to the tables it is capturing. It does not take row locks, does not compete with application queries for the query planner's attention, and does not scan tables. The practical impact for a team introducing CDC into a busy production database is that Debezium can typically be turned on against a live, high-traffic table without the kind of careful load-testing a new polling job against that same table would need.
1. A logical replication SLOT is created on the Postgres primary,
specifically for Debezium: e.g. slot name "debezium_freshcart"
2. Every committed write to a table included in that slot's
publication is appended to the WAL, exactly as it always was --
Debezium did not change how Postgres writes.
3. Debezium's connector streams from that replication slot,
decoding each WAL entry via the pgoutput logical decoding plugin
into a structured change: table, operation type, before/after row
4. Debezium turns each decoded change into a Kafka record and
produces it to the appropriate topic (see Part 05 for the shape)
5. Once Debezium acknowledges it has processed a WAL position,
Postgres is free to reclaim (garbage-collect) that portion of
the WAL -- see Part 08 for what happens when this acknowledgement
stops arrivingPOST /connectors REST endpoint covered in Module 13, running as tasks on Connect workers, with the same offset-tracking, rebalancing, and error-handling model as any other source connector.A Worked Example — Deploying a Postgres Connector Through the Connect REST API
Continuing with FreshCart's Postgres orders table: the fulfillment, fraud, and analytics teams all want a real-time stream of every insert, update, and delete on that table, including the ones a polling job would have silently dropped. Here is the connector configuration.
{
"name": "source-postgres-orders-prod",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"tasks.max": "1",
"database.hostname": "db.internal",
"database.port": "5432",
"database.user": "debezium_reader",
"database.password": "${file:/secrets/connect-creds.properties:db_password}",
"database.dbname": "freshcart",
"topic.prefix": "freshcart.pg",
"table.include.list": "public.orders",
"plugin.name": "pgoutput",
"slot.name": "debezium_freshcart_orders",
"publication.name": "dbz_publication_orders",
"snapshot.mode": "initial",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"key.converter.schemas.enable": "false",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "http://schema-registry:8081"
}
}- ✓connector.class — Debezium's Postgres connector plugin, submitted the exact same way any Connect source connector is.
- ✓plugin.name: "pgoutput" — the logical decoding plugin, built into Postgres since version 10, no extra Postgres extension install required.
- ✓slot.name — the name of the logical replication slot this connector owns. Only one consumer can read a given slot; the slot persists on the Postgres server itself until explicitly dropped (see Part 08 on why this matters).
- ✓table.include.list — restricts capture to specific tables rather than the whole database, keeping the blast radius of one connector small.
- ✓snapshot.mode: "initial" — take a full snapshot of existing rows once, then switch to streaming the WAL (see Part 06).
- ✓tasks.max: "1" — Debezium's Postgres connector runs a single task per connector; unlike the JDBC source connector from Module 13, parallelism across tables comes from running multiple connectors, not multiple tasks within one.
Submitting this is identical in mechanics to submitting any Connect source connector — one HTTP POST, one JSON payload, and the connector's status is checked the same way, atGET /connectors/source-postgres-orders-prod/status.
| Config property | What it actually controls |
|---|---|
| table.include.list / table.exclude.list | Which tables are captured — an allowlist or denylist of fully-qualified table names |
| column.exclude.list | Drops specific columns from captured events entirely — useful for excluding a sensitive column before it ever reaches Kafka |
| heartbeat.interval.ms | How often Debezium writes a heartbeat message to a dedicated topic, used to advance the WAL position even on quiet tables (see Part 08) |
| tombstones.on.delete | Whether a delete produces both a delete event and a following null-value tombstone, matching the compacted-topic tombstone convention |
Every Debezium Event Has the Same Shape — before, after, source, and op
Regardless of which database Debezium is capturing from, every change event it produces follows the same envelope structure. Learning this shape once means every Debezium-backed topic you ever consume, across any team or any database, is immediately readable.
{
"before": {
"order_id": 4821,
"customer_id": "C-991",
"status": "pending",
"total_cents": 4599
},
"after": {
"order_id": 4821,
"customer_id": "C-991",
"status": "paid",
"total_cents": 4599
},
"source": {
"version": "2.5.0.Final",
"connector": "postgresql",
"name": "freshcart.pg",
"ts_ms": 1758000000123,
"db": "freshcart",
"schema": "public",
"table": "orders",
"txId": 88213,
"lsn": 302981744,
"snapshot": "false"
},
"op": "u",
"ts_ms": 1758000000456
}| Field | Meaning |
|---|---|
| before | The row's full state immediately before this change. Null for an insert (there was no "before" row) and populated for update and delete. |
| after | The row's full state immediately after this change. Null for a delete (there is no "after" row) and populated for insert and update. |
| source | Metadata about where this change came from — the database, table, transaction ID, and log position (LSN for Postgres, file+offset for MySQL binlog) that produced it. |
| op | "c" (create/insert), "u" (update), "d" (delete), or "r" (read — an event produced during the initial snapshot, not a live change). |
| ts_ms | The wall-clock time Debezium processed and emitted this event, distinct from source.ts_ms which is when the database committed the change. |
The op field is the field most consumer logic branches on. A consumer materializing a downstream cache or search index typically upserts on c and u events using the after state, and deletes the corresponding document on a devent — a delete event's after is null, so the deletion signal is unambiguous, which is exactly the guarantee polling could never provide.
def handle_debezium_event(event):
op = event["op"]
if op in ("c", "u", "r"):
# insert, update, or initial-snapshot read -- upsert the current row
upsert_search_index(event["after"])
elif op == "d":
# delete -- 'after' is null, 'before' still has the last known row
delete_from_search_index(event["before"]["order_id"])
else:
raise ValueError(f"unexpected op: {op}"){"order_id": 4821}. This is what makes Debezium topics naturally compactable (Part 07) and what guarantees every change to the same row lands on the same partition, preserving per-row ordering exactly the way keyed partitioning does for any Kafka producer.Before Streaming Begins, Debezium Snapshots Existing Data — And That Phase Needs Care
When a Debezium connector starts for the first time against a table, the WAL or binlog only contains changes going forward from the moment the connector's replication slot was created — it has no record of the rows that already existed in the table before that point. To give consumers a complete picture, Debezium first performs a snapshot phase: it reads the table's current contents in full, producing one op: "r" (read) event per existing row, and only switches to streaming live changes from the log once the snapshot completes.
Phase 1 — Snapshot (once, at connector startup)
Debezium runs a consistent read of every row currently in the
orders table. For 2 million existing rows, that is 2 million
"op": "r" events produced to the topic, each with only an
"after" (the current row), no "before".
Phase 2 — Streaming (continuous, from here on)
Debezium switches to reading the WAL/binlog from the exact log
position that was consistent with the snapshot's starting point,
so no change is missed and no change is double-counted between
the two phases. From here, every insert/update/delete produces
a "c"/"u"/"d" event with full before/after state.
The switch from snapshot to streaming is seamless from a
consumer's point of view -- both phases write to the same topic,
using the same envelope shape, just with different "op" values.Operational care for large tables
A snapshot of a 2-million-row table is 2 million Kafka records produced in a short burst — real load on both the source database (a full table read) and the target Kafka cluster (a spike in write throughput on the destination topic). For very large tables, this deserves the same operational attention as any large backfill: running it during a lower-traffic window, watching the source database's read replica lag if the snapshot is taken from a replica, and being aware that a snapshot in progress delays the point at which the connector starts surfacing genuinely live changes.
| snapshot.mode value | Behavior |
|---|---|
| initial (default) | Snapshot existing data once on first startup, then stream. If the connector restarts later, it resumes streaming from its saved offset — it does not re-snapshot. |
| initial_only | Take the snapshot, then stop — useful for a one-time historical load into a topic without ongoing streaming. |
| never | Skip the snapshot entirely and only stream changes from the point the connector starts — consumers get no historical data, only what changes going forward. |
| when_needed | Debezium decides based on whether it has a valid saved offset to resume from — snapshots automatically if it does not. |
snapshot.mode: "never", expecting to save time, often does not realize until later that every row that existed in the table before the connector started is simply absent from the topic — only rows changed after that moment ever appear. If downstream consumers need a complete picture of current state, not just a feed of future changes, the snapshot phase is not optional.Topic-Per-Table Conventions, and Why CDC Topics Are Natural Candidates for Log Compaction
Debezium follows a topic-per-table convention by default: {topic.prefix}.{schema}.{table}. For the FreshCart example in Part 04, that produces freshcart.pg.public.orders. This mirrors the naming discipline covered for Connect topics generally in Module 13, and matters even more for CDC, because a single Debezium connector configured against a whole schema can spin up dozens of topics automatically — one per table — with no manual topic-creation step required.
Why CDC topics are a natural fit for compaction
Recall from the message-brokers module that log compaction retains only the latest value per key, forever, rather than deleting old segments by age. A Debezium topic's key is the source table's primary key (Part 05), and every event for a given row is a complete new snapshot of that row's current state in after. This means a compacted Debezium topic is, structurally, a changelog of the table — replaying it from the beginning reconstructs the table's current full state, row by row, exactly the changelog-table pattern the message-brokers module introduced conceptually.
# freshcart.pg.public.orders, compacted, key = order_id
# Raw log (before compaction):
offset 0: key=4821 op=c after={status: pending, ...}
offset 1: key=4822 op=c after={status: pending, ...}
offset 2: key=4821 op=u after={status: paid, ...}
offset 3: key=4821 op=u after={status: shipped, ...}
# After compaction, only the latest offset per key survives:
offset 1: key=4822 op=c after={status: pending, ...}
offset 3: key=4821 op=u after={status: shipped, ...}
# A new consumer reading from the beginning of this compacted
# topic sees the CURRENT state of every order that has ever
# existed -- a live, rebuildable copy of the orders table,
# without querying Postgres at all.
# A delete (op=d, after=null) becomes a tombstone once compacted
# -- the row's key is removed from the reconstructed state entirely.Not every Debezium topic should be compacted, though. If downstream consumers genuinely need the full history of intermediate updates — an audit log, a fraud model training on the sequence of state transitions — time-based retention, not compaction, is the right policy, since compaction is specifically designed to discard everything except the latest value per key.
Replication Slot Growth — The Single Most Dangerous Operational Failure Mode in Postgres CDC
Part 03 described the replication slot Postgres creates for Debezium as the mechanism that lets Debezium resume exactly where it left off. That same mechanism has a sharp edge: Postgres will not reclaim (garbage-collect) any portion of the WAL that a replication slot has not yet acknowledged as consumed — no matter how much disk space that requires. If Debezium falls behind, disconnects, or is deleted incorrectly while its slot still exists, Postgres keeps every WAL segment since the slot's last acknowledged position, indefinitely.
t=0 Debezium connector "source-postgres-orders-prod" running
normally, slot "debezium_freshcart_orders" actively
acknowledging WAL positions as it streams
t=1 day The Connect cluster the connector runs on is
decommissioned as part of an unrelated migration.
Nobody explicitly stops or drops the connector first --
it simply stops running.
t=1 day+ The replication slot "debezium_freshcart_orders" still
EXISTS on the Postgres server. Postgres does not know
the connector is gone -- from its point of view, a
consumer might reconnect and resume at any moment, so
it keeps every WAL segment since the slot's last
acknowledged position, exactly as designed.
t=3 days Postgres's WAL directory has grown by tens of GB and is
approaching the disk's capacity. Every table on this
Postgres instance is now at risk -- not just orders.
t=3 days+ If disk fills completely, Postgres refuses new writes
cluster-wide. This is now a production outage for every
application using this database, caused entirely by an
orphaned replication slot nobody remembered to drop.This is arguably the single highest-stakes operational fact in this entire module: a replication slot is not free, and it is not self-cleaning. It is a standing commitment from Postgres to retain WAL until someone tells it otherwise, and "someone" has to be a human or an automated process that explicitly drops the slot — Debezium disappearing does not drop its own slot.
Mitigations
- ✓Monitor replication slot lag directly in Postgres — pg_replication_slots exposes how far behind each slot is, in WAL bytes, and this should be an alerted metric, not something checked manually after an incident.
- ✓Always explicitly delete the Debezium connector (DELETE /connectors/{name}) before decommissioning the infrastructure it runs on — deleting the connector is what triggers Debezium to drop its own replication slot cleanly.
- ✓Set a heartbeat.interval.ms on tables that change infrequently — a table with no writes for hours means Debezium has no new WAL activity to acknowledge, which can itself stall slot advancement; a periodic heartbeat message gives Debezium something to acknowledge against even on quiet tables.
- ✓Alert on slot existence itself, not just lag — an orphaned slot that nobody is consuming from produces zero lag readings if lag is measured only against an active consumer, so a stale-slot check (a slot with no active connection for an extended period) catches what a lag-only alert would miss.
The Outbox Pattern — Debezium as the Real Fix for the Dual-Write Problem
The event-driven-architecture module covered the dual-write problem: a service that must both update its own database and publish an event describing that update has no way to make both operations atomic using two separate systems (a database transaction and a Kafka producer call) — a crash between the two leaves them inconsistent, either a committed database change with no published event, or a published event for a database change that never actually committed.
The outbox pattern solves this by turning a two-system problem into a one-system problem, and CDC is what makes it work in practice. Instead of writing to the database and separately calling a Kafka producer, the service writes both the business data change and a row describing the event — into an outbox table — in the same database transaction. A Debezium connector then captures changes to that outbox table exactly like any other table, turning each outbox row into a Kafka event. Because the outbox write and the business data write are one atomic database transaction, and Debezium is guaranteed to eventually capture every committed row via the WAL, the dual-write problem disappears — there is only ever one system (the database) that needs to commit atomically.
-- Inside ONE Postgres transaction, from the orders service:
BEGIN;
UPDATE orders SET status = 'paid' WHERE order_id = 4821;
INSERT INTO outbox (
aggregate_type, aggregate_id, event_type, payload
) VALUES (
'Order', '4821', 'OrderPaid',
'{"order_id": 4821, "total_cents": 4599, "paid_at": "..."}'
);
COMMIT;
-- Both rows commit together or neither does. There is no window
-- where the order is marked paid but no event was ever recorded,
-- and no window where an event exists for an update that rolled back.
-- A Debezium connector configured against the outbox table (using
-- Debezium's dedicated "outbox event router" SMT, which unwraps
-- the outbox row into a clean event on a topic named after
-- aggregate_type) captures each new outbox row exactly like any
-- other insert, and produces it to a Kafka topic -- with the same
-- at-least-once, near-zero-source-load guarantees as any other
-- Debezium-captured table.| Approach | Atomicity of write + publish | Failure mode |
|---|---|---|
| Direct dual write (DB write, then producer.send()) | None — two independent systems, two independent points of failure | Crash between the two leaves them inconsistent — the classic dual-write problem |
| Outbox table + Debezium | Full — both rows are one database transaction | None specific to this pattern; ordinary Debezium operational risks (Part 08) still apply |
Schema Evolution in CDC Topics — The Source Schema Changes Whether the Topic Is Ready or Not
A hand-written producer only ever writes a schema its own application code was deployed with — the team controls both when the schema changes and when the producer deploys. A Debezium connector has no such control: the moment a database migration runs ALTER TABLE orders ADD COLUMN discount_cents INTEGER, the very next captured row includes that new field, whether or not any downstream consumer or Schema Registry compatibility check was prepared for it.
This makes the converter and Schema Registry discipline from Module 13 more important for Debezium topics than almost any other kind of Connect pipeline. With an Avro converter and Schema Registry configured — as in the Part 04 configuration — a genuinely breaking schema change (removing a required field, changing a field's type incompatibly) is rejected at registration time, before it reaches the topic and breaks every downstream consumer simultaneously. Without that enforcement, a routine database migration becomes a silent, cluster-wide breaking change for every Debezium topic consumer at once.
| Database change | Effect on the Debezium event without schema enforcement |
|---|---|
| Add a nullable column | New field appears in after with a null or default value for older rows — usually safe, but still a schema change worth registering explicitly |
| Add a NOT NULL column with no default | Every subsequent event includes the new required field — consumers built against the old schema may fail to deserialize |
| Rename a column | Looks like the old field disappearing and a new field appearing simultaneously — a genuinely breaking change for any consumer reading field names directly |
| Change a column's type (e.g. INTEGER to VARCHAR) | Breaking for any consumer that deserializes strictly against the old Avro schema |
Monitoring a Debezium Deployment — What to Watch Beyond Ordinary Connect Task Health
Module 13 already covered the baseline for monitoring any Kafka Connect source connector — task state (RUNNING/PAUSED/FAILED), poll and write rates, and DLQ produce rates. Debezium connectors need everything from that baseline plus a handful of CDC-specific signals that a generic JDBC source connector does not have, because Debezium's failure modes are shaped by the replication mechanism covered in Part 03 and Part 08.
| Metric | What it tells you | Why it is specific to log-based CDC |
|---|---|---|
| Replication slot lag (pg_replication_slots.confirmed_flush_lsn vs. current WAL position) | How far behind the connector is in acknowledging WAL — the direct early-warning signal for Part 08's disk-growth failure mode | A JDBC source connector has no equivalent server-side resource being held open on its behalf; this metric only exists because of the replication slot mechanism |
| MilliSecondsBehindSource (a Debezium-exposed JMX metric) | How far behind, in wall-clock time, the connector's streaming position is relative to the most recent committed database transaction | Gives a time-based view of lag that is often more actionable for on-call than a raw LSN or byte-offset number |
| Snapshot progress / RemainingTableCount during an initial snapshot | How much of a large-table snapshot (Part 06) is left to complete | Only relevant during the one-time snapshot phase — irrelevant once a connector is fully in streaming mode |
| NumberOfEventsFiltered / NumberOfDisconnects | How often events are being filtered by include/exclude lists, and how often the connector's database connection is dropping and reconnecting | Frequent disconnects are a leading indicator of the network or database instability that can also cause replication slot lag to grow |
The single highest-leverage alert to add on top of Module 13's baseline is replication slot lag specifically, alerted well before it reaches a level that threatens disk capacity — not just when the connector's task itself transitions to FAILED. A connector can be technically RUNNING while steadily falling behind the WAL, which is exactly the silent, slow-building failure mode Part 08 describes; a task-state-only alert will not catch it until the underlying database is already in serious trouble.
-- Run on the Postgres primary, on a schedule, alert on the result:
SELECT
slot_name,
active,
pg_size_pretty(
pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)
) AS retained_wal_size
FROM pg_replication_slots
WHERE slot_name LIKE 'debezium_%';
-- Alert conditions:
-- active = false AND retained_wal_size > 0
-- -> an orphaned slot with nobody consuming from it, per Part 08
-- retained_wal_size growing steadily over a rolling window
-- -> a connected-but-lagging connector, falling behind the WAL
-- retained_wal_size approaching a disk-capacity threshold
-- -> the emergency case: act now, before Postgres refuses writesThe Postgres Story Generalizes, With Database-Specific Wrinkles Worth Knowing
Everything covered so far — the envelope shape, the snapshot phase, the outbox pattern, the topic-per-table convention — applies the same way regardless of which database Debezium is capturing from. But the specific mechanism each database uses to expose its change log, and the specific operational risks that come with it, differ enough to be worth knowing explicitly rather than assuming Postgres's exact failure modes transfer one-for-one.
MySQL — the binlog, and why ROW format is non-negotiable
MySQL's binary log can be configured in one of three formats: STATEMENT (records the SQL statement executed), ROW (records the actual row-level before/after data), or MIXED. Debezium requires ROW format specifically, because STATEMENT format only records "UPDATE orders SET status = 'paid' WHERE order_id = 4821" — the statement itself, not the row's resulting data — which is not enough information to construct the before/after envelope Part 05 describes, especially for a statement that could affect many rows non-deterministically (an UPDATE with a subquery, or one relying on auto-increment values assigned at execution time).
-- Check current binlog format:
SHOW VARIABLES LIKE 'binlog_format';
-- If it returns STATEMENT or MIXED, Debezium cannot reliably
-- reconstruct row-level before/after state. This must be set to
-- ROW before a Debezium MySQL connector is deployed:
SET GLOBAL binlog_format = 'ROW';
-- (Typically set in my.cnf for a permanent, restart-safe change
-- rather than only at the session level)
-- MySQL binlog retention also needs explicit attention -- similar
-- in spirit to Part 08's Postgres WAL retention concern, but
-- governed by expire_logs_days / binlog_expire_logs_seconds rather
-- than a replication-slot mechanism. Too short a retention window
-- risks the binlog Debezium needs being purged before it catches up
-- after an outage; too long wastes disk on an otherwise healthy setup.SQL Server — CDC must be explicitly enabled per table
Unlike Postgres's WAL or MySQL's binlog, which exist by default as part of normal database operation, SQL Server's Change Data Capture feature is off by default and must be explicitly enabled, both at the database level and per individual table Debezium needs to capture. This is an extra deployment step MySQL and Postgres do not require, and it is a common source of "the connector is running but producing nothing" confusion — the connector can be correctly configured and still capture zero changes if CDC was never turned on for the specific table in SQL Server itself.
-- Enable CDC at the database level (once per database):
EXEC sys.sp_cdc_enable_db;
-- Enable CDC for a specific table (once per table Debezium needs):
EXEC sys.sp_cdc_enable_table
@source_schema = N'dbo',
@source_name = N'orders',
@role_name = NULL;
-- Only after both of these succeed does SQL Server begin
-- populating the change tables Debezium's SQL Server connector
-- actually reads from -- the Debezium connector config itself has
-- no way to turn this on from the Kafka Connect side.| Database | Prerequisite before Debezium can capture anything | Retention/cleanup concern to monitor |
|---|---|---|
| PostgreSQL | Logical replication enabled (wal_level=logical) and a replication slot created | Replication slot WAL retention — Part 08's central risk |
| MySQL | binlog_format=ROW, binary logging enabled | Binlog expiry window (expire_logs_days) — too short risks data loss if Debezium falls behind |
| SQL Server | CDC explicitly enabled at database and table level | SQL Server's own CDC cleanup job retention window for change tables |
When Debezium Is the Right Tool, and When It Genuinely Isn't
Not every team that needs to move database changes into Kafka should deploy Debezium themselves. It is worth being explicit about the alternatives and where each one actually fits, rather than treating Debezium as the automatic answer to every CDC requirement.
Hand-rolled CDC — reading the log yourself
It is technically possible to write a custom application that reads a database's WAL or binlog directly, without Debezium, using the same underlying replication protocols Debezium itself uses. This is rarely the right choice: it means re-implementing snapshot handling, offset tracking, schema mapping, and the specific quirks of each database's logical decoding format from scratch — exactly the repetitive, error-prone integration work the Kafka Connect module's Part 01 described Connect itself as existing to eliminate. Hand-rolled CDC is occasionally justified for a database engine Debezium does not support at all, or a highly specialized capture requirement no existing connector covers — but it should be a deliberate last resort, not a default.
Fully managed CDC-as-a-service — Fivetran, Airbyte, and similar
Vendors like Fivetran (fully managed, closed-source) and Airbyte (open-source with a managed cloud offering) provide CDC as a hosted product, typically aimed at moving data directly into a data warehouse rather than into a Kafka topic a team then consumes with its own applications. These tools solve a meaningfully different problem than Debezium-on-Kafka-Connect: they are optimized for "get database changes into Snowflake/BigQuery/Redshift with minimal setup," not "give me a real-time, replayable Kafka topic that many independent internal consumers can build applications against."
| Approach | Best fit | Trade-off |
|---|---|---|
| Debezium on Kafka Connect (this module) | Real-time, replayable Kafka topics feeding multiple independent internal services and applications — the outbox pattern, event-driven architectures, stream processing | Requires operating Kafka Connect (or a managed Connect offering) and understanding the operational risks in Part 08 and Part 11 |
| Fivetran / Airbyte-style managed CDC | Getting database changes into a data warehouse with minimal engineering setup, when Kafka itself is not otherwise part of the architecture | Not designed as a general-purpose event bus for multiple internal application consumers — it is a warehouse-loading tool, not a Kafka topic producer, in its primary use case |
| Hand-rolled log reader | A database engine with no existing Debezium connector, or a narrow, unusual capture requirement | Reimplements snapshot handling, offset tracking, and log-decoding logic that Debezium has already solved and battle-tested |
Validating a Debezium Connector Before It Touches a Production Database
Because Debezium sits directly on top of a production database's replication mechanism, a misconfigured connector is not a purely Kafka-side risk the way a misconfigured hand-written producer would be — it can hold open a replication slot against a production database (Part 08) or run an unplanned full-table snapshot (Part 06) against live infrastructure. A short, deliberate validation pass before first deploying against production data catches most of the mistakes this module has covered.
- ✓Validate against a staging or replica database first — confirm the connector configuration, table include/exclude lists, and SMT chain (Part 04) produce the expected envelope shape before pointing at production.
- ✓Deliberately test the delete path — insert, update, then delete a test row, and confirm the resulting op="d" event and consumer-side handling (Part 05) behave as expected; the create and update paths are far more commonly tested than delete, and delete handling is where Part 05's Error Library entry shows up most often.
- ✓Confirm snapshot.mode is set deliberately, not left at a default assumed without checking — verify whether the pipeline actually needs the initial snapshot (Part 06) before the first production deployment, since this is a one-time decision that is awkward to reverse after the fact.
- ✓Set up the replication slot lag alert (Part 11) before, not after, the first production deployment — this is the single highest-consequence monitoring gap to close early, given Part 08's failure mode.
- ✓Load-test the initial snapshot against a production-sized (or realistically sized) copy of the table, if the real table is large — to understand the actual snapshot duration and throughput impact before it happens against the real production table.
# 1. Insert a row, confirm op="c" arrives with the expected 'after'
INSERT INTO orders (order_id, status) VALUES (999001, 'pending');
# 2. Update it, confirm op="u" arrives with correct 'before' and 'after'
UPDATE orders SET status = 'paid' WHERE order_id = 999001;
# 3. Delete it, confirm op="d" arrives with 'after' null and
# 'before' populated -- this is the path most often skipped
DELETE FROM orders WHERE order_id = 999001;
# 4. Confirm the consumer under test correctly upserts on c/u/r
# and deletes on d, per the Part 05 worked example -- not just
# that events arrive, but that downstream handling is correctFive Misconceptions About CDC and Debezium
What This Looks Like on Day One
At Confluent (the company that employs many of Kafka's original creators, and increasingly the primary commercial sponsor of Debezium's ecosystem tooling): a customer's support ticket describes their production Postgres disk filling up over a weekend, no application deploy, no obvious cause. Confluent's field engineering team's first diagnostic question, before looking at anything else, is: "do you have an orphaned replication slot?" It is common enough — teams decommissioning old Connect clusters without first deleting the Debezium connectors running on them — that it is one of the first items on their standard CDC incident runbook, precisely because Part 08's failure mode is not theoretical; it is one of the most common real support escalations in CDC deployments across the industry.
At Fivetran (a data-integration company whose product overlaps directly with what this module teaches — moving database changes reliably into downstream systems):an engineer is reviewing a new customer's Postgres-to-warehouse pipeline design and flags that the customer's schema includes several tables with frequent column renames as part of an ongoing internal refactor. The engineer explains, in terms directly out of Part 10, that a column rename looks like a field disappearing and a new one appearing simultaneously in the captured change stream — and recommends the customer either freeze renames on CDC-tracked tables during active migration windows, or explicitly coordinate them with every downstream consumer, the same discipline any breaking API change would require.
At Airbnb-scale internal tooling, or any company running its own CDC-to-search-index pipeline (the kind of infrastructure Airbyte, an open-source data integration platform, explicitly builds connectors to replace hand-rolled versions of): an incident review finds that a search index has been silently missing cancelled listings for weeks. The root cause: the original pipeline was a polling job, not CDC, and cancellations were implemented as hard deletes on the source table — exactly Part 02's Failure 1. The fix that gets shipped is a Debezium connector against the listings table, with the outbox pattern from Part 09 used for a separate, related event (listing status changes that also need to trigger email notifications) so that both the search index and the notification system are fed from the same reliable, delete-aware change stream instead of two separately fragile polling jobs.
5 Interview Questions — With Complete Answers
Mistakes Beginners Make Constantly
Errors You Will Hit — And Exactly Why They Happen
🎯 Key Takeaways
- ✓CDC captures every row-level insert, update, and delete as an individual event, in order — polling only captures net differences observed at a fixed interval, silently missing deletes and collapsing intermediate updates.
- ✓Debezium reads the database's own write-ahead log (Postgres via pgoutput) or binary log (MySQL) directly, rather than querying tables — this is what gives it near-zero impact on the source database and complete delete visibility.
- ✓Debezium runs as a Kafka Connect source connector in distributed mode, configured through the same REST API, sharing the offset-tracking, rebalancing, and error-handling model covered for Connect generally.
- ✓Every Debezium event has the same envelope: before, after, source, and op ("c"/"u"/"d"/"r"). Consumers should branch explicitly on op — upserting on c/u/r, deleting on d using the before state since after is null.
- ✓A connector first performs a one-time snapshot of existing table data (op="r" events), then switches to streaming live changes from the log — large-table snapshots deserve the same operational care as any large backfill.
- ✓The outbox pattern — writing business data and an event description in the same database transaction, then letting Debezium capture the outbox table — is the standard, reliable fix for the dual-write problem.
- ✓A Postgres replication slot is retained by the database until explicitly dropped. An orphaned or lagging slot causes unbounded WAL growth that can fill disk and take down writes for every table on the instance — monitor slot lag and slot existence directly.
- ✓A Debezium topic's schema tracks the source table's schema automatically and immediately — a column rename or type change is a breaking change for every downstream consumer the moment the migration runs, and deserves the same coordination as a breaking API change.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.