Event-Driven Architecture

Event-Driven Architecture Is a Communication Style, Not a Kafka Feature

Every module so far has treated Kafka as infrastructure — brokers, partitions, producers, consumers, replication. This module steps back and asks a different question: what does it actually mean to design a system around Kafka, rather than just point a few services at a cluster? The answer is event-driven architecture (EDA) — a system design style where services communicate by publishing facts about things that happened, and other services independently react to those facts, rather than one service directly calling another and waiting for a response.

The contrast is with request-response, the style most engineers learn first: Service A calls Service B's API, B does some work, and B returns a response that A waits for. In EDA, Service A publishes an event — order.placed, payment.charged,inventory.reserved — to a topic, and does not know or care which services, if any, are listening. Zero, one, or ten services might react. A does not wait for any of them, and A's code never changes when a new reactor is added.

This is not a Kafka-specific idea — EDA predates Kafka and can be built on many different technologies. But Kafka is an unusually good fit for it, because a durable, replayable, multi- subscriber topic is exactly the primitive EDA needs: publishers don't need to know who's listening, new subscribers can join at any time and even replay history, and the broker itself absorbs the timing mismatch between a fast publisher and a slow (or temporarily offline) subscriber — the temporal, spatial, and rate decoupling covered in earlier modules is precisely what makes EDA practical at scale rather than just a nice diagram.

The Real Trade-Offs — Decoupling and Resilience vs Consistency and Debuggability

Choosing EDA over request-response is a genuine trade-off, not a strict upgrade. Understanding both sides of the trade is what separates a thoughtful architecture decision from cargo-culting "everything should be event-driven" because it sounds more modern.

What EDA buys you

• ✓Decoupling — the order service does not need to know the notification service, the fraud-detection service, or the analytics pipeline exist. New consumers can be added with zero changes to the producer, exactly as covered for pub-sub topics in earlier modules.
• ✓Independent scaling — each consuming service scales based on its own load characteristics, not tied to the producer's request volume or any other consumer's processing speed.
• ✓Resilience to a downstream service being down — if the notification service is deployed, crashed, or being restarted, order placement still succeeds. The event sits durably in the topic and the notification service catches up whenever it comes back, instead of the order API itself failing or timing out because a downstream dependency was unavailable.

What it costs you

• ✓Eventual consistency — there is a real, non-zero window between "order placed" and "inventory actually reserved" during which the two facts are out of sync. A request-response system that synchronously reserves inventory before confirming the order has no such window, at the cost of the order API now depending on the inventory service being up and fast.
• ✓Harder end-to-end debugging and tracing — in request-response, a stack trace or a single trace ID often shows you the whole call chain. In EDA, "why didn't the confirmation email send?" can require tracing through several independently-deployed consumers, each polling a topic on its own schedule, with no single synchronous call chain to follow.
• ✓No implicit response — the publisher genuinely does not know if a consumer succeeded, failed, or even exists. Any "did this work?" signal has to be built deliberately, usually as another event, not assumed for free the way a request-response caller gets a response by construction.

Thin Events vs Fat Events — Two Different Event Payload Shapes

Once a team decides to communicate via events, there's a second design decision that matters just as much: how much information does the event itself carry? This is the distinction between event notification and event-carried state transfer, and it directly shapes how tightly consumers end up coupled to the publisher.

Event notification — a thin "something happened, go fetch details" signal

An event-notification style event carries the bare minimum: an identifier and a fact. The consumer, on receiving it, makes a separate call back to the source service (or its API) to fetch whatever additional detail it actually needs.

{
  "event_type": "order.updated",
  "order_id": "ORD-88213",
  "occurred_at": "2026-09-11T14:22:03Z"
}

# The consumer, on receiving this, calls:
# GET /api/orders/ORD-88213
# to find out WHAT changed -- shipping address? item quantity?
# order status? The event itself doesn't say.

Event-carried state transfer — a fat event with everything a consumer needs

An event-carried-state-transfer style event embeds the full relevant state directly in the event payload. The consumer can act on it immediately, with no follow-up call to anyone.

{
  "event_type": "order.updated",
  "order_id": "ORD-88213",
  "occurred_at": "2026-09-11T14:22:03Z",
  "customer_id": "CUST-4471",
  "status": "shipped",
  "previous_status": "processing",
  "shipping_address": {
    "line1": "482 Market St", "city": "Austin", "state": "TX", "zip": "78701"
  },
  "line_items": [
    { "sku": "SKU-1029", "quantity": 2, "unit_price_cents": 1499 },
    { "sku": "SKU-2201", "quantity": 1, "unit_price_cents": 3999 }
  ],
  "total_cents": 6997
}

# The consumer has everything it needs right here -- no follow-up
# API call to the order service required at all

Most production systems land somewhere in between rather than at either extreme: include enough state in the event that the overwhelmingly common consumer use case never needs a follow-up call (the fields a notification service, an analytics pipeline, and a search-index updater all actually use), while accepting that a rare consumer with an unusual need might still make an occasional direct call back to the source service for something the event genuinely doesn't carry.

The Dual-Write Problem — And the Outbox Pattern That Solves It

Here is a problem every team building EDA on top of a service with its own database eventually hits. A service that processes an order needs to do two things when an order is placed: write the order to its own database (so the service itself has a record), and publish an order.placed event to Kafka (so other services find out). These are two separate systems — a database and a Kafka cluster — with no shared transaction between them. This is the dual-write problem.

def place_order(order):
    db.save(order)                      # write #1: the database
    kafka_producer.send("order.placed", order)  # write #2: Kafka

# What if the process crashes between these two lines?
# -> order is saved to the database, but the event was NEVER published
# -> every downstream service (inventory, notifications, analytics)
#    has no idea this order exists

# What if the database write succeeds but the Kafka send fails
# (broker temporarily unreachable, producer buffer full)?
# -> same outcome: silent, permanent inconsistency between the
#    service's own database and the rest of the event-driven system

# What if you flip the order -- publish the event FIRST, then save to
# the database?
def place_order_flipped(order):
    kafka_producer.send("order.placed", order)  # write #1: Kafka
    db.save(order)                               # write #2: database

# Now a crash between the two lines means downstream services react
# to an order that was never actually saved -- inventory gets
# reserved for an order that doesn't exist in the source of truth

Neither ordering is safe, because a database write and a Kafka publish are not atomic together — there is no built-in mechanism spanning both systems that guarantees "both happen, or neither happens." The transactional outbox pattern solves this without needing a distributed transaction across two different technologies.

How the outbox pattern works

Instead of writing to the database and publishing to Kafka as two separate operations, the service writes to two tables in the same database, inside a single local database transaction — the orders table, and an outbox table that records "this event needs to be published." Because both writes are in the same database transaction, they are atomic with respect to each other using nothing more exotic than the database's own transaction guarantees. A separate process then reads new rows from the outbox table and actually publishes them to Kafka.

BEGIN TRANSACTION;

INSERT INTO orders (order_id, customer_id, status, total_cents)
VALUES ('ORD-88213', 'CUST-4471', 'placed', 6997);

INSERT INTO outbox (id, aggregate_type, aggregate_id, event_type, payload, created_at)
VALUES (
  gen_random_uuid(),
  'order',
  'ORD-88213',
  'order.placed',
  '{"order_id":"ORD-88213","customer_id":"CUST-4471","total_cents":6997}',
  now()
);

COMMIT;

# Both rows commit together, or neither does -- this is an ordinary
# ACID transaction inside ONE database, not a distributed transaction
# spanning the database and Kafka. No new coordination technology
# is needed to get this atomicity guarantee.

The remaining question is how rows land in Kafka from the outbox table. A naive approach — a background job that polls the outbox table on a timer and publishes new rows — works, but adds latency and constant polling load. The far more common production approach is to point Change Data Capture (CDC) at the outbox table specifically: a CDC connector like Debezium tails the database's own transaction log (the same mechanism covered in the next module) and streams new outbox rows to Kafka as they are committed, with very low latency and no polling overhead. This ties the outbox pattern directly into CDC — the outbox table becomes a clean, purpose-built source table for exactly the events a service intends to publish, rather than CDC-ing the entire orders table and exposing every internal column as a public event.

Choreography vs Orchestration — Who's in Charge of a Multi-Step Process

A single event is easy to reason about. The harder design question is a business process made of several steps, each potentially handled by a different service — placing an order, reserving inventory, charging a payment, creating a shipment. There are two fundamentally different ways to coordinate a multi-step process like this in an event-driven system: choreography and orchestration.

Choreography — each service reacts to the previous step's event

In choreography, there is no central coordinator. Each service knows only its own piece: it subscribes to the event that should trigger its work, does that work, and publishes its own event when done, which the next service in the chain subscribes to. The overall process emerges from the sum of these independent, uncoordinated reactions — no single service knows the whole flow.

order-service:    publishes order.placed
inventory-service: subscribes to order.placed
                   reserves inventory
                   publishes inventory.reserved
payment-service:   subscribes to inventory.reserved
                   charges payment
                   publishes payment.charged
shipping-service:  subscribes to payment.charged
                   creates shipment
                   publishes shipment.created

# No service holds a map of the whole process. Each one only knows:
# "when I see event X, I do my job, then I publish event Y."
# The full order-fulfillment flow only exists implicitly, as the
# sum of these independent reactions.

Orchestration — a central coordinator drives the process

In orchestration, a dedicated coordinator (often called a saga orchestrator) knows the entire process explicitly. It issues commands to each service in turn, waits for that step's result, and decides what happens next — including how to handle a failure partway through. Services in this model react to direct commands from the orchestrator rather than to each other's events.

order-fulfillment-orchestrator:
  step 1: send reserve_inventory command to inventory-service
          -> wait for inventory.reservation.result event
  step 2: if reserved, send charge_payment command to payment-service
          -> wait for payment.charge.result event
  step 3: if charged, send create_shipment command to shipping-service
          -> wait for shipment.creation.result event
  step 4: if any step fails, issue compensating commands to undo
          whatever already succeeded (covered in Part 06)

# The orchestrator holds the entire process definition in one place.
# Individual services don't need to know what happens before or
# after their own step -- they just respond to commands.

An Event Schema Is a Public API Contract Between Teams

In request-response systems, most teams instinctively treat a REST or gRPC API's request/response shape as a contract — you version it carefully, you don't remove fields without warning, you communicate breaking changes ahead of time. Event schemas deserve exactly the same discipline, and teams that skip this discipline usually learn why the hard way.

The reason is structural, not stylistic: once a team publishes an order.placed event with a certain schema, every consumer of that topic — inventory, payments, shipping, analytics, fraud detection, and any team that adds a new consumer six months from now — has built code that depends on that exact shape. Unlike a REST API, where the publisher can see exactly who's calling it (every caller has to authenticate and hit a known endpoint), a Kafka topic's publisher generally has no visibility into who is consuming it. A breaking schema change doesn't fail loudly at the publisher; it silently breaks an unknown number of consumers, possibly ones the publishing team doesn't even know exist.

# order.placed schema, version 1:
{ "order_id": "string", "customer_id": "string", "total_cents": "integer" }

# A well-meaning engineer renames total_cents to total_amount_cents
# for clarity, and ships it without checking who consumes this topic

# order.placed schema, version 2 (BREAKING, deployed without warning):
{ "order_id": "string", "customer_id": "string", "total_amount_cents": "integer" }

# Every consumer expecting "total_cents" now either:
# - throws a deserialization/field-access error and stops processing
# - or silently reads total_cents as missing/null and proceeds with
#   wrong data (arguably worse -- a silent correctness bug instead
#   of a loud failure)

# The publishing team may not even know inventory-service,
# fraud-detection-service, and the BI team's nightly export job
# all depend on this exact field name

The fix is the same schema evolution discipline covered for serialization formats elsewhere in this track: add new fields as optional with sensible defaults rather than renaming or removing existing ones, use a schema registry that enforces compatibility rules (backward, forward, or full compatibility) at publish time so an incompatible change is rejected before it ever reaches the topic, and treat any genuinely breaking change as a new event type or a new topic version rather than a silent mutation of the existing one.

Designing EDA for Order Fulfillment — A Worked Example

Bringing the whole module together: an order-fulfillment flow is a textbook event-driven design problem. The happy path is simple to describe. The value of doing this exercise carefully is in the failure paths — specifically, what happens when a step fails after an earlier step has already taken an action that needs to be undone.

The happy path

1. order-service:      customer places order
                       -> publishes order.placed
2. inventory-service:  subscribes to order.placed
                       reserves the ordered items
                       -> publishes inventory.reserved
3. payment-service:    subscribes to inventory.reserved
                       charges the customer's payment method
                       -> publishes payment.charged
4. shipping-service:   subscribes to payment.charged
                       creates a shipment record, notifies the warehouse
                       -> publishes shipment.created
5. notification-service: subscribes to shipment.created (and, separately,
                       to order.placed for an initial confirmation email)
                       sends the customer a shipping confirmation

What could go wrong — the payment-fails-after-inventory-reserved case

Here is the scenario that makes the design interesting: inventory was successfully reserved (step 2 above completed and published its event), but the payment charge in step 3 fails — the customer's card is declined, or the payment processor times out. Inventory has already been taken out of available stock for other customers to buy, but the order this reservation was for is not actually going to complete. Left alone, this is a data-correctness bug: reserved inventory with no corresponding successful order, quietly reducing available stock forever.

This is exactly what the saga pattern — and specifically, compensating actions — exists to solve. A saga is a sequence of local transactions where each step publishes an event on success, and if any step fails, the system runs compensating actions that semantically undo the effects of the steps that already succeeded, since there is no cross-service distributed transaction to simply roll back.

1. order-service:      order.placed
2. inventory-service:  inventory.reserved   (items removed from available stock)
3. payment-service:    attempts to charge the card -- DECLINED
                       -> publishes payment.failed (NOT payment.charged)

4. inventory-service:  subscribes to payment.failed
                       runs its COMPENSATING ACTION: releases the
                       reservation, returning items to available stock
                       -> publishes inventory.reservation.released

5. order-service:      subscribes to payment.failed
                       updates the order's own status to "payment_failed"
                       (its own local, compensating state change)

6. notification-service: subscribes to payment.failed
                       sends the customer a payment-failed notice,
                       distinct from the earlier order-confirmation email

# Nothing here is a single atomic rollback across services -- there
# is no such mechanism. Instead, EVERY step that has a real-world
# side effect (reserving inventory, charging a card) is paired with
# an explicit compensating action (releasing the reservation,
# refunding a charge) that a service runs in response to a
# downstream failure event.

Choreography vs orchestration for this specific flow

With only four steps and one failure path, choreography (as shown above) is a reasonable choice — each service reacts to the relevant event and knows its own compensating action. But notice the scattering already starting: inventory-service needs to know about bothorder.placed (do the work) and payment.failed (undo the work), and that logic lives entirely inside inventory-service with no single place to see the whole process. Add a few more steps — a fraud check, a multi-warehouse routing decision, a partial-shipment path — and the number of failure events every service must individually understand grows quickly. This is the tipping point from Part 05 where many teams migrate this same flow to an explicit saga orchestrator: one service that owns the entire process definition, including every compensating action, in one place.

Tracing a Business Process Across Services That Never Call Each Other Directly

Part 02 named "harder end-to-end debugging" as one of the real costs of EDA, and it's worth treating that cost seriously enough to design around, rather than discovering it during an incident. In a request-response system, a single trace ID generated at the entry point typically flows through every synchronous call in the chain almost for free, because each call directly invokes the next. In an event-driven system, there is no such chain — each consumer polls its topic on its own schedule, and nothing automatically threads a single identifier through five independently-deployed services reacting to five different events, minutes or hours apart.

Correlation IDs — the minimum viable fix

The baseline practice is to generate a correlation ID (often the same ID as the originating business entity, like the order ID, or a dedicated trace ID) at the point the process begins, and propagate it through every event in the chain — not as an afterthought, but as a required field on every event schema in the flow. Every service that publishes a downstream event copies the correlation ID forward unchanged, regardless of what else changes in the payload.

{
  "event_type": "order.placed",
  "correlation_id": "trace-9f2e1a",
  "order_id": "ORD-88213",
  "...": "..."
}

# inventory-service publishes its own event, carrying the SAME
# correlation_id forward, even though the event_type and payload
# are completely different:
{
  "event_type": "inventory.reserved",
  "correlation_id": "trace-9f2e1a",
  "order_id": "ORD-88213",
  "...": "..."
}

# every log line and metric emitted by every service in this chain
# is tagged with correlation_id=trace-9f2e1a, so a single query
# across all services' logs reconstructs the full process timeline

What to actually monitor across an event-driven flow

The "started but never finished" signal deserves particular attention, because it's the failure mode EDA is uniquely prone to hiding. In request-response, a process that fails partway through usually surfaces as an error returned to the original caller. In EDA, if a step fails and nobody is watching for it, the process can simply stall — the correlation ID exists in the logs of the steps that did run, and nothing points at the fact that the final expected event never showed up.

EDA Anti-Patterns — Signs the Event-Driven Answer Is the Wrong One

Every module in this track has built a case for what Kafka and event-driven design do well. It's just as important to recognize the shapes of problem where reaching for an event, out of habit rather than fit, actively makes a system worse. These are the recurring anti-patterns worth watching for.

Anti-pattern: using events for something the caller needs an answer to right now

If a checkout flow publishes a coupon.validation.requested event and then polls, waits, or blocks for a coupon.validation.completed event to come back before it can render a price to the user, this is request-response wearing an event-driven costume — with all of EDA's added latency and complexity, and none of its actual benefit, since the caller still can't proceed until it gets a synchronous-shaped answer. The honest fix is usually a direct, synchronous API call for this specific interaction, even inside an otherwise event-driven system.

Anti-pattern: chatty, fine-grained events that recreate a call graph

Some teams, having adopted EDA, publish an event for every minor internal state change — a field-level update event for every single attribute change on an entity — and end up with dozens of consumers stitching these fine-grained events back together to reconstruct something that was really just one coherent business fact. This recreates the tight coupling EDA was meant to avoid, just expressed as event names instead of function calls, while paying the full overhead (schema management, topic proliferation, ordering complexity across related events) that coarse, well-designed events would have avoided.

Anti-pattern: an event as a disguised RPC call to exactly one consumer

If a topic genuinely has exactly one consumer, always has, and always will — the publisher and consumer are really a tightly coupled pair pretending to be decoupled — some of EDA's overhead (schema registry ceremony, replication, retention tuning) may not be buying anything a well-defined, versioned direct API call between the two services wouldn't have provided more simply. This is a judgment call, not a hard rule — a single-consumer topic today can become a multi-consumer topic tomorrow, and building it as an event from the start avoids a later migration. But it's worth naming honestly rather than assuming every interaction benefits from being an event by default.

Every Consumer in an Event Chain Needs to Assume It Will See an Event More Than Once

Earlier modules established that Kafka's default delivery guarantee is at-least-once — a consumer can, under real failure conditions (a crash between processing and committing an offset, a rebalance mid-batch), see the same event more than once. In a simple single-consumer pipeline this is a known, manageable property. In a multi-step event-driven chain like the order-fulfillment example in Part 07, this property compounds: a duplicate delivery at any single step can trigger a duplicate side effect that ripples forward into every step after it.

# inventory-service receives order.placed TWICE (rebalance mid-processing)
# without idempotency protection:

# first delivery:  reserves 2 units of SKU-4471
#                   publishes inventory.reserved
# second delivery: reserves ANOTHER 2 units of SKU-4471 (duplicate!)
#                   publishes inventory.reserved AGAIN

# payment-service, subscribed to inventory.reserved, now sees TWO
# events for the same order and charges the customer's card TWICE

# shipping-service, subscribed to payment.charged, now creates
# TWO shipment records for one order

# one duplicate delivery at step 1 became a customer double-charge
# and a duplicate shipment by the time it reached step 4

The fix is the same idempotent-consumer discipline covered for single-topic pipelines earlier in this track, applied deliberately at every step of a multi-step chain, not just the first one. Each service needs a way to recognize "I have already processed this specific event" — commonly by tracking a unique event ID (or the correlation ID plus event type) in its own database, checked before performing any side effect, so a duplicate delivery is detected and skipped rather than reprocessed.

def handle_order_placed(event):
    event_id = event["event_id"]

    if already_processed(event_id):
        log.info(f"Skipping duplicate event {event_id}")
        return  # side effect already happened -- do nothing

    with db.transaction():
        reserve_inventory(event["order_id"], event["line_items"])
        mark_processed(event_id)  # recorded in the SAME transaction
        # as the side effect, so a crash between them can't leave
        # the side effect done but unrecorded, or vice versa

    publish("inventory.reserved", {...})

Event-Driven Architecture Is Not Event Sourcing — A Common Conflation

These two terms get used almost interchangeably in casual conversation, and conflating them leads to real design confusion. They are related but distinct ideas, and it's worth being precise about the difference before moving into the next module, which relies on a clear notion of what an event represents.

Event-driven architecture, as this module has covered it, is about how services communicate— publishing and reacting to events instead of calling each other directly. It says nothing about how any individual service stores its own internal state. A perfectly ordinary service using EDA to communicate can still store its current state as a normal mutable row in a relational database, updated in place on every change, exactly like a non-event-driven service would.

Event sourcing is a different, stricter idea about how a service persists its own state: instead of storing current state directly, the service stores the full sequence of events that led to that state, and derives current state by replaying those events (or a snapshot plus the events since it). A service using event sourcing never overwrites its history — every change is an immutable, appended fact, and "current state" is a read-time projection of that history rather than something stored directly.

In practice, the two ideas pair naturally, which is exactly why they get conflated: a service that already stores its history as an event log (event sourcing) has a very short path to also publishing those same events outward for other services to react to (event-driven architecture) — and Kafka, as a durable, replayable log, is a reasonable fit for both roles at once. But a team can absolutely adopt one without the other, and assuming they're the same thing leads to over-engineering a simple service's internal storage model just because it happens to publish events, or under-designing a genuinely event-sourced service's schema evolution because the team is only thinking about it as "the events we send to other teams."

Who Owns an Event? Organizational Design for a Widely-Shared Topic

Part 06 established that an event schema is effectively a public API contract. That framing has an organizational consequence worth spelling out directly: someone has to own that contract the way a team owns a public REST API — with a clear point of contact, a defined change process, and accountability for the compatibility guarantees consumers are relying on. Without explicit ownership, a widely-consumed topic tends to drift toward whichever team touched it most recently having de facto, undocumented control over a contract many other teams depend on.

None of this requires heavyweight process for every topic — a narrowly-scoped, single-consumer topic reasonably gets much lighter governance than order.placed, which several teams across the company depend on. The judgment call is recognizing, as a topic's consumer list grows, when it has crossed from "an internal implementation detail one team controls" into "a shared contract that needs the same discipline a public API would get" — and building the governance in proportionally, rather than either over-processing every topic or under-governing the ones that matter most.

Five Misconceptions About Event-Driven Architecture

What This Looks Like on Day One

5 Interview Questions — With Complete Answers

The Mistakes That Undermine Event-Driven Designs

Errors You Will Hit — And Exactly Why They Happen