What is Apache Kafka?

Kafka Is Not "Just a Queue" — It Is a Durable Event Log

Apache Kafka is a distributed event streaming platform. That phrase sounds intimidating, so let us translate it carefully, one word at a time, the way you would explain it to a smart colleague who has never touched it. Distributed means Kafka runs across multiple servers instead of one machine — if one machine dies, the system keeps running. Event means a recorded fact that already happened: an order was placed, a payment succeeded, a package shipped, a driver changed location, a user clicked a button. Streaming means these facts arrive continuously, one at a time, second after second, not as one finished file dropped at the end of the day.

The simplest definition is this: Kafka is a system for storing ordered streams of events so many applications can read those events independently. Producers write events into Kafka. Kafka stores those events durably — on disk, replicated across machines, so a single disk failure or server crash does not lose them. Consumers read the events at their own speed, whenever they are ready. The same event can be read by billing, analytics, notifications, fraud detection, and customer support without the producer sending five separate copies to five separate destinations.

This is different from traditional request-response systems, which is where most engineers start their careers and where most of their intuition comes from. In request-response, one service asks another service to do something right now, over the network, and waits for an answer. If the other service is slow, that wait is slow. If the other service is down, the caller has a problem it must handle immediately — retry, fail, or queue up locally. Kafka changes this relationship entirely. A service records what happened and moves on. Other services react when they are ready. This separates the act of producing a fact from the act of consuming it, and that separation is the single idea underneath almost everything else in this module.

What a "Message" Actually Is, and Why Direct Calls Stop Scaling

Before going further, it is worth being precise about a word this whole field overuses: message. A message, in the broadest sense, is simply a unit of data sent from one program to another. An HTTP request is a message. A row inserted into a database and read by another process is, functionally, a message. A Kafka event is a specific, disciplined kind of message: an immutable record of something that already happened, carrying enough information that a reader who was not involved in producing it can still make sense of it later — possibly much later. Module 02 goes much deeper into the exact shape of an event (key, value, timestamp, headers). For now, hold the general idea: an event is a fact, not a request, and not a command.

Now consider why REST-style request-response APIs — the tool most backend engineers reach for first — break down once a system needs to notify many independent parts of itself, asynchronously, about the same fact. Imagine an online store before Kafka. The checkout service handles purchases. After every order, the checkout service must tell the billing service to charge the card, the warehouse service to pick items, the email service to send a receipt, the analytics system to update dashboards, the fraud system to inspect risk, and the support system to show the order to agents. If checkout does this with direct HTTP calls to each of those six services, checkout is now coupled to all six of their uptimes, all six of their response times, and all six of their API contracts.

# checkout-service, naive design: 6 direct HTTP calls after every order

def place_order(order):
    save_to_db(order)
    call_billing_api(order)        # what if billing is slow right now?
    call_warehouse_api(order)      # what if warehouse deploys and drops 2% of requests?
    call_email_api(order)          # what if email provider is rate-limiting us?
    call_analytics_api(order)      # what if analytics is down for maintenance?
    call_fraud_api(order)          # what if fraud added a new required field last week?
    call_support_api(order)        # what if support's database is under load?
    return "order placed"          # this response is now only as fast as the SLOWEST of the six calls

# If any ONE of the six is down or slow:
#   - checkout's own response time degrades or times out
#   - checkout must now implement retry logic for six different failure modes
#   - if checkout crashes mid-loop, some services got the update and some did not
#   - adding a 7th interested service means editing and redeploying checkout.py

At first this works. Then the system grows. Analytics goes down for maintenance and checkout's retry logic starts eating memory. Email becomes slow during a provider outage and orders start timing out. Fraud adds a new required API field and checkout breaks in production the next deploy. Warehouse needs its own retry semantics because picking is not idempotent. Support wants to replay the last 24 hours of orders after a data corruption bug, and there is no way to do that — the HTTP calls already happened and are gone. Suddenly checkout, which should focus on one job — placing orders correctly — has become tangled with the failure modes, deploy schedules, and API contracts of every downstream system that has ever asked it for data.

Kafka solves this by becoming the shared, asynchronous event backbone between systems. Checkout publishes one event — OrderPlaced — to one topic. Billing, warehouse, email, analytics, fraud, and support each consume OrderPlaced independently, on their own schedule, using their own logic. Checkout does not need to know who reads the event, how many readers there are, or whether they are healthy right now. Consumers do not need checkout to resend history if they were briefly offline — Kafka already held it for them. This is asynchronous, decoupled, cross-system communication, and it is the specific gap that request- response APIs cannot fill at scale.

The specific asynchronous, cross-system shape Kafka is built for

It is worth naming the pattern precisely, because it is the pattern you will keep encountering throughout this track: one producer, an unknown and possibly growing number of independent consumers, none of which need to respond synchronously, and none of which should block or slow down the producer. Direct calls scale linearly in pain with the number of consumers — every new consumer is a new outbound call, a new failure mode, a new piece of coupling in the producer's code. An event log scales flat — the producer writes once, no matter how many consumers eventually exist, and the producer's code never changes when a new consumer is added.

# Day 1: checkout-service publishes OrderPlaced. Two consumers exist.
producer.send(topic="orders", key=order_id, value=order_placed_event)
# consumers: billing-consumer-group, warehouse-consumer-group

# Month 6: five more teams have joined, all reading the SAME topic.
producer.send(topic="orders", key=order_id, value=order_placed_event)
# consumers now: billing, warehouse, email, analytics, fraud, support, loyalty-points
# checkout-service's code above did not change by a single character.

The Log: A Deceptively Simple Data Structure That Changes Everything

Before Kafka's vocabulary — topic, partition, broker — makes sense, you need to understand the single data structure everything is built on: the log. Not a log file in the "application logging" sense (though that is a related idea), but a log in the computer-science sense: an ordered, append-only sequence of records, where every record gets a position number, and new records are only ever added to the end.

This is one of the oldest and simplest ideas in computing — it is the same structure underneath a database's write-ahead log, a version control system's commit history, and an accountant's ledger. What makes it powerful is what it refuses to do: it never edits history in place, and it never removes an entry to make room for a read. Reading from a log does not consume it. This is the opposite of, say, popping an item off a stack or dequeuing a task — both of those operations destroy the item as part of reading it. A log just accumulates, and different readers can each be at a different position within it simultaneously, without interfering with one another.

orders partition 0

offset 0  -> OrderPlaced(order_id=100)
offset 1  -> PaymentAuthorized(order_id=100)
offset 2  -> OrderPlaced(order_id=101)
offset 3  -> ShipmentPrepared(order_id=100)
offset 4  -> OrderCancelled(order_id=101)

# Nothing here was ever edited or removed.
# A consumer does not "take" a record out of the log — it just reads it.
# A consumer's only state is a single number: "I have processed through offset 4."
# Three different consumers can each be at a different offset in this same log
# at the same time, and none of them affects what the others see.

Kafka is, at its core, this idea implemented at massive scale, distributed across many machines, made durable against hardware failure, and made fast enough to handle millions of records per second. Every other Kafka concept you will learn — topics, partitions, offsets, retention, compaction, replication — exists to answer one of two questions about this log: how do we split it up so it can grow arbitrarily large and be read/written in parallel (that is topics and partitions), and how do we keep it durable and available when machines fail (that is replication). Module 02 spends its entire length on the first question. This module gives you enough of the picture to keep moving.

It is worth pausing on why sequential appends are so much faster than the alternative, because this is not a minor implementation detail — it is the reason Kafka can sustain the throughput companies adopt it for. A disk (spinning or SSD) is fastest when it writes to consecutive locations, one after another, because the drive never has to jump elsewhere to find the next write location. Random writes — updating a record buried in the middle of a file, as a traditional database update often does — force exactly that kind of jumping around, which is dramatically slower. By restricting itself to append-only writes at the end of the log, Kafka sidesteps this problem by construction, not by clever optimization. This is also why Kafka reads are fast: consumers read sequentially forward from their offset, the same access pattern the disk (and the operating system's page cache, which mirrors recently written data in memory) is optimized for.

The log is not free storage — retention is a deliberate boundary

It is tempting, once you see how useful replay is, to assume Kafka simply keeps every event forever. It does not, by default. A log that grows without bound eventually exceeds any disk. Every topic has a retention policy — commonly measured in days — after which the oldest events become eligible for deletion, regardless of whether every consumer has read them yet. This is a deliberate trade-off, not a limitation to work around: retention should be set based on how much replayable history your consumers genuinely need, and how far behind a consumer is realistically allowed to fall before it must fully catch up or restart from scratch. Module 02 covers exactly how this works at the partition level; for now, understand that "durable" and "permanent" are not the same claim.

How Kafka Differs From a Traditional Message Queue

Engineers coming from backend work usually already know a message queue — RabbitMQ, AWS SQS, Azure Service Bus. It is natural to assume Kafka is "just another queue with a fancier name." It is not, and the difference is not cosmetic — it changes what kinds of systems you can build on top of it.

A traditional queue is destructive on read. A message is placed in the queue; a worker pulls it out; the moment that worker acknowledges it, the message is gone from the queue forever. This is exactly the right model for distributing work across a pool of interchangeable workers — a queue of "resize this image" jobs, a queue of "send this email" jobs. Each job should be done exactly once, by exactly one worker, and then it should disappear. If you connect a second, independent worker pool to that same queue expecting it to also see every job, it will not — the two pools will simply compete for the same messages, each job going to whichever pool happens to grab it first.

Kafka's topic is a durable, ordered, non-destructive log. Reading does not remove anything. Multiple independent consumer groups can each read the entire topic, each maintaining their own separate position, without affecting each other at all. This is why Kafka naturally supports fan-out — one event reaching many unrelated systems — in a way a plain queue does not, without resorting to duplicating the message into N separate queues at write time.

# Scenario: an order is placed. Three teams eventually want to react to it.

## On a traditional queue (destructive read):
producer -> queue "order-jobs"
worker-A pulls the message -> message is now GONE from the queue
worker-B connects later, expecting to also react -> sees nothing, message already taken
# Fix requires either 3 separate queues (fan-out exchange) or redesigning entirely

## On a Kafka topic (non-destructive read):
producer -> topic "orders"
billing-group reads it   -> still in the topic
warehouse-group reads it -> still in the topic, billing-group's read did not remove it
fraud-group joins 6 months later -> resets to offset 0, reads full history, no redesign needed

How Kafka Differs From a Database

The other comparison beginners reach for is a database. Both store data durably. Both can be queried by multiple applications. But they answer fundamentally different questions. A database is optimized to answer "what is true right now?" — what is this customer's current address, what is this product's current price, what is the current balance of this account. To answer that question fast, a database typically overwrites old values with new ones and builds indexes over the current state.

Kafka is optimized to answer a different question: "what happened, in what order?" It is a history of facts, not a snapshot of current state. It does not offer arbitrary indexed queries — you cannot ask a raw Kafka topic "show me all orders over $500 placed by customers in California," the way you could a SQL database. What Kafka gives you instead is a durable, ordered, replayable record of every change that ever happened, which downstream systems can consume to build exactly the queryable views they need.

In practice, most real architectures use both, not one instead of the other. A service writes events to Kafka. Multiple downstream systems each consume those events and build their own database — a search index, an analytics warehouse, a materialized cache, a fraud-scoring model's feature store — each shaped for its own query needs. Kafka carries the facts; databases store the queryable current state built from those facts. This pattern even has a name — event sourcing, when a system's database of record is itself rebuilt by replaying a Kafka topic from the beginning — though that is a deeper topic than this module needs to cover.

A pattern worth naming explicitly, because you will see it repeatedly once you start reading real system diagrams: change data capture, or CDC. A CDC connector watches a database's internal change log (for example Postgres's write-ahead log) and publishes every row insert, update, and delete as a Kafka event, in order, without the application code that owns the database needing to change at all. This turns a database's private internal history into a shared, replayable Kafka stream that other systems can consume — a very common bridge between the "current state" world of databases and the "what happened" world of Kafka. It is not a topic this module goes deep on, but recognizing the term will help the rest of this track's later data-pipeline material make sense faster.

Eight Words You Must Own Before Anything Else Makes Sense

The rest of the Kafka track goes deep on each of these individually — events and partitions get their own full module (Module 02), producers/consumers/brokers get their own full module (Module 03). Here, you need a correct, working definition of each word — not the full depth, just enough that the vocabulary stops being a wall of jargon.

Event (record)

A fact that already happened, written as a key, a value, a timestamp, and optional headers. Named in the past tense: OrderPlaced, not PlaceOrder.

Topic

A named stream of related events — orders, payments,user-clicks. Producers write to a topic; consumers read from a topic. A topic is the unit of organization, not the unit of storage — that is the partition.

Partition

One topic is physically split into one or more partitions, each its own independent ordered log. Partitions are how Kafka parallelizes writes and reads across machines. Order is guaranteed within a partition, not across the whole topic.

Producer

An application that writes events to Kafka — a checkout service, a mobile backend, an IoT gateway, a database change-data-capture connector.

Consumer

An application that reads events from Kafka, at its own pace, tracking its own read position.

Broker

A single Kafka server. A group of brokers working together is a cluster. Brokers store partition data on disk, serve reads and writes, and replicate data to each other.

Offset

A position number within one partition — the 0th record, the 1st, the 2nd, and so on. A consumer's entire progress state is just "which offset have I processed up to," per partition it reads.

Replication

Each partition's data is copied to more than one broker, so that if one broker's disk or machine fails, the data still exists elsewhere and the partition keeps serving reads and writes.

checkout-service (a producer) produces an event:
  topic: orders
  key: order-1024
  value: {"event_type":"OrderPlaced","order_id":"1024","total_usd":149.00}

the orders topic has 6 partitions, replicated across a 3-broker cluster
kafka appends this event to:
  orders partition 2, offset 88112
  (replicated onto 3 brokers so no single machine failure can lose it)

independent consumers (each its own application) read it at their own pace:
  billing-consumer-group
  warehouse-consumer-group
  analytics-consumer-group
  support-dashboard-consumer-group

# What each consumer group tracks independently, days later:
billing-consumer-group     committed offset: 88112 (has processed this event)
analytics-consumer-group   committed offset: 88050 (still catching up, 62 events behind)
support-dashboard-group    committed offset: 88112 (also caught up, different pace than billing)

# None of these consumer groups affected each other's progress.
# The event at offset 88112 still exists in the log for all of them.

Notice which of these eight words describe something a producer or consumer application does (event, producer, consumer, offset), and which describe something the Kafka cluster itself provides (topic, partition, broker, replication). Keeping that split straight helps when you are debugging: if something is wrong with which events exist or what they contain, look at the producer. If something is wrong with reading progress, look at the consumer. If something is wrong with data being available, durable, or fast, look at the broker/partition/replication layer. This module's sibling module, Module 03, is organized around exactly that same split.

What each consumer actually does with the same event

It is worth spelling out concretely how differently each consumer can react to the exact same bytes, because this is the payoff of designing events as facts rather than commands. The billing consumer reads OrderPlaced and initiates a charge. The warehouse consumer reads the same event and creates a picking task. The analytics consumer reads it and increments a revenue counter in a dashboard. The fraud consumer reads it and runs a risk score. None of these four consumers call each other, know about each other, or depend on each other's outcome — each one independently decides what the fact OrderPlaced means for its own job. If checkout had instead published four separate commands ("ChargeCard", "CreatePickTask", "LogRevenue", "RunFraudCheck"), adding a fifth consumer later — say, a loyalty-points service — would require checkout to add a fifth explicit call. Publishing one fact instead means the fifth consumer just subscribes; checkout's code does not change.

Decoupling, Replay, Fan-Out, Durability, Throughput — The Business Case

It is worth being explicit about why real engineering organizations choose to run Kafka — not as an abstract technical exercise, but because it solves specific, expensive problems that show up once a company has more than a handful of services.

Decoupling

Teams stop needing to coordinate deploys with every downstream consumer of their data. The producer only needs to agree on an event contract, not on every consumer's implementation details, uptime, or release schedule.

Replay

When a downstream bug is discovered, or a brand-new system needs to be backfilled with history, replaying retained events is dramatically cheaper and safer than rebuilding from database snapshots, backups, or asking every upstream team to re-send data manually.

Fan-out

One event, written once, can be consumed by an arbitrary number of current and future systems — without the producer changing a single line of code when the fifth, sixth, or twentieth consumer is added.

Durability

Replicated, disk-backed storage means an event that has been acknowledged as written will survive the loss of any single machine — a guarantee that ad hoc in-memory queues or direct network calls simply cannot offer.

Throughput

Because the log's storage model is sequential appends and sequential reads (explained fully in the data-engineering message-broker internals module), a modest Kafka cluster can sustain millions of events per second — far beyond what synchronous request-response call chains can absorb without falling over.

A Short, Real History — LinkedIn, Apache, and the Ecosystem Today

Kafka was created at LinkedIn starting around 2010, by a team that included Jay Kreps, Neha Narkhede, and Jun Rao. LinkedIn's problem was exactly the one described in Part 02: dozens of internal systems needed the same streams of activity data — page views, profile updates, connection events — and the existing point-to-point pipelines and batch ETL jobs had become unmanageable, slow to extend, and unreliable for anything approaching real time. The team built Kafka specifically to be that shared, durable, replayable backbone, named after the writer Franz Kafka (reportedly because Jay Kreps liked his work and wanted a name evocative of a "writing-heavy" system).

LinkedIn open-sourced Kafka in 2011, and it was donated to the Apache Software Foundation, where it became a top-level Apache project — this is why it is formally called Apache Kafka, and why "Apache Kafka" and "Kafka" refer to the same open-source project. Several of its original creators later founded Confluent, a company built around commercial Kafka tooling, managed cloud offerings, and enterprise support — but Kafka itself has remained an open, community-governed Apache project, not something owned by any single company.

Today the Kafka ecosystem is broad. You will encounter Kafka in several forms: self-managed open- source Kafka running on your own servers or Kubernetes; managed cloud offerings such as Confluent Cloud, Amazon MSK (Managed Streaming for Apache Kafka), and equivalents on other clouds, which run and operate the brokers for you; and Kafka-API-compatible alternatives such as Redpanda, which reimplement the Kafka protocol with a different internal engine. On top of core Kafka sits an ecosystem of related tools — Kafka Connect for moving data in and out of Kafka without custom code, Kafka Streams and ksqlDB for processing events as they arrive, and Schema Registry for enforcing event contracts as they evolve. This track will introduce several of these in later modules; for now, know that "Kafka" in a modern job posting usually means this whole ecosystem, not only the open-source broker software.

One more piece of history worth knowing because it still shapes how you configure Kafka today: for its first decade, Kafka depended on Apache ZooKeeper, a separate distributed coordination system, to manage cluster metadata and broker leadership elections. Starting with KRaft (Kafka Raft), which became production-ready and the default in Kafka 3.x/4.x, Kafka replaced that external dependency with a built-in consensus protocol, so a modern Kafka cluster no longer requires running ZooKeeper alongside it. If you read older tutorials, blog posts, or job postings that mention ZooKeeper, that reflects the pre-KRaft architecture — still valid historical context, but not how new Kafka clusters are typically deployed today.

What Kafka Does — and What Kafka Does Not Do Automatically

This distinction matters because Kafka is often marketed and discussed as if it solves every real-time or distributed-systems problem by itself. It does not. Kafka is a powerful storage and transport layer for event streams. Your applications still need thoughtful schema design, idempotent processing logic, deliberate error handling, monitoring and alerting, access control, and capacity planning. Kafka gives you the primitives; it does not give you the architecture.

The Whole Picture, and When Kafka Is (and Isn't) the Right Tool

Kafka is usually a strong fit when:

• ✓Multiple systems need the same events independently, now or in the future.
• ✓You need replayable history, not only one-time message delivery.
• ✓Event volume is high enough that durable streaming infrastructure genuinely matters.
• ✓Consumers may be offline, or run slower than producers, and must still catch up safely.
• ✓You are building real-time pipelines, change-data-capture flows, event-driven services, or streaming analytics.
• ✓Ordering matters per business entity — per order, account, customer, or device — not necessarily globally.

Kafka may be the wrong tool when:

• ✓You only need a simple background job queue for one small app with one consumer.
• ✓You need complex ad hoc querying directly over the data — that is a database's job.
• ✓Your team cannot yet operate or justify the operational complexity of running or paying for Kafka.
• ✓The data is tiny, infrequent, and does not need replay or fan-out.
• ✓A normal synchronous database transaction or a direct API call would be simpler and just as reliable.

A useful way to stress-test the decision: write down, honestly, how many independent consumers this data has today, and how many you can genuinely foresee within the next year. If the answer is "one, and it's not changing," a direct call or a simple queue usually wins on simplicity. If the answer is "several today, and this is exactly the kind of fact other teams will want later," that is the shape of problem the rest of this Kafka track is built to solve.

Interview-level explanation

If someone asks "What is Kafka?" in an interview, answer like this: Kafka is a distributed event streaming platform built around an append-only log. Producers write records to topics. Topics are split into partitions for scalability and for ordering within each partition. Brokers store and replicate partitions across a cluster. Consumers read records using offsets, and consumer groups allow parallel processing across partitions. Kafka is used when systems need durable, replayable, high-throughput event streams that many independent applications can consume — as opposed to a traditional queue, which delivers each message to exactly one consumer and then discards it.

Non-technical explanation

If you need to explain Kafka to a non-technical person, say this: Kafka is a reliable timeline for business events. When something important happens, such as an order being placed, Kafka records it durably. Other teams can read that record whenever they need it, even much later. This keeps systems from constantly calling each other directly and makes it much easier to recover if one system is slow or temporarily unavailable — nothing gets lost while it catches up.

Carry three things forward into Module 02: an event is a fact, not an instruction; ordering and parallelism live in tension with each other, mediated by partitions; and Kafka is a deliberate choice with real operational cost, not a default reach for anything that touches more than one service. Module 02 takes the event, topic, and partition vocabulary introduced here and gives each one the full depth this introductory module intentionally left for later.

Five Misconceptions About Apache Kafka

Why This Module Matters on the Job

5 Interview Questions — With Complete Answers

Mistakes Beginners Make Constantly

Confusions and Symptoms You Will Actually Hit — And Exactly Why