Managed Kafka and Cloud Choices

Running Kafka Yourself Is a Real, Ongoing Team-Time Cost — Not Just a Server Bill

Every module up to this point has treated "the Kafka cluster" as a given — brokers exist, they have leaders and followers, they store data on disk. What those modules did not cover is who keeps that cluster alive: who patches broker versions, plans disk capacity ahead of growth, responds when a broker's disk fills at 3 AM, rebalances partitions after adding hardware, and keeps ZooKeeper or a KRaft controller quorum healthy. Self-hosting Kafka means a team owns all of that, indefinitely, on top of building the actual data pipelines the business needs.

This is the real reason managed Kafka exists and has grown to dominate new deployments: it is not that self-hosted Kafka is unreliable — plenty of large, sophisticated companies run it well — it is that operating a distributed, stateful system well requires specialized, ongoing expertise that many teams would rather not build and staff for, when a vendor can provide the same capability as a managed service.

None of this means self-hosting is the wrong choice for every team — Part 05 covers exactly when it still makes sense. But the decision should be made with the full, ongoing cost in view, not just the sticker price of running broker VMs, which is usually the smallest part of the real cost.

Confluent Cloud — the Broadest Feature Parity, From the Company Kafka's Original Creators Founded

Confluent was founded by the original creators of Kafka at LinkedIn, and Confluent Cloud is its fully managed Kafka offering. Because Confluent has spent over a decade building the broader ecosystem around Kafka — Schema Registry, ksqlDB, a large library of Kafka Connect connectors, and the reference implementations several of those pieces are built around — Confluent Cloud generally offers the broadest feature parity with a full, self-hosted open-source Kafka ecosystem of any managed option.

What comes managed, beyond just brokers

A meaningful distinction from other managed Kafka options: Confluent Cloud does not just manage broker infrastructure, it manages the surrounding ecosystem components as fully hosted services too — a managed Schema Registry (the same Avro/Protobuf schema-compatibility enforcement covered in the Kafka Connect module), managed Kafka Connect (many of the same source and sink connectors covered in that module, offered as a point-and-click managed service rather than something you deploy and operate on your own Connect cluster), and managed ksqlDB for stream processing without managing a Kafka Streams application's own deployment.

The practical implication for a team evaluating Confluent Cloud: if the pipeline design already depends on Schema Registry, Connect, or ksqlDB — which, having gone through the Kafka Connect module, most production pipelines in this track's scope do — Confluent Cloud is worth strong consideration specifically because it removes the operational burden of those pieces too, not just the brokers.

Amazon MSK — Managed Brokers With Deep AWS Integration, More Client-Side Responsibility Left to You

Amazon Managed Streaming for Apache Kafka (MSK) is AWS's managed Kafka offering. It manages broker provisioning, patching, and the underlying infrastructure, but MSK's managed surface is narrower than Confluent Cloud's — it does not bundle a managed Schema Registry, managed Connect, or managed ksqlDB as first-party equivalents (AWS offers a separate, related but distinct service, MSK Connect, for running Kafka Connect connectors, and Glue Schema Registry as a separate product for schema management, rather than these being unified into one MSK offering the way Confluent bundles them).

Deep AWS IAM and VPC integration

MSK's strongest differentiator is how tightly it integrates with the rest of AWS. Broker access can be authenticated using IAM roles and policies directly — the same access-control model used for S3 buckets, Lambda functions, and every other AWS service — rather than managing a separate SASL credential system. MSK clusters run inside your own VPC by default, meaning network-level access control, security groups, and private connectivity to other AWS services (S3, Lambda, Kinesis, RDS) work the same way they do for any other AWS-native resource, with no separate networking model to learn.

# A producer authenticating to MSK using IAM, rather than SASL/SCRAM:

producer_config = {
    "bootstrap.servers": "b-1.mycluster.abc123.kafka.us-east-1.amazonaws.com:9098",
    "security.protocol": "SASL_SSL",
    "sasl.mechanism": "AWS_MSK_IAM",
    # No static username/password anywhere in this config --
    # the AWS SDK's normal credential chain (an EC2 instance role,
    # an ECS task role, or an assumed IAM role) is what authenticates.
}

# The IAM policy attached to that role grants specific Kafka
# actions on specific resources, the same policy-document model
# used for every other AWS service:
{
  "Effect": "Allow",
  "Action": ["kafka-cluster:Connect", "kafka-cluster:WriteData"],
  "Resource": "arn:aws:kafka:us-east-1:123456789012:topic/mycluster/*/orders"
}

MSK Serverless — usage-based, no capacity planning

MSK Serverless is a variant that removes broker sizing and partition capacity planning entirely — you create topics and produce/consume, and AWS scales the underlying capacity automatically, billing based on actual throughput and storage rather than provisioned broker instances. This trades some of the fine-grained performance tuning available on provisioned MSK clusters (and a higher per-unit cost at sustained high throughput) for meaningfully less operational thinking about cluster sizing — closer in spirit to how a team would think about a fully serverless database.

Redpanda and WarpStream — Kafka-Protocol-Compatible, Built on Fundamentally Different Architectures

A newer category of entrant does not run Apache Kafka's actual broker code at all. Instead, these systems implement the Kafka wire protocol — the same network protocol every Kafka client library already speaks — on top of a completely different internal architecture. The practical significance is that existing Kafka producer and consumer client code can often point at one of these systems with just a configuration change, no application code rewrite, because from the client's point of view, the protocol looks like Kafka.

Redpanda — Kafka-API-compatible, not built on the JVM

Redpanda is a from-scratch reimplementation of a Kafka-compatible broker, written in C++ rather than Java, with no JVM in its runtime at all. Its stated design goals center on avoiding JVM garbage-collection pauses (a real source of the session-timeout and rebalance issues covered in the producers-consumers-brokers module) and simplifying operations by removing ZooKeeper-or-KRaft as a separate concern — Redpanda bundles its own Raft-based consensus directly into each broker process rather than requiring a distinct controller quorum to operate.

WarpStream — Kafka-API-compatible, built directly on object storage

WarpStream takes a more radical architectural departure: rather than brokers with attached local disks holding partition data the way Kafka, Redpanda, and MSK all fundamentally work, WarpStream's brokers are stateless and write data directly to object storage (S3 or an equivalent) as the primary and only durable store — there is no broker-local disk holding the canonical copy of a partition's data at all. This removes the need for the broker-to-broker replication mechanics covered in the message-brokers module (leaders, followers, ISRs) as the durability mechanism, since object storage itself provides the durability, at the cost of the higher latency inherent to writing through an object store rather than a local disk.

The reason these options are worth knowing conceptually, even for a team that ultimately chooses Confluent Cloud or MSK: they represent a genuinely different set of trade-offs — lower operational complexity and different cost curves in exchange for a younger ecosystem — and understanding what "no JVM" or "object-storage-native" actually changes structurally (not just as a marketing claim) is what lets you evaluate whether that trade-off fits a specific workload's needs.

Self-Managed Kafka on Kubernetes — Control Without Hand-Rolling Every Operational Script

Between fully managed and fully hand-rolled self-hosting sits a middle ground: running Kafka yourself, but on Kubernetes, using an operator that automates the operational mechanics that would otherwise be manual scripts and runbooks. Strimzi is the most widely adopted open-source Kafka operator for this purpose.

A Kubernetes operator, in general, is a piece of software that encodes operational knowledge about running a specific system as Kubernetes-native automation — instead of a human running a rolling restart script for a broker version upgrade, the operator watches a declarative configuration (expressed as Kubernetes custom resources) and reconciles the running cluster to match it, handling the sequencing of broker restarts, and coordinating changes across the cluster safely.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: freshcart-kafka
spec:
  kafka:
    replicas: 3
    version: 3.7.0
    config:
      default.replication.factor: 3
      min.insync.replicas: 2
    storage:
      type: persistent-claim
      size: 500Gi
  entityOperator:
    topicOperator: {}
    userOperator: {}

# Bumping "version: 3.7.0" to a newer version and re-applying this
# resource triggers Strimzi's own controlled, rolling upgrade of
# every broker in the cluster -- the same operation Part 01 described
# as a real, careful, manual process on a hand-rolled self-hosted setup.

Cost Isn't One Number — Self-Hosted Trades Infra Cost for Headcount Cost, Managed Trades It Back

Comparing "self-hosted Kafka" against "managed Kafka" purely on a cloud infrastructure bill is a category error. The honest cost comparison has to include the ongoing engineering time from Part 01 — capacity planning, patching, on-call, security administration — priced at that team's actual fully-loaded cost, not treated as free because it does not show up as a separate line item on a cloud invoice.

Self-hosted total cost, roughly:
  infrastructure cost (broker VMs, disk, network)
  + ongoing engineering time (capacity planning, patching,
    on-call, security admin) x fully-loaded engineer cost
  + the opportunity cost of that engineering time not being
    spent on product work

Managed total cost, roughly:
  usage-based or capacity-based vendor pricing
    (which already has the vendor's own ops cost baked in)
  + a smaller amount of engineering time for integration,
    monitoring, and vendor-specific configuration

Neither number is universally smaller. The crossover point
depends entirely on throughput scale and how much the team's
engineering time is worth relative to the vendor's markup.

Where the crossover point tends to sit

At low-to-moderate, steady throughput, managed pricing is usually the cheaper total-cost choice for most teams, because the ongoing engineering-time cost of self-hosting a small cluster well rarely justifies itself against a modest vendor bill — a small team keeping a Kafka expert on call for a cluster processing a few hundred megabytes a day is spending far more in engineering time than the equivalent managed service would cost.

At sustained very high throughput, usage-based managed pricing can grow to significantly exceed the cost of self-hosted infrastructure plus a dedicated operations team, because the per-unit markup on managed pricing is charged against every byte at scale, while the engineering-time cost of running a larger self-hosted team does not scale linearly with throughput the same way — a platform team sized to operate a 50-broker cluster well is not meaningfully bigger than one sized to operate a 15-broker cluster well.

Migrating Between Kafka Deployments — Client Code Doesn't Change, the Operational Surface Does

Because every option covered in this module — self-hosted Apache Kafka, MSK, Confluent Cloud, Redpanda, WarpStream, Strimzi-on-Kubernetes — speaks the same Kafka wire protocol, a producer or consumer written against one of them generally requires no application code changes to point at another. The bootstrap servers change, authentication configuration changes, and TLS settings change — the actual producer.send() or consumer.poll() call in the application does not.

# Before: self-hosted Kafka cluster
producer_config = {
    "bootstrap.servers": "kafka-1.internal:9092,kafka-2.internal:9092",
    "security.protocol": "SASL_SSL",
    "sasl.mechanism": "SCRAM-SHA-512",
    "sasl.username": "orders-service",
    "sasl.password": "...",
}

# After: migrated to Confluent Cloud
producer_config = {
    "bootstrap.servers": "pkc-abc123.us-east-1.aws.confluent.cloud:9092",
    "security.protocol": "SASL_SSL",
    "sasl.mechanism": "PLAIN",
    "sasl.username": "<API key>",
    "sasl.password": "<API secret>",
}

# The application code calling producer.send(topic, key, value)
# is IDENTICAL before and after. Every acks, retries, batching,
# and idempotence setting from the producers-consumers-brokers
# module still means exactly what it meant before.

What does change, and requires real planning, is the operational surface: monitoring dashboards and alerting rules built against self-hosted broker metrics need to be rebuilt against whatever observability surface the new platform exposes, topic and ACL administration workflows change to whatever the new platform's management interface is, and any tooling that assumed direct SSH access to broker hosts (for log inspection, disk checks, or manual intervention) has no equivalent on a fully managed platform, where that layer is intentionally not exposed to the customer.

A useful practical consequence of this: a team is not locked into a migration decision the way it might be with a database that requires a genuine data-model rewrite. A poor initial choice of managed provider, or a self-hosted setup that has outgrown a team's operational capacity, is a real but bounded migration project — mostly configuration, deployment tooling, and monitoring rework — rather than an application rewrite.

Choosing a Deployment Model — A Framework Based on Team Size, Cloud Provider, Compliance, and Throughput

With every option's real trade-offs on the table, the actual decision comes down to four factors most teams weigh, in roughly this order of practical importance.

Factor 1 — team size and existing platform expertise

A small team with no dedicated platform or infrastructure function should weight heavily toward a fully managed option (Confluent Cloud or MSK), because the ongoing engineering-time cost from Part 06 is disproportionately expensive for a small team relative to the vendor markup. A team that already runs a mature Kubernetes platform, with existing operational muscle for exactly the kind of concerns Strimzi automates, has a much more viable self-managed path than a team starting from zero.

Factor 2 — existing cloud provider

A team already deeply invested in AWS — using IAM for access control everywhere else, running workloads inside a carefully designed VPC topology — gets real, compounding value from MSK's native IAM and VPC integration that a cloud-agnostic managed service like Confluent Cloud does not replicate as tightly, even though Confluent Cloud also runs on AWS (and GCP and Azure) as underlying infrastructure. Conversely, a team intentionally avoiding cloud-provider lock-in, or running a genuinely multi-cloud footprint, often prefers a provider-agnostic managed option specifically to avoid deepening a single-cloud dependency.

Factor 3 — compliance and data-residency requirements

Regulated industries with specific data-residency requirements (data must stay within a specific country's borders, or within a specific certified environment) need to verify a managed provider's available regions and compliance certifications (SOC 2, HIPAA, PCI-DSS, FedRAMP, and equivalents) cover their specific requirement before committing — availability varies by provider and by region, and this is exactly the kind of requirement that can rule out an otherwise-preferred option outright, making it worth verifying early rather than after most of the evaluation is done.

Factor 4 — throughput and its predictability

As covered in Part 06, sustained very high and predictable throughput tends to favor self-hosted or Kubernetes-based deployment on a long enough time horizon, while low, moderate, or unpredictable throughput tends to favor managed, usage-based pricing. This factor is deliberately weighted last in this framework because, in practice, most teams evaluating this decision are not yet at the throughput scale where this factor dominates the other three — it becomes the deciding factor specifically for larger, more mature platforms.

What to Actually Check Before Committing — A Practical Evaluation Checklist

The decision framework in Part 08 narrows the field. Before signing a contract or migrating production traffic, it is worth running a short, concrete evaluation against the specific shortlisted options, rather than deciding purely on paper feature comparisons and pricing pages.

Throughput and latency, measured against your own traffic shape

Published benchmarks from any vendor — including Confluent, AWS, Redpanda, and WarpStream — are run against traffic patterns chosen to showcase that vendor's strengths. A team's own producer batching settings, message sizes, partition counts, and acks configuration (all covered in the producers-consumers-brokers module) materially change real-world throughput and latency, often by a wide margin from a generic published number. Running an actual load test with representative traffic — same message sizes, same partition strategy, same acks setting the production workload will use — against each shortlisted option is the only way to get a number that means anything for the specific workload in question.

Support responsiveness and escalation paths

A fully managed service's support tier and SLA matter disproportionately more than they would for a less operationally critical system, precisely because choosing managed means deliberately giving up the direct operational visibility self-hosting would provide. Before committing, it is worth understanding concretely what happens during an actual incident — what response-time SLA applies at the specific pricing tier being considered, whether there is a direct escalation path to an engineer versus a generic support queue, and what visibility the platform provides into an in-progress incident on the vendor's side (a public status page is a minimum bar, not a complete answer).

1. Run an actual load test with representative message sizes,
   partition counts, and acks settings -- not the vendor's
   published benchmark numbers.

2. Test every specific Connect connector, Schema Registry
   compatibility rule, or ksqlDB feature the pipeline design
   already depends on -- don't assume "supports Kafka" covers
   every feature in use today.

3. Confirm the specific compliance certifications and available
   regions cover the requirement (Part 08, Factor 3) -- in writing,
   not from a general marketing page.

4. Ask directly: what is the support SLA at this pricing tier,
   and what does escalation actually look like during an incident?

5. Estimate cost at both current throughput AND a realistic
   growth projection 12-24 months out -- usage-based pricing
   that looks fine today can cross the Part 06 crossover point
   sooner than expected if growth is fast.

6. Confirm the migration path back out is understood before
   migrating in -- per Part 07, this should be a configuration
   project, not a surprise application rewrite, but it is worth
   confirming that understanding explicitly rather than assuming it.

Multi-Region Kafka — How the Deployment Model Changes the Disaster-Recovery Story

A single-region Kafka cluster, whether self-hosted or managed, is only as durable as that region. A team with a genuine business requirement to survive a full regional outage needs a multi-region story, and how straightforward that story is differs sharply across the deployment models this module has covered.

Cluster linking and cross-cluster replication — the general mechanism

Regardless of provider, cross-region Kafka durability is generally achieved by replicating topics from a primary cluster in one region to a standby cluster in another, using a cluster-to-cluster replication tool — MirrorMaker 2 in the open-source ecosystem, or a vendor-specific equivalent (Confluent Cluster Linking, for instance) that offers the same underlying capability with tighter integration into that vendor's own management tooling. This is a fundamentally different mechanism from the in-cluster replication (leaders, followers, ISRs) covered in the message-brokers module — that replication protects against a single broker failing within one cluster; cross-region replication protects against an entire cluster, and the region it runs in, becoming unavailable.

The practical takeaway: multi-region requirements are a real, additional factor beyond Part 08's four-factor framework for a team that genuinely needs to survive a regional outage, and the operational cost of that requirement varies significantly by provider — a fully managed, first-party cross-region feature (as with Confluent Cloud) is a meaningfully different commitment than deploying and operating MirrorMaker 2 as an additional piece of self-managed infrastructure on top of an otherwise-managed cluster, which is closer to the MSK reality today.

Who Actually Owns What — Team Topology Changes More Than the Infrastructure Diagram

Moving from self-hosted to managed Kafka, or the reverse, is not purely an infrastructure decision — it changes what a platform or infrastructure team's job actually consists of day to day, and getting that team-topology shift wrong is a common, under-discussed source of friction after a migration that was technically successful.

Self-hosted — a platform team owns the full stack

With self-hosted Kafka (or Strimzi-on-Kubernetes), a platform team's Kafka-related work spans the full stack: broker capacity, version upgrades, security patching, topic and ACL administration, and being the first responder for every category of incident, from a single flaky broker to a full cluster outage. This is a substantial, specialized skill set, and teams that build it well tend to develop deep operational intuition about their specific cluster's behavior under their specific workload — a real asset, but one that takes real time and dedicated headcount to build and, importantly, to retain as engineers move on to other roles or companies.

Managed — the platform team's job shifts toward integration and governance

With a fully managed provider, a platform team's Kafka-related work shifts away from infrastructure operations and toward integration, governance, and cost management: onboarding new teams onto shared clusters or topics, administering topic-naming and schema-governance conventions (the same naming-discipline concerns covered for Kafka Connect apply directly here), managing the vendor relationship and monitoring spend against the cost model from Part 06, and being the escalation point to the vendor's own support organization during an incident rather than being the incident's first, and only, responder.

Security Configuration Looks Similar Everywhere — the Access-Control Model Underneath Differs

Every option covered in this module supports the same broad security building blocks the earlier Kafka modules assumed — TLS in transit, authentication, and per-topic authorization — but how those building blocks are configured and administered differs enough across providers to be worth a direct comparison, especially for a team that will be running many teams' workloads on one shared cluster.

Authentication mechanisms, by provider

Multi-tenancy — many teams, one cluster

A shared cluster serving many internal teams needs a clear multi-tenancy model: which team can create topics, who can read or write to a given topic, and how one team's misbehaving producer or consumer (a runaway retry loop, an unbounded topic) is prevented from degrading service for every other team on the same cluster. Quotas — per-client-ID or per-user limits on produce/consume throughput — are the standard mechanism for the second concern, available in some form across every provider covered here, though the specific configuration surface (a Kafka-native quota config on self-hosted clusters, versus a managed provider's own throughput-tier and quota controls) differs by provider.

# Limiting one client's produce and consume throughput,
# to prevent it from starving other tenants on a shared cluster:

kafka-configs --bootstrap-server broker:9092 --alter \
  --add-config 'producer_byte_rate=5242880,consumer_byte_rate=10485760' \
  --entity-type clients --entity-name reporting-service-readonly

# reporting-service-readonly is now capped at 5MB/s produce and
# 10MB/s consume, regardless of how much the underlying broker
# hardware could otherwise deliver to it -- protecting every other
# tenant's share of the cluster's real capacity.

What You Can See Differs by Provider — and That Difference Shapes On-Call

Part 01 framed self-hosting's operational cost partly in terms of who responds to a 3 AM broker failure. The other half of that picture is what an on-call engineer can actually see when something goes wrong, and that visibility differs meaningfully depending on how much infrastructure sits behind the vendor's abstraction layer.

This is not an argument against managed platforms — for most teams, trading away the deepest tier of host-level visibility is a reasonable trade for not needing to build the expertise to interpret it in the first place. But it is worth setting on-call expectations accurately: an engineer debugging an incident on a fully managed cluster will, at some point, hit a wall where the next diagnostic step is "open a support ticket," rather than "SSH into the broker" — and that hand-off point should be a known, rehearsed part of the team's incident response process, not a surprise discovered mid-incident.

Five Misconceptions About Managed Kafka and Cloud Choices

What This Looks Like on Day One

5 Interview Questions — With Complete Answers

Mistakes Beginners Make Constantly

Errors and Surprises You Will Hit — And Exactly Why They Happen