Message Queues and Asynchronous Processing

Decouple services with queues and event streams, background workers, delivery guarantees, retries and dead-letter queues.

Est. 35 min Updated Suggest a fix
// Part 01 — Why asynchronous work

Keep the User Request Focused

A checkout request may need to validate a cart and record an order, then send a receipt, update analytics, notify a restaurant, and schedule a delivery update. If the web request waits for every downstream system, the slowest call determines the user-facing latency. A failure in a nonessential email service can make a successful order appear to fail.

A queue lets a producer persist a unit of work and return, while a worker handles that work independently. This decouples timing and lets consumers scale separately. It does not erase the work: it moves complexity into delivery, retries, ordering, duplicate handling, and operational monitoring.

text
Synchronous chain                         Asynchronous flow

Browser -> Checkout -> Email              Browser -> Checkout -> Order DB
                    -> Analytics                              |
                    -> Restaurant                              +-> Outbox -> Queue
                    -> Delivery                                              |
                                                            Email worker <-+
                                                            Analytics worker <-+
                                                            Restaurant worker <-+
💡 Note
Use asynchronous processing when work can finish after the response, can be retried independently, or has bursty load. Keep a truly required decision synchronous unless the product can represent it as pending and finish it later.
// Part 02 — Queue and stream concepts

Messages, Consumers and Events Have Different Jobs

A message is a request to perform work, such as send this order receipt. An event records a fact that has already happened, such as OrderPlaced. A queue commonly distributes work among competing consumers, where one consumer handles a given delivery. A pub/sub topic or event stream can let multiple independent consumer groups each process the same event history.

ConceptTypical behaviorDesign question
QueueCompeting workers consume pending tasksWhat happens if the queue backs up?
Pub/sub topicMultiple subscriber groups receive eventsWhich consumers need independent copies?
Event streamOrdered log retained for replay within its retention policyHow far back must consumers replay?
WorkerClaims work, performs it, then acknowledgesWhen is it safe to acknowledge?
Dead-letter queueStores repeatedly failing work for inspectionWho diagnoses and replays it?

Product vocabulary varies by broker, so verify whether a feature means per-message acknowledgement, per-partition offset, visibility timeout, or consumer-group checkpoint. The underlying design questions remain the same even when queue APIs differ.

// Part 03 — Design the message

Make the Contract Small, Versioned and Safe

A useful message has a stable identifier, event type, schema version, timestamp, and the minimum data a consumer needs to do its job. Do not put credentials, payment-card details, or unnecessary personal data in a queue payload. If a consumer needs current order state, the message can carry an order ID and the worker can fetch the authorized data.

message_contract.py
python
from dataclasses import dataclass
from datetime import datetime


@dataclass(frozen=True)
class OrderPlacedV1:
    event_id: str
    order_id: str
    customer_id: str
    occurred_at: datetime
    schema_version: int = 1


def receipt_idempotency_key(event: OrderPlacedV1) -> str:
    return f"receipt:{event.event_id}"

Event IDs let a consumer identify a repeated delivery. Schema versions let producers add fields or publish a new contract without silently changing the meaning of existing data. Make consumers tolerate unknown optional fields, and keep compatibility policy explicit when changing required fields.

🎯 Pro Tip
An event should describe a past fact; a command should request an action. Keeping those intentions separate makes it easier to reason about replay, ownership, and whether a message is still valid after state changes.
// Part 04 — Reliability

Assume At-Least-Once Delivery and Make Effects Idempotent

A worker can complete an action and crash before acknowledging the message. The broker then delivers it again. Many systems therefore provide at-least-once delivery under common configurations: a message is not intentionally lost, but the same message may be processed more than once. Exactly-once claims are scoped to a system boundary and do not automatically make an external payment or email exactly once.

Idempotency means repeating an operation with the same key has the same intended effect as doing it once. Persist a processed-event record or use the downstream provider’s idempotency support where available. For order state, use a conditional transition such as pending to confirmed rather than blindly adding another charge or shipment.

worker_idempotency.py
python
class DuplicateEvent(Exception):
    pass


def handle_order(event, processed_events, send_receipt):
    if event.event_id in processed_events:
        return "already processed"

    # The durable implementation must commit this marker and its effect
    # atomically, or use an idempotency key at the external side-effect boundary.
    send_receipt(event.order_id, idempotency_key=f"receipt:{event.event_id}")
    processed_events.add(event.event_id)
    return "processed"
⚠️ Important
A process-local set is only illustrative: it disappears on restart and is unsafe across workers. Production deduplication must be durable, concurrency-safe, retained for an appropriate period, and coordinated with the business side effect.
// Part 05 — Retries and backpressure

A Queue Is Not a Bottomless Buffer

Transient network errors can be retried with bounded exponential backoff and jitter. Permanent failures such as an invalid schema should not be retried forever. After a configured number of attempts, move the message to a dead-letter queue with its error context and original ID. Operators need an alert, a diagnosis path, and a safe replay procedure.

Monitor queue depth, oldest-message age, processing rate, retry count, and dead-letter volume. Queue depth rising for a sustained period means arrivals exceed processing capacity. Add workers only if the downstream database or API can accept more concurrency; otherwise additional consumers amplify the overload.

  1. Choose a maximum in-flight count so each worker has bounded work.
  2. Set retry policy by error class and use backoff with jitter for transient failures.
  3. Move poison messages aside after bounded attempts and preserve enough context to investigate.
  4. Define ordering requirements: global order is costly; per-customer or per-order order may be sufficient.
  5. Test shutdown behavior so a worker finishes or releases its message safely.
// Part 06 — Reliable publishing

Avoid the Database-and-Queue Dual-Write Trap

Imagine the checkout service commits an order row, then crashes before publishing OrderPlaced. The order exists but notification workers never learn about it. Reversing the sequence creates the opposite bug: a message announces an order whose database transaction rolled back. This is a dual-write problem because two independent systems cannot be committed atomically by an ordinary local transaction.

text
Checkout transaction:
  BEGIN
    INSERT order (...)
    INSERT outbox(event_id, type, payload, published_at = NULL)
  COMMIT

Outbox relay:
  read unpublished rows -> publish to broker -> mark row published
  (a crash can publish twice, so consumers still need idempotency)

The transactional outbox stores the business change and an event record in the same database transaction. A relay publishes outbox rows and records progress. It may publish duplicates if it crashes after publishing but before recording success, so consumer idempotency is still required. Retain, monitor, and clean up outbox rows deliberately.

For high-volume stream processing, a log-based broker can also support replay and consumer offsets. Replaying is useful for rebuilding a projection or fixing a bug, but it can repeat external side effects. Keep projection rebuilds separate from sending another real-world receipt or payment request.

// Part 07 — Common mistakes

Make Failure Visible and Recoverable

Assuming a message is delivered exactly once

Why: A crash between the side effect and acknowledgement can cause redelivery.

Fix: Design idempotent handlers and use durable deduplication or downstream idempotency keys.

Retrying poison messages forever

Why: Permanent errors never heal and consume worker capacity.

Fix: Classify failures, cap attempts, and route exhausted messages to a monitored dead-letter queue.

Acknowledging before the work is durable

Why: A worker crash after acknowledgement loses the work.

Fix: Acknowledge only after the durable effect or idempotency record has committed.

Publishing after a database write without an outbox

Why: A crash between the two independent writes drops or invents an event.

Fix: Use a transactional outbox or a documented recovery mechanism, and keep consumers idempotent.

// Part 08 — Practice

Design Order Notifications Without Blocking Checkout

Practice

Draw the failure paths, not just the happy path

  1. Separate synchronous checkout requirements from receipt, analytics, and restaurant-notification work.
  2. Define an OrderPlaced event contract with an ID, version, minimum payload, and privacy-conscious fields.
  3. Sketch the outbox relay, broker, and at least two independent consumer groups.
  4. Walk through a worker crash after sending a receipt but before acknowledgement; explain how a duplicate is harmless.
  5. Choose retry limits and dead-letter alerts, then define who can safely replay a failed notification.

Hint: Mark every durable boundary and the moment an acknowledgement is sent. If a crash can happen between two steps, account for it.

🎯 Key Takeaways

  • ✓Queues decouple request latency and downstream capacity but add delivery, ordering, and recovery responsibilities.
  • ✓Distinguish a work command from an event recording a fact; choose queue, topic, or stream from consumer needs.
  • ✓Design versioned, privacy-conscious messages with stable IDs and small payloads.
  • ✓Assume duplicate delivery and make business effects idempotent; exactly-once behavior has a limited boundary.
  • ✓Use bounded retries, backpressure, dead-letter handling, and queue-age monitoring.
  • ✓A transactional outbox addresses database-and-broker dual writes, while consumers still handle duplicates safely.

Next

What comes next

Queues expose how copies and consumers can disagree temporarily. Next, reason about consistency guarantees, network partitions, quorums, and leader coordination.

Lesson 08 · Consistency, CAP and Consensus →

HELP US IMPROVE

How was this lesson?

Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.

Choose your feedback

Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.

Share

Discussion

0

Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.

Continue with GitHub
Loading...