APIs and Service Communication — REST, gRPC, GraphQL and Webhooks

Design clear APIs and choose between REST, gRPC, GraphQL and webhooks while handling pagination, versioning, retries and idempotency.

Est. 35 min Updated Suggest a fix
// Part 01 — The contract

An API Is a Promise Between Independent Callers

An API describes what a caller may ask for, what result or error to expect, and which constraints apply. A useful contract names resources or operations, request and response shapes, authentication, authorization, validation, pagination, compatibility and failure behaviour. It is a product boundary: changing a field, default or error can break a mobile app that has not been updated yet.

Begin with a use case and its ownership. For example, the order service owns order state; a customer app requests an order, but should not directly write an arbitrary status into the database. The interface should represent allowed transitions and make the result understandable to both the caller and the service team.

🎯 Pro Tip
Design the interface around a stable domain action, not the current shape of an internal table. Internal storage can evolve without forcing every client to migrate in lockstep.
// Part 02 — REST

Use Resource-Oriented HTTP When It Fits the Interaction

REST-style APIs commonly represent resources with URLs and use HTTP methods to express operations: retrieve a resource, create one, replace or partially update it, or remove it. HTTP status codes communicate broad outcomes; a structured response body can add a stable machine-readable error code and a safe message. Do not return stack traces, database details or secrets to clients.

text
POST /api/orders
Authorization: Bearer <access-token>
Idempotency-Key: 8f9e...client-generated-key
Content-Type: application/json

{"items":[{"productId":"course-42","quantity":1}],"addressId":"addr-7"}

201 Created
Location: /api/orders/order-812
{"id":"order-812","status":"pending_payment"}

The response indicates that the resource was created and gives its location. The exact status code and representation are design choices for the contract; consistency matters more than copying a sample blindly. Define validation errors, authentication failures, conflicts and transient service failures in the API documentation and tests.

build_url.py
python
from urllib.parse import urlencode


def course_search_url(base_url: str, query: str, page_size: int, cursor: str | None = None) -> str:
    if page_size < 1:
        raise ValueError("page_size must be positive")
    params = {"q": query, "limit": page_size}
    if cursor is not None:
        params["cursor"] = cursor
    return f"{base_url.rstrip('/')}/api/courses?{urlencode(params)}"


print(course_search_url("https://learn.example", "data engineering", 20, "next-page"))

Encoding parameters avoids corrupting a URL when a search term contains spaces or punctuation. Real clients should also enforce reasonable page-size limits and handle authentication, cancellation, timeouts and non-success responses.

// Part 03 — Choose a protocol

REST, gRPC, GraphQL and Webhooks Solve Different Problems

REST over HTTP is broadly interoperable and easy to inspect. gRPC uses typed service definitions and is often useful for efficient service-to-service calls and streaming, especially when both ends can use its tooling. GraphQL lets a client request a shape of data through a schema; it can reduce over-fetching for complex clients but needs careful authorization, query-cost limits and caching. Webhooks are outbound HTTP notifications from one system to another; they are not a replacement for a request-response API.

Choose based on clients, interaction pattern and operating constraints—not on a protocol popularity list.
StyleUseful whenCosts to plan for
REST / HTTPPublic or browser-facing resources and broad compatibilityResource and error consistency, pagination, versioning
gRPCTyped internal calls, generated clients or streamingProtocol tooling, gateway needs and coordinated schemas
GraphQLClients need flexible, related data shapesAuthorization per field, query complexity and caching strategy
WebhookA receiver needs to learn that an event occurredDelivery retries, signatures, duplicates and replay handling
text
Mobile app  -- HTTPS request --> API gateway --> Order service
                                                  |       |
                                                  |       +--> order database
                                                  |
                                                  +-- event --> durable queue --> notification worker
                                                                        |
Payment provider -- signed webhook --> webhook endpoint ----------------+

The API response acknowledges the order action; downstream notifications
and payment events have their own delivery and retry contracts.
📌 Real World Example
If a payment provider calls your webhook, verify its signature using the provider's documented method, store the event identifier and process it idempotently. Do not trust a public request merely because its JSON looks valid.
// Part 04 — Pagination and compatibility

Keep Large Collections and Evolving Clients Safe

Do not return an unbounded collection. Offset pagination is easy to understand, but deep offsets can become costly and concurrent inserts can move results between pages. Cursor pagination returns a continuation token based on a stable ordering. Treat the cursor as opaque to clients and bind it to the relevant filters and sort order; validate it rather than trusting client-provided internals.

Backward-compatible changes generally add optional fields or new capabilities without changing the meaning of existing ones. Removing a field, changing units or repurposing an enum value can break older clients. Versioning should be an explicit policy with a deprecation window, usage visibility and a migration path—not an excuse to avoid compatibility tests.

cursor_page.py
python
from dataclasses import dataclass

@dataclass
class Page:
    items: list[str]
    next_cursor: str | None


def page(items: list[str], start: int, page_size: int) -> Page:
    if start < 0 or page_size < 1:
        raise ValueError("start must be non-negative and page_size positive")
    selected = items[start:start + page_size]
    next_start = start + len(selected)
    next_cursor = str(next_start) if next_start < len(items) else None
    return Page(selected, next_cursor)


catalog = ["python", "sql", "distributed systems", "cloud"]
print(page(catalog, start=0, page_size=2))

This example uses a plain offset encoded as a cursor only to illustrate response shape. A production cursor should represent a stable ordering key and should be opaque, validated and scoped to the query. A timestamp alone can be ambiguous when records share the same timestamp; use a deterministic tie-breaker such as an identifier.

// Part 05 — Safe retries

Make Writes Idempotent Before Retrying Them

A timeout does not tell the client whether the server completed the operation. If the client blindly repeats a payment or enrolment request, it may create duplicate effects. Idempotency means repeating the same operation with the same key and payload produces one logical result. Persist the key, request identity and outcome for an appropriate retention period; reject reuse of the key with a different payload. The exact storage design depends on the domain.

idempotency_keys.py
python
import hashlib
import json


def request_fingerprint(payload: dict) -> str:
    canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()


payload = {"course_id": "course-42", "learner_id": "learner-9"}
print(request_fingerprint(payload))

A fingerprint can detect payload changes, but a hash function alone does not make an operation idempotent. The application must atomically claim a unique client key, run or resume the operation, and persist the outcome so concurrent retries do not race. Never put sensitive data into a key merely because it is opaque.

⚠️ Important
Retry only failures that may be transient, with a bounded policy and backoff. Do not retry validation or permission errors. Respect any server-provided retry guidance and avoid retrying at multiple layers without a total attempt budget.
// Part 06 — Communication

Synchronous Calls Need Deadlines; Events Need Delivery Rules

A synchronous service call is easy to reason about while every dependency is fast, but one slow dependency can hold resources and push tail latency through the entire request chain. Give calls a deadline, propagate the remaining budget, limit concurrency and return a clear outcome when the deadline expires. A timeout does not cancel work automatically unless the client and server propagate cancellation.

Asynchronous communication can isolate slow work and smooth bursts. It changes the contract: callers may receive “accepted” or “pending,” consumers may see duplicates, and status becomes visible later. Define event schema compatibility, ordering requirements, retries, dead-letter handling and how a consumer safely repeats work. An event notification is not necessarily the source of truth; identify which system owns the durable state.

ConcernSynchronous requestAsynchronous event
User feedbackImmediate success or failureUsually pending, followed by a later update
Failure behaviourTimeout and dependency failure affect callerQueue retention and consumer recovery matter
Duplicate handlingIdempotent request keysIdempotent event processing
Best fitDecision needed before responseIndependent work that may complete later
// Part 07 — Pitfalls

Common API Design Mistakes

Returning a database row as the public contract

Why: Internal schema changes then become client-breaking changes.

Fix: Define a stable, documented representation around the domain use case.

Retrying a write after every timeout

Why: The first request may already have committed its side effect.

Fix: Use an idempotency key and make the operation outcome durable.

Returning every record in one response

Why: Large responses waste bandwidth and can overload memory or databases.

Fix: Use bounded pagination with stable ordering and an opaque cursor.

Treating a webhook as trusted input

Why: Any reachable client can send a forged or replayed request.

Fix: Verify signatures, validate schema, deduplicate and process safely.

// Part 08 — Practice

Design an Enrolment API Contract

Practice

Specify the contract before writing the handler

  1. Define a request and response for enrolling a learner in a course, including the relevant identifiers.
  2. List expected validation, authorization, conflict and transient failure outcomes.
  3. Choose a pagination approach for the learner’s course history and explain its ordering key.
  4. Describe an idempotency-key lifecycle for a client that retries after a timeout.
  5. Decide whether an enrolment confirmation email is part of the response path or an asynchronous event.

Hint: Write down ownership: the enrolment service decides whether a learner is enrolled; an email worker should not invent that state.

🎯 Key Takeaways

  • ✓An API contract defines operations, data shapes, errors, access rules and compatibility expectations.
  • ✓REST, gRPC, GraphQL and webhooks fit different client and communication needs; compare their operating costs.
  • ✓Bound collection responses and use deterministic pagination that can tolerate concurrent data changes.
  • ✓A timeout leaves outcome uncertain; idempotency protects writes from duplicate side effects.
  • ✓Synchronous calls need deadlines and bounded retries; asynchronous events need delivery, ordering and deduplication rules.
  • ✓Keep public contracts independent of private database schemas and verify external callbacks.

Next

What comes next

You have defined how clients and services communicate. Next, learn how a proxy routes requests across healthy application instances and how a CDN brings cacheable content closer to users.

Lesson 04 · Load Balancing, Reverse Proxies and CDNs →

HELP US IMPROVE

How was this lesson?

Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.

Choose your feedback

Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.

Share

Discussion

0

Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.

Continue with GitHub
Loading...