Load Balancing, Reverse Proxies and CDNs

Understand reverse proxies, Layer 4 and Layer 7 load balancing, routing algorithms, health checks and safe CDN delivery.

Est. 35 min Updated Suggest a fix
// Part 01 — Why distribute

One Application Instance Is a Capacity and Failure Limit

A load balancer receives traffic and distributes it among backend instances. This can increase aggregate capacity, permit maintenance without taking the whole service offline and isolate failed instances from new traffic. It does not make the application itself correct or eliminate every failure: the balancer, network, shared database and deployment process still need attention.

A reverse proxy sits in front of one or more services on behalf of clients. It can terminate TLS, route by host or path, apply limits and add observability. A load balancer is often implemented as or alongside a reverse proxy, but the terms describe related roles rather than one mandatory product shape.

text
Clients
   |
   v
DNS / edge protection
   |
   v
Load balancer or reverse proxy
   |             |             |
   v             v             v
App instance A  App instance B  App instance C
   |             |             |
   +-------------+-------------+
                 |
                 v
       Shared data and dependencies

A healthy route needs capacity in the shared dependencies too.
💡 Note
Horizontal scaling adds instances; vertical scaling gives one instance more resources. Both have limits. A database connection pool or a single hot partition may become the bottleneck even when application CPU is low.
// Part 02 — Network layers

Layer 4 and Layer 7 See Different Information

A Layer 4 balancer routes transport connections using information such as IP addresses and ports. It generally does not need to understand HTTP paths. A Layer 7 balancer understands application protocol details such as HTTP hosts, paths, headers or cookies, making richer routing and HTTP-aware controls possible. These capabilities vary by implementation; check the chosen platform rather than assuming every product has the same feature set.

The OSI-layer framing is a useful mental model, not a substitute for checking product capabilities.
DimensionLayer 4 routingLayer 7 routing
Routing inputConnection-level addresses and portsApplication attributes such as host or path
Protocol awarenessTransport-orientedUnderstands the application protocol it proxies
Typical useHigh-throughput TCP or UDP distributionHTTP routing, TLS termination or request policies
Trade-offLess request-aware policyMore inspection and policy complexity

If a request path routes to several versions of a service, define how headers and authentication are forwarded and which hop is trusted to set them. Never treat a client-supplied forwarding header as proof of the caller’s network identity. Proxy configuration is part of the security boundary.

// Part 03 — Routing algorithms

Choose a Routing Policy That Matches the Workload

Round robin rotates across available targets and works well when instances have similar capacity and requests have similar cost. Weighted routing sends a larger share to a target with more capacity. Least-connections can help when requests remain open for different durations, but connection count is not always a good proxy for CPU or work. Hash routing can preserve affinity for a key, but hot keys may overload one target and stateful sessions make failover harder.

round_robin.py
python
class RoundRobin:
    def __init__(self, targets: list[str]):
        if not targets:
            raise ValueError("at least one target is required")
        self.targets = targets
        self.position = 0

    def next_target(self) -> str:
        target = self.targets[self.position]
        self.position = (self.position + 1) % len(self.targets)
        return target


router = RoundRobin(["app-a", "app-b", "app-c"])
print([router.next_target() for _ in range(7)])

This small program is only an algorithm demonstration; production routing must also exclude unhealthy targets, handle concurrency, connection draining, weights and failure recovery. A balancer may use different policies for long-lived connections than for individual HTTP requests.

📌 Real World Example
If one learner’s requests must reach the same server because session state lives in process memory, sticky sessions may seem convenient. A shared session store or stateless authentication can make failover and scaling more predictable, but it changes where state is stored and how it is protected.
// Part 04 — Health and draining

A Health Check Must Match the Traffic You Send

A liveness check asks whether a process should be restarted; a readiness check asks whether it can accept work now. They answer different questions. A process may be alive but unable to reach its required database. Conversely, marking every instance unready whenever an optional service is degraded can remove all capacity unnecessarily. Decide which dependencies are essential for each route and make checks resistant to transient noise.

healthy_target.py
python
def choose_target(targets: list[dict[str, object]], start: int = 0) -> str | None:
    if not targets:
        return None
    for offset in range(len(targets)):
        candidate = targets[(start + offset) % len(targets)]
        if candidate.get("ready") is True:
            return str(candidate["name"])
    return None


pool = [
    {"name": "app-a", "ready": False},
    {"name": "app-b", "ready": True},
]
print(choose_target(pool))

On deploy or scale-in, stop assigning new requests to an instance and allow in-flight work to finish within a bounded drain period. Long-lived connections need explicit handling. Monitor rejected requests, health transitions, queue depth and target utilization; “healthy” at the balancer is not proof that the user journey succeeds.

⚠️ Important
A dependency-aware health endpoint can create a cascading outage if every application instance fails readiness at once. Define the intended response for degraded but recoverable dependencies and test the transition.
// Part 05 — Capacity and resilience

The Balancer Cannot Create Capacity

Distribution helps only while there is spare capacity downstream. If each instance has finite throughput, the pool needs headroom for traffic variation, deployments and a target failure. Autoscaling can add capacity but needs a signal, a safe maximum, startup time and a plan for what happens before new instances are ready. Database pools, external API quotas and shared storage must be included in the capacity model.

Keep sessions and files off ephemeral instance disks unless their loss is acceptable. Design graceful overload behaviour: reject low-priority work, shed optional features, or return a retryable response with appropriate guidance. Unbounded queues and unlimited retries move overload around rather than resolving it.

text
                 +--> instance A --+
Request --> LB --+--> instance B --+--> database connection pool
                 +--> instance C --+          |
                                             +--> primary database
                                             +--> optional search service

Adding app instances can saturate the shared pool; measure each dependency.
// Part 06 — CDN

Serve Cacheable Content Near the User

A content delivery network (CDN) serves content through edge locations closer to users. It is especially useful for versioned static assets and other content with safe cache rules. A cache hit can reduce origin requests and improve delivery latency; a cache miss, expired object or bypass still reaches the origin. CDN configuration does not make private data safe by itself.

Cache keys commonly depend on the URL and may include selected query parameters or headers. Including every header can fragment the cache; ignoring a header that changes the response can leak one user's content to another. Use explicit cache-control policy, avoid shared caching for personalized responses unless the policy is designed for it, and test invalidation during a release. Signed URLs or cookies can gate private downloads when supported by the chosen delivery design.

ContentTypical edge-cache questionCaution
Hashed JS/CSS assetsDoes the URL change whenever the file changes?Keep old assets available during client rollout
Public course thumbnailsCan all users receive the same image?Version the URL or define invalidation
Personalized dashboard JSONDoes the response vary by identity or role?Avoid shared caching without a safe key and policy
Video segmentsAre access controls and cache lifetime explicit?Protect private media and plan egress limits
🎯 Pro Tip
A content hash in an asset filename gives a new URL when the bytes change. That can let clients cache immutable assets for a long time while the HTML page points to the latest version.
// Part 07 — Pitfalls

Common Load-Balancing and CDN Mistakes

Treating a successful health check as proof the service works

Why: The check may not cover routing, dependencies or a real user request.

Fix: Monitor end-to-end SLIs as well as liveness and readiness.

Scaling application instances without measuring shared services

Why: More callers can overload database pools or dependency quotas.

Fix: Load-test the whole path and track saturation at every bottleneck.

Caching personalized responses as public content

Why: A cache key may not distinguish identity or authorization.

Fix: Use conservative cache controls and verify isolation across users.

Depending on sticky sessions for correctness

Why: A target can fail or be removed while session state is local.

Fix: Prefer explicit shared state or stateless requests when feasible.

// Part 08 — Practice

Route a Course Platform Through an Edge

Practice

Draw a routing and content-delivery policy

  1. Sketch the path from a browser to three application instances and a shared database.
  2. Choose a routing policy and explain whether instance capacity and request durations are similar.
  3. Define separate liveness, readiness and connection-draining behaviour for a deployment.
  4. Classify course thumbnails, hashed assets and learner dashboards as cacheable or private, with a reason.
  5. Name one shared dependency that could saturate after adding instances and identify a metric to monitor.

Hint: For every cache rule, say who may see the response, which request attributes change it and how an update becomes visible.

🎯 Key Takeaways

  • ✓A load balancer distributes requests but cannot fix an overloaded shared dependency or a faulty application.
  • ✓Layer 4 routing is connection-oriented; Layer 7 routing can use application-level request attributes.
  • ✓Round robin, weights, least-connections and affinity each rely on different workload assumptions.
  • ✓Liveness, readiness and draining have different roles during failure and deployment.
  • ✓A CDN reduces origin work for safely cacheable content; cache keys and privacy controls must be deliberate.
  • ✓Measure the full request path and design overload behaviour instead of assuming added instances guarantee resilience.

Next

What comes next

The edge can serve reusable content, but dynamic applications also need a deliberate cache policy. Next, compare cache-aside, write-through and write-back, then work through invalidation and stampedes.

Lesson 05 · Caching — Strategies, Invalidation and Redis →

HELP US IMPROVE

How was this lesson?

Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.

Choose your feedback

Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.

Share

Discussion

0

Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.

Continue with GitHub
Loading...