Load Balancers and Reverse Proxies
Distribute requests, terminate connections safely, and understand health checks, algorithms, and failure modes.
Prerequisites: Be comfortable with IP routing, TCP connections, DNS, HTTP requests, and TLS at a conceptual level.
Learning objectives: Distinguish Layer 4 and Layer 7 balancing, select a backend strategy, design health checks and draining, and reason about proxy timeouts and trust boundaries.
Chapter 01
One Service, Many Servers
A load balancer accepts client connections or requests and chooses a backend to handle each one. A reverse proxy accepts requests on behalf of servers, then forwards them to an origin. These roles overlap: a reverse proxy often balances traffic, while a load balancer commonly proxies connections. The important distinction is the responsibility being discussed, not the product label.
Typical reasons to put this layer in front of an application include scaling across instances, centralising TLS, routing different paths or hostnames, hiding backend addresses, applying limits, and draining instances during deployments. It does not automatically make application code fault-tolerant: a shared database, queue, or downstream API can still be the bottleneck.
Client
│ HTTPS request: api.example.test/orders
▼
Load balancer / reverse proxy
├── app-a:8080 (healthy)
├── app-b:8080 (healthy)
└── app-c:8080 (draining)
│
▼
Shared database / queueChapter 02
Layer 4 and Layer 7 Decisions
A Layer 4 balancer makes choices using transport and network connection information—such as destination IP and TCP or UDP port—without parsing the HTTP request. This is useful for high-throughput forwarding and non-HTTP protocols. A Layer 7 balancer understands application details such as HTTP host, path, headers, or cookies, so it can route /api and /assets differently or apply HTTP-aware policies.
Application awareness adds flexibility and responsibility. An L7 proxy may need to parse requests, manage certificates, enforce header limits, and maintain connection pools. A proxy that terminates TLS sees plaintext at that point, so access to its logs, memory, and configuration becomes part of the security boundary. Re-encrypting from proxy to backend protects that second network hop.
Common proxy patterns
- Reverse proxy: clients know the proxy; it selects an origin.
- Forward proxy: clients send outbound traffic through a proxy that represents them.
- TLS pass-through: the balancer forwards encrypted bytes; the backend owns the TLS session.
- TLS termination: the proxy decrypts client TLS and may open a separate connection to the backend.
Chapter 03
How a Backend Is Chosen
Scheduling algorithms trade off simplicity, fairness, and knowledge of backend load. Round robin rotates through eligible servers and works well when servers and requests are similar. Weighted round robin sends proportionally more work to larger instances. Least connections favors a server with fewer active connections; it can help when requests last different lengths, but connection count is not the same as CPU or queue depth. Hash-based selection can keep a client or key near one backend, but can create hotspots and makes membership changes affect the mapping.
| Strategy | Useful when | Watch for |
|---|---|---|
| Round robin | Requests and instances are broadly similar | Unequal instance capacity or long-lived connections |
| Weighted | Backends have known capacity differences | Weights become stale after scaling or hardware changes |
| Least connections | Connection duration varies materially | A few costly requests can overload a server with a low count |
| Consistent hash | A key benefits from repeatable backend placement | Hot keys, membership churn, or accidental stickiness |
Algorithms distribute requests, not necessarily user sessions. Prefer stateless application instances and store session state in a shared service. If a legacy application requires affinity, understand its timeout, failover behavior, and uneven-load consequences rather than treating stickiness as a cure for state design.
Chapter 04
Health Checks and Safe Removal
A health check is a recurring test used to decide whether a backend should receive traffic. A TCP check proves that a port accepts connections; an HTTP check can prove that a specific path returns an acceptable response. A good readiness endpoint tests the dependencies needed to serve real traffic, but should be fast, bounded, and designed so that a brief optional dependency issue does not eject every instance.
# Conceptual health-check policy (provider-neutral)
interval: 10s
connect_timeout: 2s
healthy_after: 2 consecutive successes
unhealthy_after: 3 consecutive failures
request:
method: GET
path: /readyz
expected_status: 200Detection has a delay: interval, timeout, and failure threshold determine how long a broken backend may continue receiving work. Too aggressive a policy can cause flapping during a short pause; too lenient a policy extends user impact. During a deployment, mark an instance as draining, stop assigning new requests, allow in-flight work to finish within a bounded grace period, and only then terminate it. Long-lived WebSocket or streaming connections need an explicit drain strategy.
Chapter 05
A Small Reverse-Proxy Configuration
The following Nginx excerpt illustrates a common pattern: TLS ends at the proxy, requests go to a named upstream, and the original host and client address are forwarded. Certificate paths and backend names are deployment-specific; the example is not a complete production TLS policy.
upstream app_pool {
least_conn;
server app-a.internal:8080 max_fails=3 fail_timeout=10s;
server app-b.internal:8080 max_fails=3 fail_timeout=10s;
keepalive 32;
}
server {
listen 443 ssl;
server_name api.example.test;
location / {
proxy_pass http://app_pool;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Connection "";
}
}The application must only trust forwarded headers from known proxies. Otherwise, a direct client can send a forged X-Forwarded-For value and confuse audit logs, rate limits, or IP-based policy. Restrict backend access so clients cannot bypass the proxy, configure trusted proxy hops, and avoid using a client IP as an authentication factor.
Chapter 06
Timeouts, Retries, and the Failure Cascade
Proxies add queues, connection pools, and timeouts. Set finite connect, read, and idle timeouts that match the service contract. If a backend is slow, queued work can consume memory and connections; if every layer retries independently, one request can multiply into several backend attempts and amplify load. Retries are safest for operations known to be idempotent and should have a small budget, backoff, and a deadline shared across the request path.
# Retry decision checklist
request is safe to repeat? yes / no
failure is plausibly transient? yes / no
request deadline remains? yes / no
retry budget remains? yes / no
If any answer is no: return the original failure.
For writes, use an application idempotency key rather than
assuming a timeout means the server did not commit the action.Use connection limits and queue limits to protect the proxy and its backends. Return an explicit overload response when the system is at capacity instead of accepting unlimited work that will time out later. Observe requests per backend, active connections, queue time, health transitions, 4xx/5xx rates, and latency percentiles; aggregate metrics can hide one failing instance.
Chapter 07
Debugging the Request Path
Trace one request hop by hop. Verify DNS points to the expected frontend; inspect TLS certificate and SNI; check that the proxy selected the expected route and backend; compare proxy and application logs using a request ID; then test backend reachability from the proxy network. Record the request ID in logs, but never place secrets or full sensitive payloads in it.
# Useful read-only checks from an authorised environment
curl -sS -D - -o /dev/null https://api.example.test/readyz
curl -sS -H 'X-Request-ID: lab-001' https://api.example.test/health
# Compare DNS answers and the TLS peer certificate
getent ahosts api.example.test
openssl s_client -connect api.example.test:443 -servername api.example.test </dev/nullA 502 commonly means the proxy could not obtain a valid upstream response; a 503 often means no backend is considered available or the service is refusing work; a 504 indicates the proxy timed out waiting. These are clues, not universal definitions: examine the specific proxy's logs and documentation. Also check asymmetric routing, firewall rules, mismatched ports, name resolution from the proxy, and backend TLS trust.
Chapter 08
Mistakes to Avoid and Production Review
- One giant health endpoint: slow dependency checks can make every instance fail together. Keep readiness scoped and bounded.
- Unlimited retries: retries turn a partial outage into a load spike. Bound attempts and honor deadlines.
- Trusting client-supplied proxy headers: accept them only from the known proxy chain.
- Assuming sticky sessions provide resilience: affinity can pin users to a broken or overloaded server.
- Ignoring proxy logs: log route, backend, status, duration, and request ID while redacting sensitive data.
Practice
- Sketch the client-to-proxy-to-two-backend path for a service used by a mobile app. Mark where TLS starts and ends.
- Choose a balancing strategy for identical stateless API instances, then explain why least-connections might not reflect CPU load.
- Design a readiness check for an order API. Which dependencies are essential, and what timeout prevents a cascade?
- Describe a safe rolling deployment for a service with long-running requests.
- A user reports intermittent 502s. List five observations you would collect before changing configuration.
🎯 Key Takeaways
- ✓A reverse proxy represents backend servers to clients; a load balancer distributes work. One component can do both jobs.
- ✓Layer 4 decisions use connection information; Layer 7 decisions can use HTTP hosts, paths, and headers.
- ✓Choose scheduling based on request duration, instance capacity, and state—not by habit.
- ✓Health checks, draining, finite timeouts, and bounded retries are part of availability design.
- ✓TLS termination creates a trust boundary; protect the proxy-to-backend hop and trust forwarded headers only from known proxies.
- ✓Observe per-backend latency, error rates, health transitions, queue time, and request IDs to diagnose failures.
HELP US IMPROVE
How was this lesson?
Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.
Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.
