Load Balancers, Reverse Proxies, and Forward Proxies

Build the mental model

Three boxes get confused constantly, and the difference is entirely about which side they represent. A forward proxy sits in front of clients and acts on their behalf: a corporate egress proxy, a caching proxy. The server sees the proxy, not the client. A reverse proxy sits in front of servers and acts on their behalf: the client thinks it is talking to your site, and the proxy decides which backend actually answers. A load balancer is a reverse proxy whose defining job is distributing that traffic. Same wire, opposite allegiance.

The next distinction is the layer it works at. An L4 load balancer forwards TCP connections. It sees addresses, ports, and bytes, nothing else. That makes it very fast and protocol-agnostic, and it means it cannot route by URL path, read a header, or terminate TLS. An L7 load balancer terminates the connection, parses HTTP, and can send /api to one pool and /images to another, rewrite headers, retry idempotent requests, and read cookies. The cost is CPU, plus holding the certificate, which is why L7 balancing and TLS termination almost always live in the same box.

Algorithms matter less than people expect. Round-robin distributes requests evenly, which distributes load evenly only if requests cost the same. Least-connections adapts by sending work to whoever is least busy, and is usually the better default. Hashing on client IP or a header always sends the same client to the same backend, buying cache locality with no shared state.

Health checks keep the pool honest: a backend failing its check is removed, so a crash becomes a slowdown instead of an outage. And sticky sessions, which pin a user to one backend, quietly reintroduce the problem load balancing solved, because the pinned backend cannot be drained, and losing it loses those sessions.

LOAD BALANCER TOPOLOGY AND L4 VS L7 VISIBILITY
----------------------------------------------
  clients             balancer            backend pool
  -------            ----------          --------------
                                          +------------+
   [c1] --\          +----------+     /-->| 10.0.1.11  |
           \         |          |    /    +------------+
   [c2] ---------->  |   VIP    |---+     +------------+
           /         | 1.2.3.4  |    \--->| 10.0.1.12  |
   [c3] --/          |          |    \    +------------+
                     +----------+     \   +------------+
                          |            \->| 10.0.1.13  |
                          |               +------------+
                          \-- health checks --> all three


  L4 can see:  src IP, dst IP, ports, TCP connection state
     -> very fast, works for any protocol
     -> CANNOT read a URL path, a header, or a cookie
     -> CANNOT terminate TLS

  L7 can see:  all of the above PLUS method, path, headers,
               cookies, and the body
     -> route /api and /images to different pools
     -> retry a failed idempotent request on another backend
     -> must hold the certificate and pay CPU to parse


  forward proxy:  client -> [PROXY] -> internet   (acts FOR client)
  reverse proxy:  client -> [PROXY] -> servers    (acts FOR servers)

Connect it to a real scenario

A concrete scenario: the Tutorial Platform runs three application servers behind one balancer, and users intermittently get logged out. The team's first theory is a session bug.

It usually is not. It is almost always that sessions live in the process memory of whichever backend served the login, and the next request landed elsewhere. There are two fixes and they are not equal. Sticky sessions pin each user to a backend by cookie or source IP, and the symptom vanishes immediately. It also means you can never drain a backend without dropping everyone pinned to it, load skews toward whichever backend collected the long-lived users, and every deploy becomes disruptive. The better fix is to move session state out of the process, into Redis or a signed cookie, after which any backend can serve any request.

The second thing to get right is the health check. A check that only opens a TCP connection tells you the process is running, not that it works. A backend that has lost its database accepts TCP all day while returning 500 to every request, and an L4 check keeps sending it traffic. An L7 check that requests a real endpoint and requires a 200 catches this. Tune the interval and failure threshold too: too aggressive and a garbage-collection pause ejects a healthy server.

• Reverse proxy faces servers; forward proxy faces clients
• L4 is fast and blind; L7 can route on paths, headers, and cookies
• Sticky sessions hide a state problem instead of fixing it

Try the working example

import hashlib

# A backend pool. 'inflight' is how many requests each server is still
# working on -- the number a least-connections balancer actually reads.
BACKENDS = ["10.0.1.11", "10.0.1.12", "10.0.1.13"]

# Hardcoded request costs, so both algorithms see the same workload.
# Request 3 is a slow one: it occupies whichever backend takes it.
REQUESTS = [
    ("GET /",          1),
    ("GET /style.css", 1),
    ("POST /report",   5),   # slow
    ("GET /logo.png",  1),
    ("GET /about",     1),
    ("GET /favicon",   1),
]


def round_robin(requests, backends):
    inflight = {b: 0 for b in backends}
    idx = 0
    log = []
    for name, cost in requests:
        chosen = backends[idx % len(backends)]
        idx += 1
        inflight[chosen] += cost
        log.append((name, chosen))
    return log, inflight


def least_connections(requests, backends):
    inflight = {b: 0 for b in backends}
    log = []
    for name, cost in requests:
        # min() over a sorted list breaks ties by address, deterministically
        chosen = min(sorted(backends), key=lambda b: inflight[b])
        inflight[chosen] += cost
        log.append((name, chosen))
    return log, inflight


def show(title, log, inflight):
    print(title)
    for name, chosen in log:
        print("  " + name.ljust(14) + " -> " + chosen)
    print("  load: " + "  ".join(b + "=" + str(inflight[b])
                                 for b in BACKENDS))
    print("")


show("round-robin", *round_robin(REQUESTS, BACKENDS))
show("least-connections", *least_connections(REQUESTS, BACKENDS))

# Hashing: the same client always lands on the same backend, with no
# shared state between balancers. That is also its weakness -- a hot
# client cannot be spread out.
print("consistent-ish hashing by client IP")
for client in ["192.0.2.7", "198.51.100.4", "203.0.113.5"]:
    digest = hashlib.sha256(client.encode()).hexdigest()
    chosen = BACKENDS[int(digest, 16) % len(BACKENDS)]
    print("  " + client.ljust(14) + " -> " + chosen)

round-robin
  GET /          -> 10.0.1.11
  GET /style.css -> 10.0.1.12
  POST /report   -> 10.0.1.13
  GET /logo.png  -> 10.0.1.11
  GET /about     -> 10.0.1.12
  GET /favicon   -> 10.0.1.13
  load: 10.0.1.11=2  10.0.1.12=2  10.0.1.13=6

least-connections
  GET /          -> 10.0.1.11
  GET /style.css -> 10.0.1.12
  POST /report   -> 10.0.1.13
  GET /logo.png  -> 10.0.1.11
  GET /about     -> 10.0.1.12
  GET /favicon   -> 10.0.1.11
  load: 10.0.1.11=3  10.0.1.12=2  10.0.1.13=5

consistent-ish hashing by client IP
  192.0.2.7      -> 10.0.1.11
  198.51.100.4   -> 10.0.1.12
  203.0.113.5    -> 10.0.1.13

Cloudflare Learning Center - What is load balancing? — Computer Networking