Load Balancing

Build the mental model

Once a system has multiple servers (the outcome of horizontal scaling), something has to decide which server handles each incoming request — that's a load balancer. Load balancers operate at different layers of the network stack, and the layer determines both their speed and their intelligence. An L4 (transport-layer) load balancer routes based only on IP address and port, without looking at the actual content of the request; it's very fast and cheap in CPU terms because it doesn't need to parse anything beyond packet headers. An L7 (application-layer) load balancer inspects the actual HTTP request — URL path, headers, cookies — and can make smarter routing decisions, like sending all /api/search traffic to servers optimized for search, but that inspection costs more CPU per request. On top of the layer choice, there's an algorithm choice: round robin cycles through servers in fixed order — simple, but blind to how busy each server actually is — while least connections routes each new request to whichever server currently has the fewest active connections, adapting to real load at the cost of tracking connection state.

Connect it to a real scenario

As the Tutorial Platform runs on more than one server, a load balancer sits in front of them deciding which server answers each learner's request. A simple round-robin setup is fine at first, but once some requests are cheap (loading a cached lesson page) and others are expensive (running a code playground execution), round robin can overload a server that happens to get several heavy requests in a row — that's when the platform's engineers would switch to least connections, or even an L7 balancer that routes playground requests to servers provisioned with more CPU specifically for that workload.

L4 load balancers route by IP/port only — fast and cheap but blind to request content; L7 balancers inspect the request itself (URL, headers) for smarter but costlier routing.
Round robin distributes requests in fixed rotation regardless of server load; least connections routes to the server with the fewest active connections, adapting to real load.
A load balancer is only needed once there are multiple servers to distribute across — it's the direct next step after choosing horizontal scaling.

Try the working example

text

                     ROUND ROBIN                        LEAST CONNECTIONS
Client requests      (fixed rotation)                    (checks current load)
   |  |  |  |            |                                    |
   v  v  v  v             v                                    v
+------------------+                              +------------------+
|   Load Balancer   |                              |   Load Balancer   |
+------------------+                              +------------------+
   |       |       |                                 |       |       |
   v       v       v                                 v       v       v
+----+  +----+  +----+                            +----+  +----+  +----+
| S1 |  | S2 |  | S3 |                             | S1 |  | S2 |  | S3 |
|req1|  |req2|  |req3|                             |4conn| |9conn| |1conn|
|req4|  |req5|  |req6|                             |     | |     | |<-new|
+----+  +----+  +----+                            +----+  +----+  +----+
(cycles blindly,                                   (new request goes to S3,
 ignores real load)                                 the least-busy server)

You should see

The diagram contrasts round robin blindly cycling through servers with least connections deliberately routing to whichever server is least busy.

5-minute try-it

Sketch out what happens sending mixed-cost requests to 3 servers via round robin, versus via least connections — what's the observable difference?

One important caution

Using round robin for a workload where request cost varies wildly, letting one server get unlucky and receive several expensive requests in a row while a neighbor sits nearly idle.

Choosing an L7 load balancer everywhere for its smarter routing without accounting for the extra CPU cost of inspecting every request, which can itself become the new bottleneck at very high request volume.

Wikipedia — Load balancing (computing) — System Design