Fault Tolerance and Redundancy

Build the mental model

At small scale, a server crash is a rare, notable incident. At the scale of hundreds or thousands of servers, the math flips: if any individual server has even a small daily probability of failure, then across a large fleet, something is failing almost continuously. This reframes the whole design problem — you can't assume failure won't happen, you must design to keep working while failures are happening. Redundancy is the foundational technique: never run a single instance of anything critical, so one failure removes only a fraction of capacity, not the capability entirely. Failover is the automated half — detecting an unhealthy instance (via health checks) and rerouting its traffic to healthy instances, faster than a human could react. The circuit breaker pattern addresses a subtler failure: when a downstream service is struggling, naively retrying calls to it doesn't help — it piles more load onto an already-overwhelmed system and ties up the caller's own resources waiting on timeouts that will fail anyway. A circuit breaker tracks recent failure rates and, once a threshold is crossed, 'opens' — failing calls immediately without attempting the network call — giving the downstream service room to recover, then periodically sends a trial call to test whether it's safe to resume.

Connect it to a real scenario

As the Tutorial Platform runs dozens of servers across multiple regions, individual server failures happen weekly, not never — a disk fills up, a process crashes, a VM gets rebooted for maintenance. Because every service runs at least three redundant instances behind a load balancer with health checks, a single failed instance is automatically routed around within seconds, and most students never notice. When the recommendation-service — a nice-to-have, not critical-path feature — starts timing out under its own load spike, the lesson-service that calls it has a circuit breaker wrapping that call: after a few failures it stops calling recommendation-service entirely for 30 seconds, serving a generic 'popular lessons' fallback instead, rather than letting every page load hang waiting on a service that's already struggling.

• At large fleet scale, some component is failing at nearly all times — systems must be designed assuming failure, not hoping to avoid it.
• Redundancy means no single point of failure; failover automatically routes traffic away from unhealthy instances.
• A circuit breaker stops calling a struggling downstream service instead of retrying into it, failing fast and giving it room to recover.

Try the working example

          failures exceed threshold
   +------------------------------------+
   |                                      v
+--------+                          +--------+
| CLOSED |                          |  OPEN  |
|--------|                          |--------|
| calls  |                          | calls  |
| pass   |                          | fail   |
| through|                          | instantly,
| to the |                          | no network
| service|                          | attempt  |
+--------+                          +--------+
   ^                                      |
   |                                      | cooldown timer expires
   |          trial call succeeds         v
   |    +--------------------------+  +-----------+
   +----|                          |<-|HALF-OPEN  |
        |  trial call fails -------|->| one trial |
        |  (back to OPEN,          |  | call sent |
        |   restart cooldown)      |  | to test   |
        +--------------------------+  +-----------+

The state diagram shows a circuit breaker moving from normal operation (closed) to failing fast on every call (open) once failures cross a threshold, then cautiously testing recovery with a single trial call (half-open) before deciding whether to fully reopen or fall back to failing fast.

Wikipedia — Fault tolerance — System Design