Latency vs Throughput

Build the mental model

Latency and throughput sound similar but measure different things, and improving one can quietly hurt the other. Latency is how long a single request takes from start to finish — the time one user waits for one response. Throughput is how many requests the system completes per second across all users combined. Batching is the clearest place they conflict: grouping 100 database writes into one batch improves throughput dramatically (fewer round trips, more total work done per second) but makes any individual write wait until the batch fills up, increasing its latency. Just as important is how you measure latency at all: averages lie, because a handful of very slow requests get smoothed away by many fast ones. Percentiles fix this — p50 (median) tells you the typical experience, p95 tells you what the slower 5% of users see, and p99 tells you the worst 1%. A p50 of 50ms sounds excellent, but if p99 is 3 seconds, 1 in 100 requests is nearly unusable — and at a scale of millions of requests, that 'rare' 1% is a very large number of genuinely frustrated real people.

Connect it to a real scenario

As the Tutorial Platform grows, its engineers can't just watch 'average response time' on a dashboard and call it healthy — a comfortable-looking average of 80ms can hide the fact that learners on a slow connection, or requests that hit a cold cache, are waiting 4 seconds at p99. Tracking p50/p95/p99 for endpoints like lesson loading or search-suggest lets the team catch that a real slice of learners is having a bad experience, even while the average metric looks perfectly fine. This distinction — and the batching trade-off — comes back throughout the course whenever a design choice claims to make the platform 'faster'.

Latency = time for one request; throughput = requests completed per second — they are different axes and can trade off against each other.
Batching is a classic latency-for-throughput trade: it increases total work done per second but delays individual requests.
Percentiles (p50/p95/p99), not averages, reveal how the slowest requests actually feel, because averages get smoothed over by the many fast ones.

Try the working example

text

Request count
   ^
   |        ___
   |      _/   \_
   |    _/       \___
   |  _/              \________
   | /                          \________________
   |/_________________________________________________\____> latency (ms)
        ^p50            ^p95              ^p99
        50ms             400ms             3000ms

  Most requests cluster fast (near p50) --
  but the tail stretches far right: p99 is 60x slower than p50.

You should see

The diagram shows a latency distribution where p99 can be dramatically slower than p50 even while the median looks fast.

5-minute try-it

Make up 10 response times (in ms) for an endpoint, then compute the average, p50, and p99 by hand — what gap do you notice?

One important caution

Reporting only the average latency on a dashboard, which hides a heavy tail of slow requests that a meaningful fraction of real users are actually experiencing.

Adding batching to improve throughput without noticing it raises the latency floor for every individual request, which can break a feature that actually needed a fast, immediate response (e.g. an autocomplete box).

Wikipedia — Latency (engineering) — System Design