Project: Per-Client Rate Limiter Service

Build the mental model

A single global token bucket protects a server from overall overload, but it has a fairness problem: if one client sends a burst of requests and drains the shared bucket, every other client gets throttled too, even though they did nothing wrong. The fix is to give each client its own token bucket, keyed by an identifier like an API key or user ID, so buckets refill and drain independently — one noisy client only ever exhausts their own quota. The natural implementation is a dict mapping client_id to a TokenBucket instance, created lazily the first time a given client is seen, which avoids pre-allocating buckets for clients who may never show up. The real design trade-off is where that dict lives: keeping it in a single process's memory is simple and fast, but it only works if all of a client's requests hit the same server instance. The moment the service scales to multiple instances behind a load balancer, per-client state has to move to a shared store like Redis (with atomic increment/expire operations), or a client could get more than their fair share of tokens by having their requests spread across instances that each think the bucket is empty.

Connect it to a real scenario

This models the rate limiting the Tutorial Platform would need on its public API — for example an endpoint serving lesson content to third-party integrations or a mobile app. Each API key gets its own bucket so a runaway script from one integration partner can't degrade response times for every other learner hitting the same endpoint. Today the platform is a single Next.js deployment, so an in-memory dict is a legitimate stopgap; the moment it scales past one server instance (multiple Vercel regions, for example), this in-memory version would silently under-enforce limits per instance, which is exactly the trigger for moving the bucket state into Redis.

• One token bucket per client_id, not one global bucket, so a single abusive client can't throttle everyone else.
• Buckets are created lazily on first request per client, avoiding wasted memory for clients that never appear.
• In-memory per-client state only works on a single server instance; multi-instance deployments need a shared store like Redis.

Try the working example

import time


class TokenBucket:
    def __init__(self, capacity: int, refill_rate: float):
        self.capacity = capacity
        self.refill_rate = refill_rate  # tokens per second
        self.tokens = float(capacity)
        self.last_refill = time.monotonic()

    def _refill(self):
        now = time.monotonic()
        elapsed = now - self.last_refill
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)
        self.last_refill = now

    def allow(self) -> bool:
        self._refill()
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False


class RateLimiterService:
    def __init__(self, capacity: int, refill_rate: float):
        self.capacity = capacity
        self.refill_rate = refill_rate
        self._buckets = {}  # client_id -> TokenBucket

    def _get_bucket(self, client_id: str) -> TokenBucket:
        if client_id not in self._buckets:
            self._buckets[client_id] = TokenBucket(self.capacity, self.refill_rate)
        return self._buckets[client_id]

    def allow_request(self, client_id: str) -> bool:
        return self._get_bucket(client_id).allow()


if __name__ == "__main__":
    limiter = RateLimiterService(capacity=3, refill_rate=0)  # no refill during demo

    print("Client A (bursty, sends 5 requests):")
    for i in range(5):
        print(f"  request {i + 1}: {'allowed' if limiter.allow_request('client-a') else 'THROTTLED'}")

    print("Client B (sends 2 requests):")
    for i in range(2):
        print(f"  request {i + 1}: {'allowed' if limiter.allow_request('client-b') else 'THROTTLED'}")

Client A's first 3 requests are allowed and the last 2 are throttled once its bucket is empty, while Client B's 2 requests are both allowed because it has its own separate, untouched bucket.

Wikipedia — Token bucket — System Design