Serving, Docker, and Hybrid Architecture

Build the mental model

A model server built for one interactive user doesn't automatically behave well with several. Concurrency is the core issue: a GPU running one generation at a time can't simply run five requests in parallel the way a stateless web API can, because each request holds GPU memory — weights plus its own KV cache — for the duration of generation.

Real serving setups handle this with three mechanisms:

Request queuing — accept more requests than can run simultaneously, process them in order or by priority
Batching — some engines process several prompts' decode steps together, improving throughput at some per-request latency cost
Rate limiting — cap concurrent or per-minute requests per client, so one heavy user doesn't starve everyone else

Containerizing a local model with Docker is mostly about isolation and reproducibility, not performance. A typical shape is two containers: one running the inference server with GPU passthrough, one running the application that calls it, connected over a private Docker network rather than the host's public interface.

A named volume for model weights lets a multi-gigabyte download survive container rebuilds instead of re-downloading every time.

Don't Publish the Inference Server's Port

The inference server's port should not be published to the host by default — only the application container needs to reach it, over the internal network.

A local network server extends this to multiple devices: one machine with a capable GPU serves models to laptops and phones on the same LAN, which requires the same authentication and TLS considerations as any exposed service, not an exception because it's 'just the home network.'

Hybrid routing acknowledges that no single model is right for every request:

Route sensitive or simple queries to a small local model
Route requests needing more capability to a larger local model, if hardware allows
Route anything beyond local capability — where data isn't sensitive — to a cloud model

Decide by task properties, not a fixed default.

text

HYBRID ROUTER: CHOOSING A MODEL PER REQUEST
-------------------------------------------
HYBRID ROUTER: CHOOSING A MODEL PER REQUEST
------------------------------------------------

                     incoming request
                            |
                            v
                    +---------------+
                    |    router     |
                    +---------------+
                     /      |       \
                    /       |        \
           sensitive/    needs more   needs capability
           simple task   local power  beyond local HW,
                |             |        data not sensitive
                v             v              v
        small local     larger local     cloud model
          model            model         (external API)
        (fast, cheap)  (slower, better) (most capable,
                                          leaves the machine)

Connect it to a real scenario

The shape above is deliberately conservative about exposure: `model-server` has no `-p`/`ports:` mapping to the host at all, so the only way to reach it is from another container on the same Docker network — the app container, specifically, over its internal hostname (`model-server:8080`), which Docker's embedded DNS resolves automatically inside that network.

Only `app` publishes a port to the host (`3000:3000`), because that's the piece actually meant to be reached from outside the container.

The named volume `model-weights` matters for a reason beyond convenience: without it, a multi-gigabyte model file lives inside the container's writable layer, and `docker rm` or a rebuild throws the download away, forcing a re-pull; a named volume persists independently of the container's lifecycle.

Docker Doesn't Solve Concurrency

Containerizing a model server doesn't change how many requests its single GPU can actually run in parallel — it only changes how the process is packaged and networked. If multiple users need to share one GPU, the model server itself (or a proxy in front of it) needs real queuing or rate-limiting logic; Docker's job here is isolation and reproducible deployment, not scheduling.

Serving more than one user means handling queuing, batching, and rate limiting explicitly — a GPU running one generation at a time doesn't parallelize requests for free.
A safe Docker layout keeps the model server unpublished to the host, reachable only from the app container over an internal network, with model weights in a named volume.
Hybrid routing sends requests to a small local model, a larger local model, or a cloud model based on task properties like sensitivity and required capability — not a single fixed default.

Try the working example

bash

# docker-compose.yml (conceptual shape, not a full working file)
#
# services:
#   model-server:
#     image: ghcr.io/ggml-org/llama.cpp:server
#     command: ["--model", "/models/model.gguf", "--host", "0.0.0.0", "--port", "8080"]
#     volumes:
#       - model-weights:/models
#     networks:
#       - internal
#     # no "ports:" mapping to the host -- not published externally
#
#   app:
#     build: ./app
#     depends_on:
#       - model-server
#     environment:
#       - MODEL_SERVER_URL=http://model-server:8080
#     ports:
#       - "3000:3000"
#     networks:
#       - internal
#
# volumes:
#   model-weights:
#
# networks:
#   internal:

docker network create internal
docker volume create model-weights
docker run -d --name model-server --network internal \
  -v model-weights:/models \
  ghcr.io/ggml-org/llama.cpp:server \
  --model /models/model.gguf --host 0.0.0.0 --port 8080
docker run -d --name app --network internal -p 3000:3000 \
  -e MODEL_SERVER_URL=http://model-server:8080 my-app-image

You should see

None of these commands produce meaningful console output to show — `docker network create` and `docker volume create` print the created resource's ID, and `docker run -d` prints the new container's ID and returns immediately since both run in detached mode. The behavioral result to check is connectivity, not printed text: from inside the `app` container, `curl http://model-server:8080` should succeed, because both containers share the `internal` network and Docker's DNS resolves the service name. From the host machine or any other device on the LAN, attempting to reach the model server directly on port 8080 should fail to connect, because that port was never published to the host — only 3000 was, and only for the app container.

5-minute try-it

Write the router logic described in the diagram as a plain function: given a request's estimated sensitivity (boolean) and estimated complexity (low/medium/high), return which of 'small_local', 'large_local', or 'cloud' it should be routed to, following the rule that any sensitive request always stays local regardless of complexity.

One important caution

Publishing the model server's port to the host ('just to debug it quickly') and forgetting to remove the mapping — it silently defeats the whole point of keeping it reachable only from the app container.

Assuming Docker itself provides request queuing or concurrency limits for a shared GPU — containerizing a model server changes packaging and networking, not how many requests its GPU can actually run at once.

Docker Docs — Networking — Local AI / Local LLM