The Local AI Software Ecosystem

Build the mental model

When people say “local AI,” they often picture one thing: a single app you download and chat with. In practice, the local AI ecosystem is made of several distinct layers, each solving a different problem, and understanding the layers matters more than memorizing brand names.

• Model file — at the bottom, weights saved in a format like GGUF or safetensors, just numbers on disk with no way to talk to it directly.
• Runtime — the engine that loads those weights into memory and actually performs inference; llama.cpp and Ollama's internal engine are both runtimes.
• API layer — usually a small local HTTP server exposing endpoints like /api/chat or /v1/chat/completions, so other software can send a prompt and get a JSON response back.
• Application layer — desktop chat apps like LM Studio or Jan, developer frameworks like LangChain or LlamaIndex, and agent frameworks like CrewAI or AutoGen.
• Model hubs — such as Hugging Face and Ollama's library, where you discover and download model files.
• Vector databases — such as Chroma and Qdrant, which store embeddings for retrieval.

Alongside this vertical stack, model hubs and vector databases are two more categories that cut across it. Recognizing which category a tool belongs to tells you what problem it actually solves, and stops you from expecting a model hub to run inference or a chat app to serve an API to other programs.

Runtime

The engine that loads a model's weights into memory and performs the actual computation to generate output. llama.cpp and Ollama's internal engine are both runtimes.

Inference Server

A runtime wrapped with a network-facing API (usually HTTP), so other programs — not just a single command line — can send it requests and get responses, typically running continuously in the background.

MODEL FILE TO APPLICATION STACK
-------------------------------
------------------------------------------------------------
APPLICATION   LM Studio, LangChain, LlamaIndex, CrewAI, AutoGen
                 ^  (chat apps, dev frameworks, agent frameworks)
                 |
              calls
                 |
API LAYER     HTTP server: /api/chat, /v1/chat/completions
                 ^  (JSON request in, JSON response out)
                 |
          exposed by
                 |
RUNTIME       llama.cpp, Ollama engine
                 ^  (loads weights, runs inference)
                 |
             loads
                 |
MODEL FILE    weights on disk: model.gguf, model.safetensors
              (just numbers -- cannot be "talked to" directly)

SIDE CATEGORIES (cut across the stack, not the vertical flow):
  MODEL HUBS        Hugging Face, Ollama Library -> source of files
  VECTOR DATABASES  Chroma, Qdrant                -> store embeddings

Connect it to a real scenario

Imagine your team wants to add a “chat with your documents” feature to an internal tool, and someone drops a list of names into a planning doc: Ollama, LM Studio, LangChain, Chroma, Hugging Face, CrewAI. Without a mental map, that list looks like seven competing products you have to choose between.

• One runtime — to actually run a model (Ollama, since the team is comfortable in a terminal and wants an API by default).
• One model hub — to pick a model from (Hugging Face, to compare options, though you'll likely pull the final choice through Ollama's own library).
• One vector database — to store document embeddings (Chroma, because it's simple to run locally for a first version).
• One developer framework (optional) — to wire the pieces together instead of writing all the HTTP calls by hand (LangChain).

LM Studio and CrewAI drop out entirely, not because they're worse, but because a GUI desktop app and a multi-agent orchestrator aren't the layer this feature needs yet. The exercise below asks you to do exactly this sorting exercise yourself with a different tool list, so the habit of asking “what layer does this actually sit at” becomes automatic before you touch any code.

• The stack runs model file -> runtime -> API -> application, each layer solving a distinct problem.
• Model hubs and vector databases are side categories that cut across the vertical stack rather than sitting inside it.
• Knowing a tool's category tells you what it actually does and prevents mismatched expectations.

Try the working example

CATEGORY_BY_TOOL = {
    "LM Studio": "Desktop App",
    "Jan": "Desktop App",
    "GPT4All": "Desktop App",
    "Ollama": "CLI Runtime",
    "llama.cpp": "CLI Runtime",
    "vLLM": "Inference Server",
    "text-generation-webui": "Inference Server",
    "LangChain": "Developer Framework",
    "LlamaIndex": "Developer Framework",
    "Hugging Face Hub": "Model Hub",
    "Ollama Library": "Model Hub",
    "Chroma": "Vector Database",
    "Qdrant": "Vector Database",
    "AutoGen": "Agent Framework",
    "CrewAI": "Agent Framework",
}


def classify_tools(tool_names):
    result = {}
    for name in tool_names:
        result[name] = CATEGORY_BY_TOOL.get(name, "Uncategorized")
    return result


if __name__ == "__main__":
    tools = ["Ollama", "LM Studio", "vLLM", "LangChain", "Chroma", "CrewAI", "Hugging Face Hub"]
    classified = classify_tools(tools)
    for tool_name, category in classified.items():
        print(f"{tool_name}: {category}")

Running the script prints each tool name mapped to its category, one per line, using the exact classification: Ollama: CLI Runtime, LM Studio: Desktop App, vLLM: Inference Server, LangChain: Developer Framework, Chroma: Vector Database, CrewAI: Agent Framework, Hugging Face Hub: Model Hub.

Ollama on GitHub — Local AI / Local LLM