Exercise: Design Your Own Local AI Stack

Build the mental model

Every lesson in this course has fed into one repeatable decision framework, and this exercise is where you finally use all of it at once. The framework asks three questions, in order.

What is your main task: general chat, coding help, question-answering over your own documents, or letting the model take actions through tools as an agent?
What hardware do you actually have: CPU-only, an integrated GPU, or a dedicated GPU, and how much RAM and VRAM does that hardware give you to work with?
What matters most to you when those first two answers create a tradeoff: raw speed, answer quality, keeping everything private and offline, or fitting inside a tight memory budget?

None of these questions has a universally correct answer. A student on a five-year-old laptop and a developer with a workstation GPU can both build a working local AI stack, but the model size, quantization level, runtime, and whether they need retrieval-augmented generation or agent tooling on top will look completely different for each of them.

A framework, not a checklist

What makes this a framework rather than a checklist is that the three answers combine: task sets a floor on capability, hardware sets a ceiling on size, and priority breaks the tie when the floor and the ceiling do not leave much room to work with.

Models, quantization formats, and hardware will all keep changing after you finish this course. The three questions, and the order you ask them in, will not. That is the actual skill this exercise is testing.

text

THE THREE-QUESTION LOCAL AI DECISION FRAMEWORK
----------------------------------------------
THE THREE-QUESTION LOCAL AI DECISION FRAMEWORK
-----------------------------------------------------------
   TASK                HARDWARE              PRIORITY
   chat / coding /      CPU-only / iGPU /     speed / quality /
   documents / agent    dGPU + RAM + VRAM     privacy / low-mem
        \                    |                     /
         \                   |                    /
          v                  v                   v
                    ---------------------
                    |   RECOMMENDATION   |
                    ---------------------
                    - model size band
                    - quantization level
                    - runtime: CPU-only or GPU
                    - RAG needed?  agent needed?

Connect it to a real scenario

Let's apply the framework to two concrete people.

Persona: small business owner

A small business owner wants private question-answering over contracts and invoices, on an ordinary office laptop with 16 GB of RAM and no dedicated GPU.

Task -> documents

Task: documents, which strongly implies retrieval-augmented generation, since a compact local model should not be expected to memorize every file.

Hardware -> CPU-only, 16 GB

Hardware: CPU-only, 16 GB, which points to a 7B-13B model at 4-bit quantization.

Priority -> privacy

Priority: privacy, which confirms everything should run fully offline, no data ever leaving the laptop, ruling out any cloud API regardless of convenience.

Persona: developer with a 24 GB GPU

The second person is a developer with a 24 GB GPU who wants a coding assistant.

Task -> coding

Task: coding, which wants strong reasoning and code training, not just raw speed.

Hardware -> 32 GB RAM + 24 GB VRAM

Hardware: 32 GB RAM plus 24 GB of VRAM, which comfortably supports 8-bit or larger quantization without the quality loss heavier compression causes.

Priority -> quality

Priority: quality, since a slightly slower but correct suggestion beats a fast wrong one.

Neither person needed agent tooling, but a third persona automating research tasks would add that as a fourth layer on top.

A teaching aid, not a sizing tool

The `recommend()` helper below encodes this same reasoning as a small, deliberately simplified rule-based function: a teaching aid, not a production sizing tool, since it ignores exact model architecture, context length, and runtime overhead.

The decision framework is task -> hardware -> priority, in that order: task sets a capability floor, hardware sets a size ceiling, and priority breaks ties inside that range.
The same three questions produce very different, equally valid stacks for different people -- there is no single correct local AI setup, only a correct process for finding yours.
Write your own answers down. A framework you've applied once to your own real hardware and task is one you'll actually reuse the next time a model or your hardware changes.

Try the working example

python

def recommend(task, ram_gb, vram_gb, priority):
    """Illustrative, rule-based local-AI stack recommendation.

    This is a teaching simplification, not a production sizing tool.
    Real choices depend on the exact model, quantization method,
    runtime, and how much of the model can be offloaded to GPU vs CPU.
    """
    usable_gb = max(ram_gb, vram_gb)

    # Step 1: pick a rough model size band from usable memory.
    if usable_gb >= 40:
        size_band = "13B-34B"
        quant = "8-bit or FP16"
    elif usable_gb >= 16:
        size_band = "7B-13B"
        quant = "4-bit"
    else:
        size_band = "1B-3B"
        quant = "4-bit"

    # Step 2: bias the choice by stated priority.
    if priority == "quality" and usable_gb >= 16:
        quant = "8-bit" if usable_gb >= 24 else quant
    if priority == "speed":
        quant = "4-bit"
    if priority == "low_memory":
        size_band = "1B-3B" if usable_gb < 16 else size_band
        quant = "4-bit"

    # Step 3: task-specific add-ons.
    needs_rag = task == "documents"
    needs_agent = task == "agent"
    privacy_note = "safe to run fully offline" if priority == "privacy" else None

    runtime = "GPU-accelerated (llama.cpp/vLLM with GPU offload)" if vram_gb >= 6 \
        else "CPU-only (llama.cpp)"

    return {
        "task": task,
        "size_band": size_band,
        "quantization": quant,
        "runtime": runtime,
        "needs_rag": needs_rag,
        "needs_agent": needs_agent,
        "privacy_note": privacy_note,
    }


personas = [
    ("Small business owner: private document Q&A", "documents", 16, 0, "privacy"),
    ("Developer with a strong GPU: coding assistant", "coding", 32, 24, "quality"),
    ("Student on an old laptop: quick chat helper", "chat", 8, 0, "low_memory"),
]

for label, task, ram, vram, priority in personas:
    result = recommend(task, ram, vram, priority)
    print(label)
    print(f"  inputs: task={task}, ram={ram}GB, vram={vram}GB, priority={priority}")
    print(f"  recommendation: {result}")
    print()

You should see

Running `recommend()` for the three personas prints a dict for each. The business owner (documents, 16 GB RAM, 0 GB VRAM, privacy) gets `{'size_band': '7B-13B', 'quantization': '4-bit', 'runtime': 'CPU-only (llama.cpp)', 'needs_rag': True, 'needs_agent': False, 'privacy_note': 'safe to run fully offline'}`. The developer (coding, 32 GB RAM, 24 GB VRAM, quality) gets `{'size_band': '7B-13B', 'quantization': '8-bit', 'runtime': 'GPU-accelerated (llama.cpp/vLLM with GPU offload)', 'needs_rag': False, 'needs_agent': False}`. The student (chat, 8 GB RAM, 0 GB VRAM, low_memory) gets `{'size_band': '1B-3B', 'quantization': '4-bit', 'runtime': 'CPU-only (llama.cpp)', 'needs_rag': False, 'needs_agent': False}` -- three different stacks from three different inputs, exactly as the framework predicts.

5-minute try-it

Run the three-question framework for your own real hardware and your own real main task. First, write down your answers to all three questions: task (chat / coding / documents / agent), hardware (CPU-only / integrated GPU / dedicated GPU, plus your actual RAM and VRAM in GB), and priority (speed / quality / privacy / low memory footprint). Then call `recommend()` with your own numbers, or reason through it by hand using the same logic. Write down the resulting stack: a model size range, a quantization level, which runtime you'd use, and whether you'd add RAG (if you have documents you want to query) or agent tooling (if you want the model to take multi-step actions). Finally, write one sentence justifying why your priority answer pushed the recommendation in the direction it did -- that justification is the actual skill this course was teaching, more than any specific model name.

One important caution

Treating this exercise's numbers as the answer -- the framework and the reasoning process are what's reusable; specific model names, sizes, and quantization formats will be outdated within a year or two.

Skipping the priority question and defaulting to 'the best quality I can afford' -- for many real tasks, privacy or low memory footprint is the actual constraint, and ignoring it produces a stack you can't actually run.

Ollama Model Library — Local AI / Local LLM

Local AI Glossary — Common Terms

Term	Meaning
AI	Artificial Intelligence: the broad field of building systems that perform tasks normally requiring human intelligence, like understanding language or recognizing images.
ML	Machine Learning: a subfield of AI where systems learn patterns from data instead of being explicitly programmed with rules.
LLM	Large Language Model: a machine learning model trained on huge amounts of text to predict and generate human-like language.
Model	The trained files that hold everything an LLM learned; running it against input produces output.
Weights	The numeric values inside a model, learned during training, that determine what it predicts for any given input.
Parameters	The individual learned weights and biases in a model, counted in millions or billions, which roughly indicates model size.
Inference	The process of running a trained model on new input to produce an output, as opposed to training it.
Training	The process of adjusting a model's weights by showing it huge amounts of data until it learns useful patterns.
Fine-tuning	Continuing to train an already-trained model on a smaller, specific dataset to specialize its behavior.
Prompt	The text input you give a model to get a response, including instructions, questions, or context.
System Prompt	A hidden or fixed instruction set given to a model before the user's message, shaping its overall behavior and rules.
Token	A small chunk of text, roughly a word or part of a word, that is the basic unit a model reads and generates.
Tokenizer	The component that splits text into tokens and converts them into the numeric IDs a model actually processes.
Context Window	The maximum number of tokens a model can consider at once, including the prompt, conversation history, and its own output.
Quantization	Storing a model's weights at lower numeric precision to shrink its memory footprint, usually with some quality tradeoff.
GGUF	A file format designed for running quantized models efficiently on CPUs and consumer hardware, used by tools like llama.cpp.
Safetensors	A safer, faster file format for storing model weights that avoids the security risks of Python's older pickle format.
Embedding	A numeric vector representation of text (or other data) that captures its meaning, so similar meanings end up as similar numbers.
Vector	A list of numbers representing a point in multi-dimensional space, used to represent embeddings mathematically.
Vector Database	A database optimized to store embeddings and quickly find the most similar ones to a given query vector.
RAG	Retrieval-Augmented Generation: looking up relevant documents first, then giving them to the model as context so its answer is grounded in real data.
Chunking	Splitting long documents into smaller pieces before embedding them, so retrieval can return focused, relevant sections.
API	Application Programming Interface: a defined way for one piece of software to request functionality or data from another.
Endpoint	The specific URL or address an application sends requests to in order to use an API's functionality.
localhost	A name that always refers to your own machine, used when running a local server or model that only your computer can reach.
CPU	Central Processing Unit: the general-purpose processor in a computer, able to run a local model but usually slower than a GPU for this task.
GPU	Graphics Processing Unit: a processor built for massive parallel math, which makes it dramatically faster than a CPU at running AI models.
NPU	Neural Processing Unit: dedicated chip hardware built specifically to accelerate AI workloads, increasingly found in modern laptops and phones.
RAM	Random Access Memory: the main working memory of a computer, where a model's weights must fit if it is running on CPU.
VRAM	Video RAM: the dedicated memory on a GPU, which must be large enough to hold a model's weights for GPU-accelerated inference.
KV Cache	Stored intermediate calculations from earlier tokens that a model reuses instead of recomputing, which speeds up generating new tokens but uses extra memory.
TTFT	Time To First Token: how long a user waits after sending a prompt before the model's response starts appearing.
Tokens/sec	A speed measurement showing how many tokens a model generates per second after it starts responding.
Agent	A system built around a model that can plan multi-step actions, call tools, and use their results to decide what to do next.
Tool Calling	A model's ability to recognize it needs an external function, like a calculator or a search, and request it in a structured, machine-readable way.
LoRA	Low-Rank Adaptation: an efficient fine-tuning method that trains a small set of extra parameters instead of updating the whole model.
Open Weights	Model weights that are published for anyone to download and run themselves, as opposed to models only accessible through a paid API.
Prompt Injection	An attack where malicious text hidden in input tricks a model into ignoring its instructions and following the attacker's instead.
Hybrid AI	An approach that combines local and cloud AI, for example running a small model locally and escalating harder tasks to a larger cloud model.