Hardware Basics: CPU, GPU, RAM, VRAM

Build the mental model

A model file is useless sitting on a hard drive - something has to load it into working memory and do the actual arithmetic. That 'something' is a mix of hardware parts, and understanding what each one does removes a lot of the mystery around running AI locally.

The CPU (central processing unit) is your computer's general-purpose brain - it can do anything, including run a model, but it processes instructions mostly one after another.

The GPU (graphics processing unit) was originally built for rendering video game graphics, and it turns out that job requires doing the same simple math on huge amounts of data at once - which is exactly what running a neural network needs too. So a GPU can run a model dramatically faster than a CPU, though a CPU can still do it, just more slowly.

Some newer devices also include an NPU (neural processing unit), a chip built specifically for this kind of AI math.

Memory works the same way in two tiers. RAM is your computer's general working memory, shared by every program you have open. VRAM is memory that lives directly on the GPU, reserved just for graphics and GPU calculations, and it is usually much faster to access than RAM but far more limited in size.

When you run a model on the GPU, it needs to fit inside VRAM; when you run it on the CPU, it needs to fit inside RAM instead. Neither path is wrong - a GPU with plenty of VRAM gives you speed, but a CPU-only laptop can still run smaller models comfortably, just with a longer wait for each answer.

CPU: The computer's general-purpose processor, which runs most instructions largely one after another.

GPU: A chip originally built for graphics that also happens to be very good at the parallel math AI models need.

NPU: A neural processing unit - a chip purpose-built for AI math, found in some newer devices.

RAM: The computer's general working memory, shared by every program currently running.

VRAM: Memory that lives directly on the GPU - faster than RAM to access, but usually much smaller in size.

Storage: The hard drive or SSD where the model file sits before it is loaded - not directly used during actual computation like RAM/VRAM.

text

WHERE A MODEL CAN LOAD
----------------------
YOUR DEVICE
-----------
  [CPU] <---> [RAM]
    |            model can load here (slower, always works)
    |
  [GPU] <---> [VRAM]   (optional, if present)
                 model can load here (faster, limited size)

  STORAGE (disk): where the model file sits before loading

Connect it to a real scenario

Say a hobbyist wants to try running a local model on the laptop they already own - no budget for a new GPU. Knowing that CPU-only inference genuinely works, just slower, means they can start today with a small model instead of assuming they need to buy anything first.

Compare that to a developer who has a gaming PC with a GPU and 12GB of VRAM sitting mostly idle outside of gaming sessions - for them, checking VRAM size before picking a model tells them immediately whether a given model will run fast on the GPU or need to fall back to slower CPU memory.

In both cases, the practical skill is the same: before downloading any model, look at what hardware you actually have - how much RAM, whether there is a GPU and how much VRAM it carries - so you pick a model size that matches your machine instead of discovering the mismatch after a failed download.

A GPU helps with speed but is not strictly required
RAM is the whole computer's memory; VRAM belongs only to the GPU
Check your available hardware before downloading a model

Try the working example

python

def estimate_memory_gb(params_billion, bytes_per_param):
    """Rough rule of thumb: memory needed = parameter count x bytes
    per parameter. Real usage is a bit higher (activations, overhead)
    but this gives a useful ballpark before downloading a model."""
    total_bytes = params_billion * 1_000_000_000 * bytes_per_param
    return total_bytes / (1024 ** 3)


models = [
    ("TinyLlama 1.1B (FP16)", 1.1, 2),
    ("Llama 7B (FP16)", 7, 2),
    ("Llama 70B (FP16)", 70, 2),
]

for name, params_b, bytes_per_param in models:
    gb = estimate_memory_gb(params_b, bytes_per_param)
    print(f"{name}: ~{gb:.2f} GB")

You should see

TinyLlama 1.1B (FP16): ~2.05 GB
Llama 7B (FP16): ~13.04 GB
Llama 70B (FP16): ~130.39 GB

5-minute try-it

Add ("Mistral 7B (FP16)", 7, 2) to the models list and re-run. Then try changing bytes_per_param to 4 (FP32) and observe how much the estimated memory grows.

One important caution

Assuming you absolutely need a GPU to run a local LLM - CPU-only inference works, just slower

Confusing RAM with VRAM and picking a model size that does not actually fit in your available VRAM

Wikipedia: Graphics processing unit — Local AI / Local LLM