Exercise: Diagnose a Slow Local Setup

Build the mental model

Diagnosing a slow or barely-functional local AI setup is a skill in itself, separate from knowing how to install a runtime or download a model. The core diagnostic question is always a matching problem: does the hardware you have match the model size and quantization level you chose, and does the model's capability match the difficulty of the task you are asking it to do?

Two very different symptoms point to two very different mismatches. If a model loads slowly, crashes, or the machine grinds to a halt swapping to disk, that is almost always a memory mismatch: the model's weights, at the precision you downloaded them in, simply do not fit in the RAM or VRAM available.

If instead the model loads fine and responds quickly but gives shallow, wrong, or repetitive answers, that is a capability mismatch: you picked a model too small, or too aggressively quantized, for the reasoning the task demands.

The fix for each is different, so figuring out which one you are facing matters before you touch any settings. The estimation method from the Basic chapter, parameter count multiplied by bytes-per-parameter and converted to gigabytes, is the tool for the first kind of mismatch: it turns a vague symptom like "it feels slow" into a concrete number you can compare directly against your hardware.

This lesson walks through one realistic case of that memory mismatch in detail, with real numbers, so you practice the calculation yourself instead of just reading about it. The same habit, estimate before you run, is what separates confident troubleshooting from trial-and-error guessing whenever something in your local AI stack does not behave the way you expected.

text

MEMORY MISMATCH: 70B MODEL VS 16 GB LAPTOP RAM
----------------------------------------------
MEMORY MISMATCH: 70B MODEL AT FP16 VS 16 GB LAPTOP RAM
-----------------------------------------------------------
REQUIRED  (70B params x 2 bytes/param, converted to GB)
  [########################################] 130.4 GB

AVAILABLE (laptop RAM, no dedicated GPU)
  [#####]                                     16.0 GB

  0                                              140 GB
  |----|----|----|----|----|----|----|----|----|

GAP: about 114.4 GB more than the laptop actually has

Connect it to a real scenario

In practice, this diagnostic habit saves you from two common traps. The first trap is trial-and-error hardware shopping: buying more RAM, or a bigger GPU, before you have actually confirmed that memory is the bottleneck.

A five-minute calculation, done before you spend money, tells you exactly how much headroom you need and whether a cheaper fix, like a lower quantization level or a smaller model, gets you there instead.

The second trap is blaming the wrong layer of the stack. A model that barely fits in memory can look like a "buggy install" or a "slow runtime" when the real problem is that the operating system is constantly swapping model weights to disk, which is dramatically slower than RAM. Running the estimate first tells you whether you are even in a fair fight before you start tweaking configuration files.

Look up the parameter count

Find the parameter count of the model you are considering.

Decide on a quantization level

Decide which quantization level you would use.

Run the same arithmetic

Run the same estimation arithmetic used in this lesson.

Compare to your actual hardware

Compare the result to the RAM or VRAM you actually have, not the RAM you wish you had.

Treat the number it gives you as a floor, not a guarantee, since the operating system, other running applications, and the runtime's own overhead all eat into that same pool of memory too.

A slow or crashing local model is usually a memory mismatch; a fast but low-quality model is usually a capability mismatch -- diagnose which one you have before changing anything.
The estimation formula (parameters x bytes-per-parameter -> GB) turns a vague complaint like 'it's slow' into a concrete number you can compare against your actual RAM or VRAM.
Treat the estimate as a floor, not a ceiling -- KV cache, context length, the operating system, and other running applications all add to the real memory footprint.

Try the working example

python

def estimate_memory_gb(param_count_billions, bytes_per_param):
    """Estimate RAM/VRAM needed to load a model's weights.

    param_count_billions: number of parameters, in billions (e.g. 70 for
                           a 70B-parameter model)
    bytes_per_param: bytes used to store each parameter (2 for FP16/BF16
                      full precision, 1 for 8-bit quantization, ~0.5 for
                      4-bit quantization)
    Returns the estimated size in gigabytes (GiB).
    """
    params = param_count_billions * 1_000_000_000
    total_bytes = params * bytes_per_param
    return total_bytes / (1024 ** 3)


def check_fit(required_gb, available_gb):
    fits = required_gb <= available_gb
    shortfall = max(0.0, required_gb - available_gb)
    return fits, shortfall


# The scenario: a 70B-parameter model downloaded at FP16 (2 bytes/param),
# on a laptop with 16 GB of RAM and no dedicated GPU.
required = estimate_memory_gb(param_count_billions=70, bytes_per_param=2.0)
available = 16
fits, shortfall = check_fit(required, available)

print(f"Estimated requirement: {required:.1f} GB")
print(f"Available RAM:         {available} GB")
print(f"Fits in available memory? {fits}")
print(f"Shortfall: {shortfall:.1f} GB")

You should see

Running the function against the scenario prints: an estimated requirement of 130.4 GB (70 billion parameters x 2 bytes each, converted to gigabytes), available RAM of 16 GB, `Fits in available memory? False`, and a shortfall of 114.4 GB -- confirming this is a severe memory mismatch, not a minor slowdown.

5-minute try-it

A colleague downloads a 70B-parameter open-weights model at FP16 (2 bytes per parameter) onto their laptop, which has 16 GB of RAM and no dedicated GPU. Loading takes several minutes, the fans spin at full speed, and once it finally responds it produces one token every few seconds. Before changing anything, calculate the estimated memory requirement for this model at FP16 using the method in this lesson, and compare it to the 16 GB available. Then work out two alternative configurations that would actually fit in 16 GB of RAM: one where you keep the same 70B model but change the quantization level, and one where you keep FP16 precision but change the parameter count. For each alternative, state the resulting estimated memory requirement. Which of the two alternatives would you recommend, and why -- consider not just whether it fits, but what quality tradeoff each one makes.

One important caution

Assuming the estimate is the full story -- actual memory usage also includes the KV cache, context length, and runtime overhead, so real usage typically runs higher than the raw weights calculation.

Jumping straight to buying more hardware instead of first checking whether a different quantization level or a smaller model solves the problem for free.

Hugging Face -- Model Memory Calculator — Local AI / Local LLM