Quantization: Making Models Smaller

Build the mental model

In lesson 3 you saw that a 7B model needs roughly 13GB of memory when its numbers are stored at full precision. Many people do not have 13GB of free VRAM sitting around, so a real question follows: can we shrink the model without throwing it away and starting over? The answer is quantization - storing each of the model's internal numbers using fewer bits than it was originally trained with.

Models are usually trained in FP32 (32-bit floating point) or FP16 (16-bit floating point), which give very fine-grained precision. Quantization converts those numbers down to formats like INT8 (8-bit whole numbers) or even INT4 (4-bit whole numbers) - roughly like rounding a very precise measurement to fewer decimal places.

Fewer bits per number means the whole model file takes up dramatically less memory, and that is exactly why quantization is what makes running large models on ordinary consumer hardware even possible.

But this is not a free lunch. Rounding always loses some information, and a model that has been quantized aggressively can give noticeably worse answers than the original:

• More factual mistakes
• Weaker reasoning on hard questions
• Odd phrasing

How much quality is lost depends heavily on the specific quantization method used, not just the bit count alone; some clever techniques lose very little quality even at low bit widths, while cruder ones lose more.

The practical rule beginners should hold onto is simple: quantization trades memory savings for some amount of quality, and you should treat that trade as real, not assume it is free.

Quantization

The process of storing a model's numbers with fewer bits, reducing its memory footprint.

FP16

A 16-bit floating point number format, a common precision level for training and running models.

INT4

A 4-bit whole-number format - saves a great deal of memory but usually carries the largest quality tradeoff.

QUANTIZATION PIPELINE
---------------------
[Original Model]  -->  [Quantization]  -->  [Smaller Model]
  FP16, high precision      round each          INT8 or INT4
  ~13 GB for 7B params       number down          ~3-7 GB

  RESULT: less memory needed, faster to load and run,
          but usually some loss in answer quality

Connect it to a real scenario

Picture someone with a laptop that has only 6GB of VRAM who wants to run a 7B model. At full FP16 precision the model needs about 13GB - it simply will not fit. Quantized down to INT4, that same model shrinks to roughly 3-4GB, comfortably fitting on that laptop and running at usable speed.

This is the everyday reason quantization matters: it is not an advanced optimization trick, it is the difference between a model running at all on your hardware or not running at all.

But the tradeoff is real too - if that person's task is legal document review where small factual errors genuinely matter, choosing the most aggressively quantized version just to save memory could introduce mistakes they cannot afford.

The practical skill is knowing when to accept the quality dip for the memory savings, and when the task is important enough that a less aggressive quantization, or more memory, is worth the cost.

• Quantization stores a model's numbers using fewer bits
• It dramatically reduces memory needs
• There is a real quality tradeoff that depends on the specific method used

Try the working example

def estimate_memory_gb(params_billion, bytes_per_param):
    total_bytes = params_billion * 1_000_000_000 * bytes_per_param
    return total_bytes / (1024 ** 3)


precisions = [
    ("FP16", 2),
    ("INT8", 1),
    ("INT4", 0.5),
]

params_b = 7
print(f"Model size: {params_b}B parameters")
for name, bytes_per_param in precisions:
    gb = estimate_memory_gb(params_b, bytes_per_param)
    print(f"  {name}: ~{gb:.2f} GB")

Model size: 7B parameters
  FP16: ~13.04 GB
  INT8: ~6.52 GB
  INT4: ~3.26 GB

Hugging Face docs: Quantization — Local AI / Local LLM