Phase 6 · Modern LLM engineering · Inference
Quantization
Store weights in fewer bits to run models cheaply.
Open this lesson Create a free account
Why it matters
A 70-billion-parameter model in 16-bit numbers needs 140 GB of memory, more than any single consumer GPU. Stored in 4 bits it needs about 35 to 40 GB, runs faster, and loses surprisingly little. Quantization is how open models run on laptops, how providers serve models cheaply, and one reason the same model can come in several sizes on a download page.
What you'll be able to do
- Quantize weights to 8, 4, 3 and 2 bits with a scale per group, and measure the error
- Explain why outlier values make per-tensor scales fail, and how groups fix it
- Measure a real model's quality and size at each precision
- Estimate memory and speed gains, and know the common formats (int8, 4-bit, GGUF, FP8)
Before this lesson
Next in this phase: Serving models