Glossary · Phase 6 · Modern LLM engineering
GGUF
The file format llama.cpp and tools built on it use for quantized models you run on your own computer.
Learn it properly
Quantization · Store weights in fewer bits to run models cheaply. About 35 minutes.
Other terms from the same lesson
- AWQ: Activation-aware weight quantization: keeps the few weights that matter most for real inputs more precise.
- FP8: Floating-point numbers stored in 8 bits (FP4: 4 bits), used to run or train models faster and in less memory than 16-bit numbers.
- GPTQ: A popular way to quantize a model's weights after training, adjusting the rounding so the layer's outputs change as little as possible.
- quantization: Storing a model's numbers with fewer digits so it uses less memory and runs faster, at a small cost in quality.