Glossary · Phase 6 · Modern LLM engineering
QLoRA
LoRA on top of a model whose frozen weights are stored in 4 bits, so a big model fits on one GPU for fine-tuning.
Learn it properly
LoRA · Fine-tune a big model by training small low-rank updates. About 40 minutes.
Other terms from the same lesson
- adapter: A small set of extra trained weights added to a frozen model to change its behavior. Easy to swap, share and remove.
- LoRA: Low-rank adaptation: fine-tuning by training a small add-on instead of all the weights, which is far cheaper.
- rank: In LoRA, the width of the two thin tables that make up the update. Higher rank can learn more but costs more.