LoRA

Why it matters

Fully fine-tuning an 8-billion-parameter model needs well over 100 GB of GPU memory and produces a new 16 GB copy of the model for every task. LoRA trains less than 1% of that, fits on one GPU, and produces adapters measured in megabytes that can be swapped on top of one shared base model. It's why fine-tuning open models became practical for small teams, and it's the default way they're customized today.

What you'll be able to do

• Explain how LoRA replaces a full weight update with two small low-rank matrices
• Count the trainable parameters and memory LoRA saves on a real model size
• Implement a LoRA layer, fine-tune with it, and merge it back into the weights
• Choose rank, scaling, target layers and learning rate, and know when QLoRA helps

Before this lesson

• Supervised fine-tuning
• SVD and PCA

Next in this phase: Fine-tune an open model