Phase 6 · Modern LLM engineering · Adapting models
Distillation
Train a small model to imitate a large one.
Open this lesson Create a free account
Why it matters
Many of the fast, cheap models you use were trained partly by imitating bigger ones. Distillation is how a lab turns an expensive model into an affordable one, how reasoning skills were passed from DeepSeek-R1 to much smaller models, and how a company can turn a prompted frontier model into a small model that does one job well. It's also a reminder that what a model outputs is itself valuable training data, which has legal consequences.
What you'll be able to do
- Explain knowledge distillation: training a small student on a large teacher's outputs
- Use temperature to expose the information in a teacher's full probability distribution
- Train a student with soft targets and compare it with training on hard labels
- Distinguish logit distillation from training on generated text, and know the legal limits
Before this lesson
Next in this phase: Quantization