Glossary · Phase 6 · Modern LLM engineering
Distillation
Training a small model to imitate a big one, keeping much of its skill at lower cost.
Learn it properly
Distillation · Train a small model to imitate a large one. About 35 minutes.
Other terms from the same lesson
- soft labels: A teacher's full spread of probabilities over the answers, which carries more information than just the single right answer (a hard label).
- student model: In distillation, the smaller model trained to copy a larger teacher model.
- teacher model: In distillation, the large model whose answers or probabilities a smaller student model learns to copy.