Glossary · Phase 5 · Transformers
Validation loss
The model's average error on held-back text it never trains on. If it rises while training loss keeps falling, the model is memorizing.
Learn it properly
Build a tiny GPT · Write and train a small GPT from scratch. About 45 minutes.
Other terms from the same lesson
- training loss: The model's average error on the examples it learns from. It should fall as training goes on.