Glossary · Phase 2 · Math, when you need it
KL divergence
A measure of how different one set of probabilities is from another.
Learn it properly
Cross-entropy and KL divergence · The loss function language models are trained on. About 30 minutes.
Other terms from the same lesson
- cross-entropy: A score for how surprised a model is by the right answer. Lower is better; it's the usual training loss for classifiers and language models.
- perplexity: How many words a language model is effectively choosing between at each step. Lower is better; it's e to the cross-entropy.