Phase 2 · Math, when you need it · Information
Cross-entropy and KL divergence
The loss function language models are trained on.
Open this lesson Create a free account
Why it matters
This is the loss function language models are trained on. Every time you read "the model reached a loss of 1.8" or "perplexity 6", it's cross-entropy. KL divergence, its close relative, keeps fine-tuned models from drifting too far from the original in RLHF, and measures how well a small model mimics a big one in distillation. After this lesson Phase 2 is complete: you have the math to read how models are trained.
What you'll be able to do
- Compute cross-entropy between a true distribution and a model's prediction
- Show that with one correct answer, cross-entropy is −log of the probability given to it
- Compute KL divergence and explain it as the extra cost of using the wrong distribution
- Turn an average loss into perplexity