Phase 2 · Math, when you need it · Information
Entropy and surprise
Measure uncertainty in bits.
Open this lesson Create a free account
Why it matters
Information theory gives a precise number for uncertainty, measured in bits. It explains why "the" is easy to predict and a product code isn't, why compression works, and, most important here, it's the foundation of the loss function every language model is trained with (cross-entropy, the next lesson). It also gives you a way to say how confident a model's next-token distribution is.
What you'll be able to do
- Measure the surprise of an event in bits
- Compute the entropy of a distribution as its average surprise
- Explain why entropy is highest for uniform distributions and zero for certain ones
- Use entropy to describe how uncertain a model's prediction is
Before this lesson
Next in this phase: Cross-entropy and KL divergence