Phase 2 · Math, when you need it · Information
Softmax
Turn any list of scores into probabilities.
Open this lesson Create a free account
Why it matters
A network's last layer produces raw scores, one per option: "great" 2.1, "good" 1.3, "bad" −0.4. Softmax turns them into probabilities that are positive and add up to 1. It sits at the end of every language model (one score per token in the vocabulary) and in the middle of every transformer (attention weights). The temperature setting you see in AI tools is a dial on this exact function.
What you'll be able to do
- Turn a list of scores (logits) into probabilities with softmax
- Implement softmax so it never overflows
- Explain how temperature sharpens or flattens the distribution
- Recognize softmax at the output of classifiers, language models and attention
Before this lesson
Next in this phase: Entropy and surprise