Phase 5 · Transformers · Attention
Multi-head attention
Several attention patterns in parallel.
Open this lesson Create a free account
Why it matters
A single attention head gives each token one weighted blend of the others. But a token needs several kinds of context at once: the previous word, the subject of the sentence, the noun a pronoun refers to. Multi-head attention runs several attention computations side by side, each free to learn a different relation, for the same parameter budget. Every transformer uses it, and "heads" are a unit researchers study when they look inside LLMs.
What you'll be able to do
- Explain why one attention pattern per token isn't enough
- Split the model width into several heads, run attention in each, and recombine
- Track shapes through split and merge, and count the parameters
- Recognize the kinds of patterns different heads learn
Before this lesson
Next in this phase: Positional encodings