Phase 5 · Transformers · Attention
Attention: the intuition
Each token decides which other tokens matter to it.
Open this lesson Create a free account
Why it matters
"Attention is all you need" is the title of the 2017 paper that introduced the transformer, and attention is the mechanism that lets a language model connect "it" to the right noun, a pronoun to a name three paragraphs back, or an instruction at the top of a prompt to the answer at the bottom. Before the formulas, this lesson builds the intuition with a real (if simple) example you can see.
What you'll be able to do
- Explain why a word's meaning depends on other words, possibly far away
- Describe attention as a weighted average where each token chooses its own weights
- Compute content-based attention weights from similarity and a softmax
- Say what attention fixes compared with fixed windows and recurrent networks
Before this lesson
Next in this phase: Queries, keys and values