Glossary · Phase 5 · Transformers
Attention
How a transformer lets each token look at the other tokens and pull in the information it needs.
Learn it properly
Attention: the intuition · Each token decides which other tokens matter to it. About 30 minutes.
Other terms from the same lesson
- attention weights: The shares of focus a token gives to each earlier token. They are all positive and add up to 1.