Phase 5 · Transformers · The transformer
Positional encodings
Tell the model where each token sits.
Open this lesson Create a free account
Why it matters
"The dog bit the man" and "the man bit the dog" contain the same words. Attention, as built so far, can't tell them apart: it treats its input as a set. Every transformer therefore injects information about where each token sits, and the method chosen (learned, sinusoidal, rotary) is one of the main things that determines how long a context a model can handle.
What you'll be able to do
- Show that attention on its own ignores word order
- Add learned or sinusoidal position embeddings to token embeddings
- Explain rotary position embeddings (RoPE) and why they encode relative distance
- Connect position methods to how far a model's context can stretch
Before this lesson
Next in this phase: The transformer block