Phase 5 · Transformers · Attention
Scaled dot-product attention
The attention formula, line by line.
Open this lesson Create a free account
Why it matters
This one line is the heart of every transformer:
What you'll be able to do
- Read the attention formula softmax(QKᵀ/√d_k)V term by term
- Show why dividing by √d_k keeps the softmax from saturating
- Implement batched attention in PyTorch and check every shape
- Recognize the function libraries provide for this (and why it's optimized)
Before this lesson
Next in this phase: Causal masking