Causal masking

Why it matters

A language model learns to predict the next token. If, while predicting token 5, attention could peek at token 5 itself, training would be trivial and the model would learn nothing usable. The causal mask is the small triangle of −∞ that prevents this, and it's also what makes training efficient: one pass over a sequence gives a prediction, and a loss, at every position.

What you'll be able to do

• Explain why a language model must not attend to future tokens
• Build the lower-triangular causal mask and apply it to attention scores
• Verify causality: changing a later token can't change earlier outputs
• See how masking lets one forward pass train on every position at once

Before this lesson

• Scaled dot-product attention

Next in this phase: Multi-head attention