Scaled dot-product attention

The standard attention recipe: compare queries with keys by dot product, divide by √(size), softmax, then mix the values.

Other terms from the same lesson

• FlashAttention: A faster way to compute exactly the same attention on a GPU, by working in small tiles so it rarely touches slow memory.
• head size: How many numbers each query and key holds in one attention head. Scores are divided by its square root.