FlashAttention

A faster way to compute exactly the same attention on a GPU, by working in small tiles so it rarely touches slow memory.

Other terms from the same lesson

• head size: How many numbers each query and key holds in one attention head. Scores are divided by its square root.
• scaled dot-product attention: The standard attention recipe: compare queries with keys by dot product, divide by √(size), softmax, then mix the values.