Head size

Also written: d_k, d_head

How many numbers each query and key holds in one attention head. Scores are divided by its square root.

Other terms from the same lesson

• FlashAttention: A faster way to compute exactly the same attention on a GPU, by working in small tiles so it rarely touches slow memory.
• scaled dot-product attention: The standard attention recipe: compare queries with keys by dot product, divide by √(size), softmax, then mix the values.