Online softmax

Computing a softmax one chunk at a time with a running maximum and a running total, so the full row of scores never has to be stored.

Other terms from the same lesson

• position interpolation: Stretching a model's context by squeezing the new, longer positions into the range it was trained on, then fine-tuning it briefly.
• sliding-window attention: Attention where each token only looks at a fixed number of recent tokens, so the cost grows in step with the length instead of with its square.
• state-space model: An alternative to attention that carries a fixed-size summary forward as it reads, so long texts cost no more per token. Mamba is the best known.