RoPE

Also written: rotary position embedding, rotary position embeddings, rotary embeddings

Rotary position embeddings: marking word order by turning each query and key by an angle that grows with position, so attention scores depend on how far apart two tokens are. Most modern open models use it.

Other terms from the same lesson

• positional encoding: Extra numbers that tell the model where each token sits in the sequence.