Rotary positional encoding (RoPE) is a technique for injecting positional information into transformer models by rotating the query and key vectors in attention layers, rather than adding position embeddings to inputs. RoPE encodes relative positions naturally, enabling models to generalize to sequence lengths not seen during training. It is used in LLaMA, Mistral, Qwen, and many other modern open-weight models, offering better length generalization than absolute positional encoding.
LLMs & Models
Rotary Positional Encoding (RoPE)
Related terms
Learn to use these concepts in practice.
Join the 212AY Academy →