<aside> 📢
<aside> 🎯
<aside>
Standard Transformers originally relied on absolute positional encodings (sinusoidal or learned vectors) added directly to token embeddings. These approaches provide position information but do not explicitly model relative distances between tokens, making length extrapolation difficult when inference sequences exceed the training distribution.
RoPE and ALiBi address this limitation by injecting positional information directly into the attention mechanism.
Instead of adding positional embeddings to token representations, RoPE applies a position-dependent rotation to Query (Q) and Key (K) vectors before the attention dot product.
RoPE divides the embedding space into $d/2$ two-dimensional pairs. Each pair is treated as a 2D plane and rotated by an angle determined by:
The attention score becomes:
$$ \langle R_{\Theta,m}q, R_{\Theta,n}k\rangle $$
which has the important property:
q^T R_{\Theta,m-n} k $$
This means that although RoPE uses absolute positions when rotating Q and K, the resulting attention interaction naturally incorporates the relative distance $(m-n)$ between tokens.
Important clarification: RoPE does not remove absolute position information. Instead, it transforms absolute positions into relative position awareness through the attention computation.
ALiBi removes positional embeddings completely.
Instead, it adds a fixed distance-based bias directly to the attention logits before softmax:
q_i k_j^T - m_h |i-j| $$
where:
Different attention heads use different slopes, allowing different heads to learn different preferred distance ranges.
ALiBi introduces a locality bias: nearby tokens receive less penalty, while distant tokens receive larger penalties.
However, distant tokens are not forbidden from receiving attention. If semantic similarity is strong enough, the model can still attend to far-away tokens.
ALiBi introduces a fixed locality prior:
This can make retrieval-heavy tasks and multi-hop reasoning more challenging because distance bias may compete with content-based attention.
RoPE can struggle with long-context extrapolation.
When a model is extended beyond its training context length, the rotation phases and relative angular relationships may enter regions that were not observed during training.
Techniques such as:
are used to recover long-context performance.
\begin{pmatrix} \cos(m\theta_1)&-\sin(m\theta_1)&0&0\\ \sin(m\theta_1)&\cos(m\theta_1)&0&0\\ 0&0&\cos(m\theta_2)&-\sin(m\theta_2)\\ 0&0&\sin(m\theta_2)&\cos(m\theta_2) \end{pmatrix} $$
Each 2D dimension pair has its own rotation frequency.
q^TR_{\Theta,m-n}k $$
The attention interaction depends on the relative position difference between tokens.
$$ \theta_i=b^{-2(i-1)/d} $$
where:
Earlier dimension pairs use higher frequencies, while later dimension pairs use lower frequencies.
High-frequency components change quickly and are more sensitive to short positional differences.
Low-frequency components change slowly and help represent longer positional relationships.
Note: These dimensions are not explicitly assigned as "local" or "global" features. The model learns how to use different frequency components during training.
q_i k_j^T - m_h |i-j| $$
Each attention head receives a different fixed slope.
The slopes follow a geometric progression across heads, creating multiple distance preferences.
RoPE divides the $d$-dimensional embedding space into $d/2$ two-dimensional rotation pairs.
Each pair rotates according to:
$$ angle=m\theta_i $$
where:
<aside> <img src="/icons/reorder_gray.svg" alt="/icons/reorder_gray.svg" width="40px" />
</aside>