1 min lesson
Positional information
Choose two examples from the table in "Positional information" and explain what each teaches you to do.
Step 1 of 2
Positional informationRoPE vs learned/absolute
Attention is permutation-invariant on its own, so position has to be injected. Learned absolute embeddings add a per-position vector and break the moment you exceed the trained length. RoPE (rotary position embeddings) instead rotates the query and key vectors by an angle proportional to position, so attention depends on the relative offset between tokens.
- Scheme
- Learned absolute
- How position enters
- Add a trained vector per index
- Long-context behavior
- Hard cap at trained length; no extrapolation
- Scheme
- Sinusoidal absolute
- How position enters
- Add fixed sin/cos of position
- Long-context behavior
- Extrapolates weakly; rarely used now
- Scheme
- RoPE (rotary)
- How position enters
- Rotate Q/K by position angle
- Long-context behavior
- Relative by construction; extends via base/NTK scaling
| Scheme | How position enters | Long-context behavior |
|---|---|---|
| Learned absolute | Add a trained vector per index | Hard cap at trained length; no extrapolation |
| Sinusoidal absolute | Add fixed sin/cos of position | Extrapolates weakly; rarely used now |
| RoPE (rotary) | Rotate Q/K by position angle | Relative by construction; extends via base/NTK scaling |
RoPE is the default in state-of-the-art code models because long agent-assisted sessions need context that extends past the trained window.
A Cursor agent editing a real repo carries file contents, tool outputs and many turns of history. Relative position via RoPE matters because the model must reason about a function defined 8,000 tokens earlier the same way whether it sits at position 8k or 80k. Absolute schemes would treat those as different.