Periodic RoPE variants keep long-context accuracy flat to 16k tokens, arXiv paper reports
A new positional encoding method reuses a small set of rotation angles so models trained on short contexts can handle far longer inputs, according to the paper's abstract.
In plain words
Language models track token positions using rotating angles, but they run out of trained angles on long inputs. The proposed methods repeat a fixed set of angles that fit inside a sliding attention window, so the model only ever sees angles it was trained on. In the paper's tests, this kept accuracy stable at 16k tokens where standard approaches degraded.
What was announced
The arXiv paper, by authors of the Periodic RoPE (P-RoPE) proposal, addresses what it calls position exhaustion: rotary position embedding (RoPE) assigns each token a new rotation angle, so a model eventually encounters angles it never saw during training. P-RoPE works alongside sliding window attention by repeating the first window-size W angles of the RoPE table, meaning the model needs only W distinct angles. The authors note this introduces a wrap error at the window boundary, which their second method, Harmonic Periodic RoPE (HP-RoPE), removes by choosing frequencies that complete whole turns within W tokens.
In comparisons where models share the same architecture, data, and schedule and differ only in the window-layer position encoding, the authors report that P-RoPE and HP-RoPE keep perplexity nearly flat from the training length of 768 to 16k, while RoPE, NTK, and YaRN rise. In a format-free Passkey test at 16k, the periodic variants remained at 99–100 percent, against 11–37 percent for the RoPE interpolations. The code is available on GitHub, per the abstract.
What it does not show
- The evidence is the paper's abstract only; the full article's methods, experiments, and peer review status are not covered.
- All results, including the perplexity and Passkey figures, are the authors' own reported measurements and have not been independently confirmed.
- The abstract does not state model sizes, training costs, or whether the method applies to existing deployed models.
For security and IT teams
- Scope
- The reported gains apply to models using sliding window attention with the window-layer position encoding changed; comparisons held architecture, data, and schedule fixed.
Sources
Source-support review completed 21 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 21 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.