Linearized 2-simplicial attention cuts cubic cost to quadratic
A new arXiv paper rewrites 2-simplicial attention so a model can run without any softmax attention, using a fixed-size state instead.
In plain words
Some attention methods compare three things at once, which normally gets very expensive as text gets longer. The authors rewrote the math so it can be approximated and stored compactly, making the cost grow much more slowly. In their test, a model built this way slightly beat a comparable hybrid model.
What was announced
Researchers report on arXiv a linearized form of 2-simplicial attention, a method whose trilinear score is rewritten as an inner product between a composite query and a key so the sum over keys matches ordinary softmax attention. They approximate this sum with positive random features and keep all previous keys in a fixed-size state, while the sum over second keys remains exact across all previous tokens. According to the paper, this lowers the cost from cubic to quadratic in sequence length, and both keys can range over all previous tokens, which windowed 2-simplicial attention does not allow.
The authors implemented the approach with custom Triton kernels and combined it with Kimi Delta Attention to build a model containing no softmax attention at all. Trained on 15B equal training tokens from FineWeb-Edu, the model reportedly lowered validation loss from 2.35 to 2.32 compared with a KDA hybrid, and reduced LAMBADA perplexity from 40 to 38. These figures come from the paper's own experiments and have not been independently verified.
What it does not show
- The evidence is the paper's abstract only; the full article's methods, experiments and caveats were not reviewed.
- Reported loss and perplexity gains rest on the authors' own 15B-token experiments and are not independently confirmed.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.