Mathematicians partially prove when attention tokens split into two clusters
An arXiv paper resolves part of a 2024 conjecture about causal attention, showing token dynamics form two clusters when a key eigenvalue is positive, and generally not otherwise.
In plain words
Researchers studying how transformer attention models process tokens proved part of an earlier conjecture: token states settle into two groups rather than one, but only when a specific value, the largest eigenvalue of the value transformation, is positive. When it is not positive, the two-cluster behavior generally fails. The work also shows token paths effectively reduce to a two-dimensional structure.
What was announced
The authors, publishing on arXiv, partially resolve Conjecture 1 of Karagodin et al. from NeurIPS 2024 and extend it to time-dependent parameter regimes. The conjecture predicted that causal attention dynamics converge to two clusters aligned with eigenvectors of the largest eigenvalue of the value transformation. The paper proves this holds when that eigenvalue is positive and is generally false when it is zero or negative, describing the two-cluster regime as common in practice and harder to analyze than the single-cluster case studied before.
A second contribution is a dimension reduction result: although tokens evolve in a high-dimensional embedding space, their trajectories collapse onto a two-dimensional structure governed by a few dominant directions. The authors also introduce an analytical framework, called instability within the flow and inspired by physical systems, which they say enables analysis of steady-regime behavior in non-monotone, non-autonomous attention dynamics. The theory is illustrated with simulations and experiments on Group Query Attention layers.
What it does not show
- The source is only the paper's abstract, so the proofs, assumptions and experimental details are not visible in the supplied evidence.
- The claim that the two-cluster regime is common in practice rests on the authors' own statement and is not independently confirmed.
Sources
Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 17 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.