LLMs rank party stances well but struggle to place them on a scale
A new arXiv study finds language models can order Japanese and South Korean parties by documented positions, yet fail when asked to locate a single party on the same scale.
In plain words
Researchers tested whether language models truly know where political parties stand on issues. The models could say which of four parties ranks higher than another, but could not accurately put one party at a specific point on a scale. Giving the models the official coded numbers fixed that gap, so ranking skill and scale knowledge appear to be separate abilities.
What was announced
The study, posted as an arXiv preprint, examines whether models hold a genuine measurement of social positions or only a sense of relative order. The authors argue that text about social entities contains many comparisons but few anchored values, so a model may learn reliable rankings without any underlying scale. To test this, they used party positions in Japan and South Korea, where coded data from the Comparative Manifesto Project provides an external answer key.
The results split sharply by question type. When asked to order four parties, the models outperformed the strongest fixed-answer baseline by more than a factor of two, but when asked to place a single party on the same scale, they scored below that baseline. Ordering ability still showed through inside the failed placements, and once the coded values were supplied, placement accuracy rose to near ceiling. The authors conclude that comparative and absolute questions about the same fact measure different skills, so a benchmark built from one type cannot stand in for the other when testing party issue salience.
What it does not show
- The evidence is the paper's abstract only, so methods, model versions and full results are not available for verification.
- Findings cover party positions in Japan and South Korea; the abstract does not establish how the pattern extends to other domains or countries.
- The reported performance gaps come from the authors' own tests and have not been independently confirmed.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.