Polar benchmark measures political bias in 38 LLMs across US and South Korea
A 4,026-item multiple-choice benchmark from arXiv researchers scores bias via option likelihoods, finding all tested models lean left-progressive on US content but more centered on South Korean material.
In plain words
Researchers built Polar, a large set of multiple-choice questions about politics, to measure whether AI language models favor one political side. Instead of asking models to write answers, it checks which options the models statistically prefer. Testing 38 models showed bias shifts depending on the country, topic, and language involved.
What was announced
The benchmark draws its questions from the Manifesto Project and covers two ideological axes and eight issue categories, running each model through parallel US and South Korean political contexts. Rather than relying on generated text, Polar scores bias through option-level likelihoods, a design the authors say makes measurements more reproducible across political and linguistic settings.
Across 38 large language models, measured bias varied systematically with political context, issue category, model group, and presentation language, according to the paper. All models leaned left-progressive on US political content but showed more centered and mixed patterns on South Korean content. Translation experiments indicated that presentation language alone could shift the measured bias. The authors argue this supports multilingual, cross-contextual evaluation, and they have released the code and dataset on GitHub and Hugging Face.
What it does not show
- The evidence is limited to the paper's abstract; full methodology, scoring details, and per-model results are not available in the supplied passages.
- The findings describe the 38 models tested and do not establish how any untested model behaves.
- The left-progressive and centered patterns are the authors' reported measurements, not independent confirmations.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.