MInfoVisQA benchmark finds language changes destabilize multimodal model accuracy on charts
A new arXiv paper introduces a parallel multilingual benchmark testing whether multimodal large language models understand information-rich visuals consistently across 24 languages.
In plain words
Multimodal large language models answer questions about images such as charts and infographics. The MInfoVisQA authors built the same questions and visuals in 24 languages to test whether answers stay accurate when only the language changes. They found accuracy varies widely by language, even for models with strong average scores.
What was announced
The benchmark, described in a paper by the MInfoVisQA team, contains 8,960 question-answering configurations built from 128 seed instances spanning 24 languages. The authors constructed seed-matched, language-parallel visuals and QA pairs by separating numerical-structural data from text and programmatically synchronizing translated labels, so each language version corresponds to the same underlying content. They evaluated 25 multimodal large language models under two settings: Language-Aligned QA, where visual and question languages match, and Cross-Language QA, where the QA language stays fixed while the visual language varies.
The results suggest that average scores hide meaningful weaknesses. Two models with nearly identical mean Language-Aligned QA accuracy of 80.4 percent showed performance spreads of 7.8 and 20.3 percentage points across languages, according to the paper. In the cross-language setting, all 25 models scored lower when visuals were non-English while questions stayed in English. The authors argue this underscores the need for cross-language sensitivity metrics when reporting model capability.
What it does not show
- The evidence is the paper's abstract only; full methodology, model names, and per-language results are not available in the supplied passages.
- The abstract reports the authors' own evaluation; no independent replication is included in the supplied evidence.
- The abstract is truncated mid-sentence in the supplied passage, so the complete LQA/XQA description and any further findings are not visible.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.