Study finds AI reviewers rarely diagnose figure-text conflicts in papers
A new arXiv benchmark study tests whether multimodal models can autonomously find figure-text inconsistencies in full scientific manuscripts, and reports that most attempts fall short.
In plain words
When a scientific paper's figure contradicts its text, an AI reviewer should spot it. Researchers tested six multimodal models on full papers containing known inconsistencies. The models often flagged the flawed figure but almost never explained the contradiction correctly, and pointing them to the right figure helped far more than general instructions.
What was announced
The authors, writing on arXiv, built an autonomous evaluation on 31 full manuscripts, each containing a reviewer-documented figure-text inconsistency drawn from PRISMM-Bench and hidden from the model. Six multimodal models reviewed each PDF under a neutral prompt, with additional inputs added one at a time. Across 1,445 open-ended reviews, 60.6% mentioned the flawed figure, but a grading rubric accepted only 3.3% as diagnosing its contradiction. A blind audit found the rubric lenient, so the authors treat these as rubric-accepted rates and compare conditions rather than absolute values.
In a separate controlled run, naming the figure, without saying it was flawed, raised rubric-accepted diagnosis from 4.3% to 20.4%, while a consistency instruction and coarse visual cues did not help. When the contradiction was visible only in the rendered figure, text-only review produced no accepted diagnosis, while reviewing rendered pages with the figure named reached 15.8%. A baseline letting the model choose which figures to check, ranked by centrality, stayed at 4.8%. The authors conclude that deciding what to verify is a large part of the difficulty, and that the tested selection rule did not resolve it.
What it does not show
- The source is an abstract-level excerpt; full methodology and results are not available for verification.
- The rubric was audited as lenient, so reported rates are for condition comparison, not absolute performance.
- Only six multimodal models and one selection rule were tested; broader model coverage is not established.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.