Study finds GPT and Claude review weaknesses cluster in paper appendices
Researchers analyzing 2,000 papers report that leading LLMs draw roughly half or more of their review weaknesses from appendices, while human reviewers draw under a tenth from there.
In plain words
Some researchers now use large language models (LLMs), the AI systems behind chatbots, to help critique academic papers. A new study says these models focus their criticism on the appendix, the supplementary section at the end of a paper, far more than human reviewers do.
What was announced
The authors examined 2,000 papers from ACL, EMNLP, COLM, and ICLR and found that recent GPT and Claude models attribute about 50% or more of their review weaknesses to the appendix. The pattern held whether the models were used through an API, a coding agent, or a web interface. Human reviewers, by contrast, drew only 8.4% of their weaknesses from the appendix, concentrating instead on presentation and content tied to a paper's core idea.
According to the study, the models' appendix-based criticisms target fine-grained technical details deep in the supplementary material that are not central to a paper's contributions. The authors suggest this matters because LLM outputs often shape the reviews humans submit, and they have released their analysis code publicly.
What it does not show
- The evidence is the paper's abstract only; full methodology, model versions, and statistical details are not available in the supplied passages.
- The abstract's human comparison sentence is cut off mid-statement, so the complete human-side findings cannot be verified from the excerpt.
- The study covers four NLP and ML venues and does not establish how these findings generalize to other fields or review settings.
Sources
Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 17 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.