← News

Agents & MCPSources: arXiv · 18 Nov 2026

Retrieved tool output costs GPT agents accuracy on geochemistry tasks, study finds

A study of LLM agents using the PHREEQC geochemistry simulator finds that keeping long reports out of context and retrieving them on demand hurts some models' accuracy through a failure the authors call evidence substitution.

In plain words

AI agents often save long tool outputs to files and search them later instead of holding everything in their context window. Researchers tested this with a geochemistry simulator and found some GPT models answer worse this way: they compute the right value but then answer from the wrong file and misround it. Claude models showed no such loss.

What was announced

The arXiv paper, from authors studying agent harnesses that store long tool outputs in searchable files, asks where this design costs accuracy. Using PHREEQC, a USGS geochemistry simulator whose reports reach 846.8k characters, the researchers built 158 execution-verified multiple-choice items. When agents read the report through a section index rather than seeing it in context, GPT-5.1 and GPT-5.2 lost 11.3 and 7.9 points of accuracy, while Claude Opus 4.6 and Claude Sonnet 4.6 showed no detectable cost.

Re-running all 11,007 logged simulator calls showed the GPT models still computed the keyed value about as often; what changed was the evidence they answered from. Under retrieval, they often skipped the report, answered from their own higher-precision output files, and misrounded to a neighboring option — a failure the authors name evidence substitution. Even with the simulation held fixed, retrieval cost GPT-5.2 9.4 points, and GPT-5.4's more frequent correct simulations masked an evidence-use loss of about ten points. Naming the authoritative output raised accuracy by 6.3-7.9 points for the three GPT models, and inlining outputs that fit worked as well for GPT-5.1 and GPT-5.2, while asking for correct rounding did not. The authors conclude that context-management choices should be judged by where accuracy is lost, not only by final accuracy.

What it does not show

  • The evidence is the paper's abstract only, so experimental details, methods and full results are not available for verification.
  • The findings cover specific GPT and Claude model versions on one simulator and 158 multiple-choice items; no independent confirmation is reported.

Sources

Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 19 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.