Budget-paired counterfactual audits proposed for evaluating visual document agents
Researchers introduce an evaluation protocol that separates how visual agents plan their document reading from how they answer, by holding evidence fixed and varying only the observation budget.
In plain words
Visual agents that read dense documents must decide which parts to look at under a limited budget of observations. The proposed audit changes only how many observations remain, then measures exactly how much accuracy is lost at each step. This splits errors into planning mistakes and final-answer mistakes.
What was announced
In an arXiv abstract, the authors argue that standard answer accuracy conflates three distinct factors: where an agent chooses to look, how it adapts to the reads that remain, and how it answers once observation stops. Their protocol, called budget-paired counterfactual audits, holds the public evidence and region bindings fixed while perturbing only the remaining budget or declared task parameters. Every acquired history is credited with exact continuation values, so lost expected accuracy can be decomposed into stepwise acquisition loss and terminal decision loss, separating planning from fixed-role heuristics.
The abstract also references results across forty paired routing anchors and five open-weight models, though the excerpt cuts off mid-sentence, so the full findings and any performance conclusions are not available from this source.
What it does not show
- The source is an abstract only; the excerpt ends mid-sentence, so the reported results across forty paired routing anchors and five open-weight models cannot be fully verified, and no independent confirmation is available.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.