← News

Agents & MCPSources: arXiv · 16 Nov 2026

Case study probes whether a shipped finance agent's benchmark score can be trusted

Researchers audited the reference agent from the Vals Finance Agent Benchmark v2, finding unstable scores, diagnosable failures and one change that cut costs.

In plain words

A finance agent is software that answers research questions using a language model. The authors ran the same agent many times and found its score and cost varied even with identical inputs. They also showed that failures could be reliably diagnosed and that one targeted change cut costs without hurting accuracy.

What was announced

The study, published as an arXiv paper, examined the reference agent shipped with the Vals Finance Agent Benchmark v2 in its untuned state. Seven identical runs on 18 questions passed between 105 and 123 of 149 rubric criteria, with billing varying by up to 17%. To assign responsibility for failures, the authors classified 20 failed trajectories from three runs of nine held-out questions using a taxonomy fixed in advance: 13 failures traced to the agent's instructions, 3 to its tools, 4 to the model and none to its data.

Diagnosis held up under independent checks: four blind readers from three model families, seeing only the traces, named the same fault as the authors' LLM analyzer in 94% of cases, matching the readers' 93% agreement rate with each other. An optimization loop that made one targeted change per round, keeping it only if it cleared a pre-set threshold, produced a final configuration costing 0.34x the baseline on held-out questions (95% CI [0.22, 0.58]) at accuracy within noise. Under an accuracy objective it gained 14 points on development questions but only 1.5 on held-out ones (CI [-5.3, +7.2]). Judged against measured run-to-run noise under a rule written before testing, the authors call the cost saving real and the accuracy gain not.

What it does not show

  • The authors state this is a case study of one agent and nine test questions, not a general result.
  • The accuracy gain of 1.5 points on held-out questions was not distinguishable from run-to-run noise by the authors' pre-registered rule.

For security and IT teams

Scope
Findings cover a single reference agent and nine held-out test questions from the Vals Finance Agent Benchmark v2, not a class of agents.

Sources

Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 17 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.