← News

Agents & MCPSources: arXiv · 18 Nov 2026

Researchers release Gandalf, an open-source verifier for rubric-graded AI agent tasks

A new arXiv paper introduces BankerVerifierBench, a human-judged test set for agent-output verifiers, and Gandalf, an open-source verifier that reportedly matches or beats existing baselines at lower cost.

In plain words

As AI agents complete open-ended tasks, checking their work is hard, so many systems use a verifier that grades the agent's output against written criteria. The authors built a benchmark of human-judged criteria and an open-source verifier called Gandalf. In their tests, Gandalf matched or outperformed three existing verifiers while costing less to run.

What was announced

The paper, whose abstract is the available source, reports two main contributions. First, the authors created BankerVerifierBench (BVB), a meta-evaluation dataset of 2,464 human-judged rubric criteria spanning 24 investment-banking tasks, and derived from rubric data a nine-capability taxonomy for verifiers, distilled into three design principles: reactive verification, environment alignment, and domain guidance. Second, they implemented these principles in Gandalf, an open-source verifier, and evaluated it against three existing verifiers on BVB.

According to the reported results, Gandalf matched or exceeded every baseline's F1 score at lower cost: using Gemini 3 Flash it reached 0.864 versus 0.852 for the strongest baseline at one-fifth of the cost, and with GPT-5.4 models it improved F1 by more than 0.12 at lower cost. The authors also report that Gandalf generalized to OpenClaw, a structurally different personal-productivity environment, scoring 0.955 F1 against 0.934 for the best baseline. They state they are releasing BVB, Gandalf, and the code and data needed to reproduce the experiments.

What it does not show

  • Only the paper's abstract is available, so the methodology, task details, and evaluation setup cannot be independently checked, and the reported results are the authors' own claims without independent confirmation.

For security and IT teams

Scope
The released artifacts cover the BVB benchmark, the Gandalf verifier, and the code and data to reproduce the paper's experiments; the abstract does not describe production deployment guidance.

Sources

Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 19 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.