ExecJudges: arXiv paper proposes program-generated judges as an alternative to LLM relevance labeling
A new arXiv paper describes ExecJudges, a method where a frontier LLM converts assessment guidelines into executable programs that label document relevance without per-document model calls.
In plain words
Retrieval systems are often evaluated by asking a large language model to judge whether each document is relevant, which is costly and hard to reproduce. ExecJudges instead has the model write a small program per topic that applies the assessment guidelines using lexical matching and standard NLP tools. The programs can be inspected, rerun cheaply, and reused across many documents.
What was announced
The authors, writing in an abstract posted to arXiv, say LLM judges have become common for creating large-scale relevance judgments but each judgment requires a potentially expensive call to a frontier model, and the resulting labels are hard to interpret and may not be reproducible. ExecJudges addresses this by using a frontier LLM to turn assessment guidelines into executable programs that rely only on lexical matching, standard NLP libraries such as parsers, and pool statistics. An ExecJudges-L variant also uses examples of human labels when available.
Evaluated on seven TREC collections against trained NIST assessors, frontier LLM judges, and per-topic classifiers, the method reportedly agrees with NIST assessors at the level of Claude Sonnet 5.5 LLM judges on TREC DL 2019-2021. With a few human labels per topic, ExecJudges-L agrees with NIST more than per-topic classifiers on all seven collections, and either exceeds LLM judges given the same labels or stays within about 0.05 in Krippendorff's alpha. The labels rank official TREC DL runs with Kendall's tau of 0.87 to 0.94, and program writing costs a one-time $0.17 to $0.49 per topic with no grading costs. These are the authors' reported results, not independently confirmed.
What it does not show
- The evidence is the paper's abstract only; full methods, datasets and caveats are not available for verification.
- All agreement, ranking and cost figures are authors' self-reported results and have not been independently confirmed.
- The abstract does not state whether the programs or code are publicly available.
For security and IT teams
- Scope
- The findings apply to automatic relevance labeling for retrieval evaluation on TREC-style collections, as described in the paper's abstract; no production deployment is reported.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.