Requiring executable explanations cuts ARC-AGI-2 scores, study finds
An arXiv study measures the cost of demanding verifiable programs from language models, and reports a search method that recovers part of the loss while saving compute.
In plain words
Researchers tested what happens when a language model must explain its answer by writing a program that can be run and checked, rather than just giving an answer. The program requirement lowered scores on a benchmark, but reusing already-generated programs in a small search recovered much of the gap and allowed earlier stopping.
What was announced
The authors measured both costs on the 120-task ARC-AGI-2 evaluation set. Requiring a verifiable artifact from a large language model dropped the score from 93 to 68 of 120 tasks, or 77% to 57%. Of the 25 tasks lost, 21 were not solved by any program-producing configuration they ran, at any budget, by sampling or search. Yet for 27 of the 28 tasks only the answer-only arm solved, some program arm held a program reproducing every demonstration while still failing the test. The authors conclude the bottleneck is obtaining an explanation that generalizes, not finding one.
The study also reports a partial remedy. A short evolutionary search over two already-sampled programs, at no additional model cost, beat best-of-2 sampled code by 9.2 plus or minus 1.4 tasks, and beat 1-shot sampling by 9.9 plus or minus 1.3. Because programs can be checked against training examples, any program-producing protocol can stop early once a task is explained: under the same stopping rule, best-of-n sampling declined up to 38% of its budget and the evolved arm up to 49%, while an answer-only baseline cannot terminate early. The same methodology applied to decoder synthesis for quantum error correction produced interpretable decoders evaluated against a certified panel of published decoders on held-out exam shots.
What it does not show
- The evidence is the paper's abstract only; full methods, model identities and peer-review status are not available in the supplied passages.
- Results are reported for the ARC-AGI-2 benchmark and one quantum error correction application; performance on other tasks is not established.
- The paper does not name which language models were tested or their exact configurations in the supplied text.
For security and IT teams
- Scope
- Findings cover the 120-task ARC-AGI-2 evaluation set and a quantum error correction decoder synthesis experiment, as described in the paper's abstract.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.