← News

ResearchSources: arXiv · 20 Nov 2026

Requiring executable explanations cuts ARC-AGI-2 scores, study finds

An arXiv study measures the cost of demanding verifiable programs from language models, and reports a search method that recovers part of the loss while saving compute.

In plain words

Researchers tested what happens when a language model must explain its answer by writing a program that can be run and checked, rather than just giving an answer. The program requirement lowered scores on a benchmark, but reusing already-generated programs in a small search recovered much of the gap and allowed earlier stopping.

What was announced

The authors measured both costs on the 120-task ARC-AGI-2 evaluation set. Requiring a verifiable artifact from a large language model dropped the score from 93 to 68 of 120 tasks, or 77% to 57%. Of the 25 tasks lost, 21 were not solved by any program-producing configuration they ran, at any budget, by sampling or search. Yet for 27 of the 28 tasks only the answer-only arm solved, some program arm held a program reproducing every demonstration while still failing the test. The authors conclude the bottleneck is obtaining an explanation that generalizes, not finding one.

The study also reports a partial remedy. A short evolutionary search over two already-sampled programs, at no additional model cost, beat best-of-2 sampled code by 9.2 plus or minus 1.4 tasks, and beat 1-shot sampling by 9.9 plus or minus 1.3. Because programs can be checked against training examples, any program-producing protocol can stop early once a task is explained: under the same stopping rule, best-of-n sampling declined up to 38% of its budget and the evolved arm up to 49%, while an answer-only baseline cannot terminate early. The same methodology applied to decoder synthesis for quantum error correction produced interpretable decoders evaluated against a certified panel of published decoders on held-out exam shots.

What it does not show

  • The evidence is the paper's abstract only; full methods, model identities and peer-review status are not available in the supplied passages.
  • Results are reported for the ARC-AGI-2 benchmark and one quantum error correction application; performance on other tasks is not established.
  • The paper does not name which language models were tested or their exact configurations in the supplied text.

For security and IT teams

Scope
Findings cover the 120-task ARC-AGI-2 evaluation set and a quantum error correction decoder synthesis experiment, as described in the paper's abstract.

Sources

Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 20 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.