ABACUS pairs frozen judgment models with reusable programs for multistep reasoning
The arXiv paper reports that saved composition programs let a judgment model answer multi-step questions without changing its parameters, with accuracy close to a larger model's free-form responses.
In plain words
A judgment model answers questions by scoring candidate outcomes rather than writing free-form text. The ABACUS framework has a generative teacher build a reusable program in advance; at run time the judgment model handles small semantic checks and the program chains those decisions into a full answer. The authors report this raised accuracy on selected tasks while keeping the model's parameters fixed.
What was announced
According to the paper's abstract, a generative teacher designs the procedure offline while the judgment model resolves local semantic questions online, and the saved program composes those decisions into an answer. The authors describe three allocations of design effort between humans and the teacher, centred on construction through a shared interface, and instantiate the framework with a model called Kev.
The reported results come from two studies. On three selected BIG-Bench Hard tasks, nine independently constructed programs solved all 450 executions of 150 development inputs. In a separate deployment study, three programs frozen after development-set refinement reached 99.67% macro accuracy on 600 held-out questions, compared with 64.00% for direct Kev and 68.80% for final-answer-only Qwen3.5-4B, and matched Qwen's unrestricted response policy (99.07%) with a reported 8.43-fold online speedup of 0.255 versus 2.150 seconds per question.
What it does not show
- The evidence is the paper's abstract only; full methods, task selection details and any external replication are not covered, and the benchmark tasks were selected by the authors.
- The accuracy and speed figures are the authors' own measurements on their chosen tasks and have not been independently confirmed.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.