← News

ResearchSources: arXiv · 16 Nov 2026

Study finds LLMs build their own task harnesses, but self-revision gains stay small

A controlled arXiv study had frontier models construct and twice revise harnesses for ARC-AGI-3. Autonomous revisions helped little on held-out games, though selecting among snapshots by development score did better.

In plain words

A harness is a program that manages a language model's calls, handling perception, memory and tools. Researchers let models build and improve such harnesses themselves for an interactive benchmark. The models' own revisions barely improved results on unseen games, but picking the best existing version worked better.

What was announced

In the study, each model used a fixed coding bootstrap to construct a complete task harness, act through it on five development games, and revise the entire codebase twice. Every version was then evaluated on twenty games that never influenced adaptation, with performance measured by Relative Human Action Efficiency and inference cost recorded in dollars. Across 27 lineages, harness performance ranged from zero to near-ceiling scores.

The authors report that two revisions raised mean held-out RHAE by only 2.80 points, with a 95% interval spanning -1.44 to 7.35, and that development gains overstated held-out improvement. Cost reductions could come at the expense of task performance, and no revision retained its parent. A post hoc rule selecting among existing snapshots by development score raised the held-out gain to 4.53 points, which the authors say separates generating revisions from deciding to deploy them. They conclude that even strong task performance does not ensure revisions generalize or preserve existing capabilities.

What it does not show

  • The evidence is the paper's abstract only; the full methodology and results were not available for verification.
  • Findings apply to a two-revision protocol on ARC-AGI-3 and may not transfer to other benchmarks or settings.
  • The reported intervals include the possibility of no held-out gain, so the size of the improvement is uncertain.

For security and IT teams

Scope
The source is the paper's abstract, so conclusions rest on the authors' own summary rather than the full article.

Sources

Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 17 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.