Continual-ARC benchmark tests whether agent harnesses learn over a working lifetime
A new arXiv paper proposes a lifelong task stream built from ARC-AGI editions, scoring memory-based and weight-based learners on one scale. Early results favor weight-based learning.
In plain words
Researchers introduced Continual-ARC, a benchmark that feeds AI systems a long stream of puzzle tasks, with old tasks returning later in new versions. It is designed to measure whether a system remembers what it saw and whether it can handle tasks it has never met. In the paper's own tests, systems that update their weights learned more on new tasks than systems that relied on an external memory or harness.
What was announced
The authors, writing on arXiv, argue that existing benchmarks for continual learning usually cover a single applied domain, show each task instance only once, and assume one type of learner, so memory-based and weight-based systems are seldom compared on equal terms. Continual-ARC addresses this by building a lifelong stream from the three ARC-AGI editions, whose tasks require no domain knowledge. Tasks recur after dormant gaps with fresh instances while new tasks keep arriving, and a learner may keep anything between instances, putting both system types on the same scoring scale.
In the stream, a learner can pay for demonstrations, submit answers, or act inside the harness, with retries on incorrect submissions also paid for; each instance is scored by the share of this feedback budget consumed. After the stream, the frozen learner is tested on fresh instances of known tasks to measure memorization and on unseen tasks to measure generalization. Testing an agent harness with frozen weights on Qwen3.5-9B and GLM-5.3 Flash, against prompting floors, a program-induction method and weight training, the authors report that persistence rarely improved memorization without expert guidance, most harness-based methods learned less than a floor that simply caches demonstrations, and harness-based methods gained little on unseen tasks while weight-based learning showed substantial gains.
What it does not show
- The evidence is the paper's abstract only, so experimental details and full results cannot be verified from the excerpt.
- The reported findings come from the paper's own runs on two named models and have no independent confirmation.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.