IBM researchers let agents distill their own memory, with gains that vary by model
A new pipeline mines lessons from an agent's past runs, merges them into guidelines, and shows that how much memory helps depends on the model itself.
In plain words
AI agents that work on long tasks often repeat the same mistakes. The researchers built a system where the agent reviews its own past attempts, extracts short lessons, and merges them into a compact rulebook it reads before new tasks. The useful amount of this stored memory turned out to differ per model.
What was announced
The team, writing on arXiv, describes a two-stage process: an agent's completed runs are first mined for one to three short lessons each, and those lessons are then consolidated into a smaller set of guidelines that record how many lessons they absorbed. At run time the agent gets every guideline or a chosen subset. The code ships in the open-source altk-evolve library, and the approach is deployed in IBM's CUGA agent and the watsonx Orchestrate platform.
Testing on the AppWorld benchmark across models with baseline completion rates of 40 to 92 percent, the authors' central claim is that the right amount of memory is a property of the model, not the memory. DeepSeek-V3.2 kept improving with more guidelines, adding 9.5 points of task completion and 16.1 points of scenario completion with its full 191-guideline set. Weaker models did better with less: gpt-oss-120b, starting at 39.9 percent, hit 51.2 percent with all 199 guidelines but 55.4 percent, averaged over two runs, with a subset of roughly 29. The authors also report failures along the way, including a popular guideline that hurt performance, provenance lost during consolidation, and a swallowed API error that failed every task while logs looked plausible.
What it does not show
- Evidence comes from the paper's abstract only; the full article was not reviewed, so methods and caveats could not be checked.
- Results are self-reported by the authors on the AppWorld benchmark and have not been independently confirmed.
- The abstract does not state which tasks beyond AppWorld, if any, were tested, so generalization to other settings is unknown.
For security and IT teams
- Scope
- The pipeline is available in the open-source altk-evolve library and is deployed in IBM's CUGA agent and watsonx Orchestrate, per the authors.
Sources
Source-support review completed 21 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 21 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.