SkillBuffBench tests agent Skills under realistic access and memory conditions
A new arXiv paper introduces a benchmark that evaluates file-based agent Skills when agents must find them in an installed library and carry memory from earlier work.
In plain words
Coding agents can load 'Skills', which are files of step-by-step instructions, to help with tasks. The authors of SkillBuffBench argue that past evidence tested cases where the right Skill was chosen by hand and the agent started fresh. Their benchmark instead makes the agent find Skills on its own, with and without memory, across several harnesses and models.
What was announced
The paper's authors state that Agent Skills, file-based packages of procedural knowledge that agents discover and load on demand, are now shipped by every major agent harness, but that supporting evidence comes almost entirely from settings where the correct Skill is hand-picked and the agent begins with no history. Deployment differs: agents pick from whatever library is installed and often carry memory from earlier work. SkillBuffBench therefore treats Skill access as a three-level factor: no Skill, a curated Skill, or automatic invocation from an installed library.
The design crosses Skill access with memory (off and on), four agent harnesses, and multiple models, including proprietary models from Anthropic, Google and OpenAI and open-weight models from Chinese providers, with every model run under every harness. Tasks come from domain experts in materials science and biology, and the suite is extended on a rolling basis. Each run is checked by a deterministic verifier, uncertainty is reported with a paired task-level bootstrap, and cost, agent rounds and wall-clock time are reported alongside scores. The authors note this version describes the design and current tasks only: evaluation is in progress, and results will be added once runs complete.
What it does not show
- The paper version described covers only the benchmark design, protocol and current tasks; no evaluation results are included yet, so no performance conclusions can be drawn from this source.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.