Study parses 143,209 coding-agent runs to reveal behavior behind benchmark scores
Researchers mapped raw agent execution traces into comparable actions, finding that agents with similar solve rates can behave very differently when exploring, editing and verifying code.
In plain words
Coding agents are usually judged only on whether they complete a task, which hides how they actually work. The researchers built a parser that turns raw agent logs into readable actions like searching, editing and running tests. This lets them compare how different agents behave, not just whether they succeed.
What was announced
The study, published as an arXiv preprint, analyzed 143,209 execution trajectories from 125 agent configurations spanning 117 unique models. Standard benchmarks mostly measure binary task success, treating execution as a black box. To compare across different agent interfaces, the authors built a rule-based parser that maps raw traces to high-level actions such as repository search, file editing and test execution, and audited its predictions against a six-model consensus reference on 500 events.
The authors report that agents with similar solve rates can differ substantially in how they explore, modify and verify code. Using aggregate behavioral profiles, without task outcomes, token counts or tool names as features, they retrieved the corresponding agent stack across disjoint repository sets and programming languages, though they note task composition was not fully controlled in the cross-repository setting. Analyses of frontier models and agent scaffolds showed that scaffold choice and reasoning effort systematically shift search, editing and testing behavior. The parser, annotation and analysis pipeline, and an annotated trajectory subset are being released.
What it does not show
- The source is an abstract only; full methods, results and caveats are not available for verification.
- The authors state task composition was not fully controlled in the cross-repository setting, so behavioral-profile retrieval may partly reflect task differences.
Sources
Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 17 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.