Measurement choices, not just the code, decide how long AI agent code survives
A re-analysis of 88 survival analyses finds that published verdicts on agent-written code can flip with a single methodological decision, especially for autonomous agents.
In plain words
Researchers asked whether code written by AI coding agents lasts longer in software projects than code written by people. A new paper shows the answer depends heavily on how you measure it: small statistical decisions can change the conclusion, particularly for fully autonomous agents like Devin.
What was announced
The authors revisited their earlier study of 854,139 lines added by 2,802 merged pull requests across 215 AIDev repositories. By varying nine measurement choices, they built a specification curve of 88 survival analyses under different variance estimators, observation windows, and permuted labels. For code from human-supervised agents, mostly GitHub Copilot, methodological choices altered the effect's size but not its direction. For Devin, a fully autonomous agent, the choices decided the verdict outright: across repositories, 15 of 22 specifications showed its lines modified sooner when treated as independent, but only 3 when standard errors were clustered by repository, a level no better than permuted labels. Within repositories, no specification found any difference.
The paper also reports that LLM-based intent classifiers matched human labels about as well as two human raters match each other, while keyword rules performed worse, with kappa of 0.69 to 0.72 versus 0.38. Humans made most modifications to agent code under every unit and window tested. When the authors re-ran five published study designs on their data, each changed its result after a single choice was altered. The team concludes that whether agent code outlasts human code is decided as much by measurement as by the code itself, and they offer a checklist for future post-merge studies.
What it does not show
- The source is an abstract, so full methodological detail and complete results are not available for verification.
- Findings come from one dataset of 215 repositories and one research group's re-analysis; no independent confirmation is reported.
- The abstract does not state which measurement choices are defensible or which verdict is correct, only that choices change outcomes.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.