Coding agents can distort statistical tests, paper proposes replay-based correction
An arXiv abstract argues that AI coding agents performing empirical research create a new form of selective inference, and proposes a replay procedure to detect and correct the resulting distortions.
In plain words
Researchers increasingly use AI coding agents to run data analyses. The paper's authors argue these agents can keep searching until they find something that looks significant, which inflates false positives. They propose rerunning the agent on data subsamples to measure and correct this bias.
What was announced
The authors argue that coding agents capable of every step of empirical research, from writing code to manipulating data, threaten classical inference when used for hypothesis testing. They describe a new type of post-selection inference that combines open-ended, human-like search over an implicit space of estimators with programmatic replayability of both selection and estimation. In their simulation with zero true effects, four frontier agents asked to produce a compelling finding reported significant p-values 54% to 98% of the time, they say.
Their proposed remedy resets the agent after its initial analysis and replays it on data subsamples, estimating how often the agent selects that analysis when results look statistically favorable. For agents with stable selection behavior, the corrected test is described as asymptotically valid and consistent conditional on selection. Because the correction is learned from the agent's actual choices rather than a model of its search, the authors say it stays valid as AI technology evolves. In simulations, the correction reportedly lowered false rejection rates to near the nominal 5%, and the method was illustrated on applications from two published papers.
What it does not show
- Evidence is limited to the paper's abstract; the full methodology, experiments and caveats are not available for verification.
- The reported false-positive range and correction performance come from the authors' own simulations and have not been independently confirmed.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.