Cross-vendor model auditing shows low recall on general code, stronger results on scientific tasks
A study of one deployed auditing setup finds model reviewers catch only a fraction of code errors, while deterministic checks excel on scientific reports.
In plain words
Researchers measured how well an AI auditor from one vendor checks code written by other models, alongside fixed rule-based checks. On general code the auditor found about 30% of real errors, while on scientific reports and data the combined checks performed far better.
What was announced
The paper examines a shipped arrangement in which an auditor from a different vendor reads model-written work while deterministic checks verify whatever code can verify. On general-purpose code with hidden tests, eight readings of the auditor reached 30.0% recall, with a 16.0% flag rate on correct code, showing diminishing returns and no plateau. The authors report that the reading model, one rulebook sentence and the decision rule each shifted that figure, and post hoc rating suggested much of what was missed was behaviour the specification never determined.
In three preregistered science studies, source-tracing checks flagged all 49 inconsistent reports and none of 25 consistent fabrications, of which the model auditor flagged 23. On data, a documentation-derived validator and the auditor flagged overlapping but different faulty items, together 112 of 140, with a 2.9% flag rate on clean data. On scientific code, reframing the task around one step cut flags on correct code from 68.7% to 36.0%. Where both tiers ran, the deterministic tier flagged no clean item but could not see consistent error, while the model saw some and made its own errors.
What it does not show
- The abstract reports one shipped arrangement and preregistered science studies; the general-code recall figure carries wide intervals and the authors note the reading model is confounded with its sampling, so results may not generalise to other setups.
For security and IT teams
- Scope
- Findings cover one specific cross-vendor auditing arrangement with deterministic checks, not model auditing in general.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.