Audit finds benchmark case text, not simulator leakage, inflates LLM diagnostic scores
The HindAudit study separates three hindsight channels in clinical agent benchmarks and reports that retrospective case wording, not simulated lab results, drives the largest measured effect.
In plain words
Benchmarks that test AI diagnostic agents are built from hospital records written after doctors already knew the outcome, which can quietly favor the AI. The HindAudit audit checked three places where this hindsight could enter. Simulated lab results were clean, but the way cases were rewritten for benchmarks raised model accuracy substantially.
What was announced
The audit, described in an arXiv abstract, analyzes benchmarks for LLM diagnostic agents, which are built from retrospective records: agents read cases rewritten from discharge summaries, and a simulator generates unrecorded test results, often with access to the final diagnosis. HindAudit separates three hindsight channels — the diagnosis given to the simulator, the retrospective context it conditions on, and the agent-facing case text — and measures each using judges trained only on real data.
On 27,489 emergency admissions from MIMIC-IV, five registered tests found no leakage into simulated laboratory results; giving the diagnosis to five simulators had a negligible effect. The largest effect lay in the case text: against a presentation-time reconstruction from triage records, benchmark-style summaries raised zero-shot accuracy of four LLMs from 25.4% to 39.7%, and discharge histories to 45.3%. Six interactive agents also scored higher, though their ranking barely changed. The authors note MIMIC lacks a contemporaneous bedside history, so the measured gap is an upper bound on hindsight, and they report all registered outcomes.
What it does not show
- The evidence is the paper's abstract only; full methods, thresholds and caveats are not available in the supplied passages.
- The case-text gap is an upper bound on hindsight because MIMIC-IV lacks a contemporaneous bedside history, per the authors.
- Exploratory findings, such as gatekeepers raising agent scores, are described as exploratory and were reproduced only in a registered replication on held-out patients.
For security and IT teams
- Scope
- Findings apply to LLM diagnostic agent benchmarks built on MIMIC-IV emergency admissions, not to live clinical deployment.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.