← News

ResearchSources: arXiv · 18 Nov 2026

DiagTrace benchmark tests how LLMs seek diagnostic information step by step

A new arXiv paper introduces DiagTrace, a benchmark built from case reports that evaluates whether language models can gather patient evidence incrementally rather than answer with everything given upfront.

In plain words

Most tests of AI in medicine hand a model all the patient data at once and grade only the final answer. DiagTrace instead checks whether models can work like a clinician, requesting information piece by piece from incomplete records. The benchmark's authors built it from published case reports and used it to evaluate four commercial or research language models.

What was announced

The paper's authors argue that existing medical evaluations of large language models share a weakness: they supply all patient information upfront and judge only the final diagnosis or stated rationale. Real diagnosis, they note, requires acquiring evidence incrementally from incomplete information. DiagTrace addresses this by asking whether a model can reconstruct a documented diagnostic trajectory through interactive information seeking. Each case is grounded in a PubMed Central case report and includes an initial presentation, an event bank, a clinical course, and diagnostic references.

The benchmark contains 510 cases spanning eight specialties. Of these, 225 cases were used to evaluate four LLMs on three measures: alignment with the clinical course, quality of the interaction, and efficiency. The authors report that results reveal trade-offs among coverage, source-consistent ordering, and turn use. The paper does not name the models tested or state which performed best, and the abstract does not claim that any model matched clinician performance.

What it does not show

  • The source is only the paper's abstract, so evaluation methods, model identities, and detailed results cannot be verified from the supplied text.
  • The abstract reports trade-offs without quantifying them, and no independent replication of the findings is available.

Sources

Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 19 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.