WHBench: expert-built women's health test finds no LLM above 75 percent
Researchers introduce a 47-scenario benchmark for women's health, scored with expert validation. Across 22 models, the strongest reached 72.1 percent, with low fully correct rates and varying harm rates.
In plain words
A research team built a test set of 47 clinical scenarios covering 10 women's health topics, to see how well large language models answer medical questions. Twenty-two models were graded against a 23-point rubric; none averaged above 75 percent, and the best scored 72.1 percent. The authors say this shows expert oversight is still needed before using such models for clinical guidance.
What was announced
The Women's Health Benchmark (WHBench) consists of expert-crafted scenarios designed to surface clinically meaningful failure modes, including outdated guidelines, unsafe omissions, dosing errors, and equity-related blind spots. The authors evaluated 22 models using a 23-criterion rubric covering clinical accuracy, completeness, safety, communication quality, instruction following, equity, uncertainty handling, and guideline adherence, with safety-weighted penalties and server-side score recalculation.
Across 3,102 attempted responses, of which 3,100 were scored, no model's mean performance exceeded 75 percent; the best model reached 72.1 percent. Even top models showed low rates of fully correct answers and substantial variation in harm rates. The authors report that agreement between raters was moderate at the level of individual response labels but high for model rankings, which they say supports the benchmark's use for comparing systems. They position WHBench as a public, failure-mode-aware benchmark for tracking safer and more equitable progress in women's health AI, while stressing that expert oversight remains necessary for clinical deployment.
What it does not show
- The evidence is the paper's abstract only; full methodology, scenario details, and per-model results are not available in the supplied passage.
- The abstract does not name the models tested or identify which model achieved 72.1 percent.
- Inter-rater reliability was only moderate at the response label level, so individual answer labels carry uncertainty even though model rankings were stable.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.