Frozen language models paired with small trained heads answer medical exam questions
Researchers rebuilt a single-pass question-answering recipe for MedMCQA, training only two projection heads and a temperature on top of frozen language models. Accuracy stayed well below leading medical systems.
In plain words
A team tested whether a language model that is not retrained can answer medical multiple-choice questions in one step. They froze models like Qwen3-8B and trained only two small matching layers on top. The best setup reached 46.0% accuracy, above the 25% chance level but below Med-PaLM 2's 72.3%.
What was announced
The method, described as a System 1 approach, answers a question in a single pass: the frozen model turns the question and each option into vectors, two small trained heads place them in a shared space, and the closest option wins. The authors say the design had been shown on agent decisions, but they are not aware of a prior study on exam questions requiring medical knowledge, where the correct option is often no more similar to the question than the wrong ones. They rebuilt the recipe from scratch for MedMCQA, training only two projection heads and one temperature on frozen Qwen3.5-0.8B, Granite-4.2-3B and Qwen3-8B.
With plain last-token vectors, the two small encoders answered about one question in three correctly, 33.1% and 33.0% on 3,842 held-out questions where chance is 25.0%. A one-word-summary prompt and a lower learning rate for the heads raised this to 41.7% and 42.5%, and to 46.0% plus or minus 0.2 across three seeds for Qwen3-8B. Shuffling questions against options returned accuracy to chance, which the authors take as evidence the heads use the question. They stress the system is a cheap, fast baseline, not a competitive one: Flan-PaLM and Med-PaLM 2 report 57.6% and 72.3% on MedMCQA under a different protocol.
What it does not show
- The evidence comes from the paper's abstract, so full experimental details are not available for verification.
- Validation gains were about 1.7 to 2 times the test gains, and the authors could not separate the effect of reading several encoder layers from the other changes.
- Flan-PaLM and Med-PaLM 2 results use a different protocol, so the comparison is not like-for-like.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.