RL study finds LLM-judge rewards pass category behaviour, not answer knowledge
Across 814 reinforcement learning runs, the authors report that a judge's verdicts on individual answers stay tied to those questions, while its stance on question types transfers to new ones.
In plain words
Some AI models are trained with rewards from another model, called a judge, when no answer key exists. The paper reports that the judge's knowledge of specific answers does not transfer to the trained model, but its attitude toward whole categories of questions does. A judge that rejects certain correct answers taught the model to refuse similar new questions, even though it still knew the answers.
What was announced
The authors trained 7-8B models with reinforcement learning to answer closed-book questions or abstain, using LLM judges as rewards where no answer key exists. They report that a wrong answer a judge rewards is memorised only for that question: a model trained with the answer key, or with a far stronger judge, chose which new questions to answer no better than an untrained model sampling 32 answers and abstaining where they disagree. They estimate the grader's knowledge adds about one corrected answer per hundred questions.
Category-level behaviour did travel. A judge that rejected eleven of twelve correct training answers about capitals of former states taught the model to refuse 98% of such capitals on new questions, a loss the authors say accuracy and abstention rates do not reveal. Hand-changing only those twelve verdicts moved refusal from 5% to 90%, even with a grader as lenient as the answer key; at that leniency, two of four judges still cost 38-42% of these capitals. The knowledge was not forgotten: with "I don't know" blocked, the model answered 94% correctly. Rewarding agreement among the model's own untrained samples, with no labels and no judge, earned the answer key's reward and kept the category.
What it does not show
- The evidence is the paper's abstract only; full methods, datasets and peer review are not covered by the supplied passage.
- The findings concern 7-8B models on closed-book question answering with abstention and do not establish results for other model sizes or tasks.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.