Calibrated judge rewards lift Qwen3-1.7B math accuracy without answer keys
Researchers propose RLCDR, turning an LLM judge into a calibrated decision model whose probability scores guide reinforcement learning, matching answer-key training on MATH in their reported tests.
In plain words
Reinforcement learning needs a reward signal telling a model whether its answers are good. Exact answer checkers require reference answers, so the authors instead convert a language model into a judge that outputs calibrated probabilities for questions like "is this answer correct?". Training with these rewards reportedly improved a small model's math accuracy almost as much as training on the answer key.
What was announced
In an arXiv abstract, the authors introduce RLCDR (Reinforcement Learning from Calibrated Decision Rewards). An LLM is converted into a decision model that answers typed questions about a response, such as whether the final answer is correct or which step contains the first error, outputting a probability distribution in one forward pass. The scores are debiased across option orders and calibrated using roughly a hundred labels. Training Qwen3-1.7B with GRPO on MATH, the reference-free reward reportedly raised MATH500 accuracy from 72.2% to 76.8%, close to the 76.9% achieved with the answer key, while a random reward added only 0.65 points. Gains grew with problem difficulty, and the reward's expected calibration error was reported as half that of a softmax judge on the same 27B backbone, with no observed reward hacking.
The authors also report limits. Without step-level training, their model outperformed a dedicated 7B process reward model on ProcessBench (F1 0.772 vs 0.722). However, when trained only on problems the base model mostly fails, the reward still helped but trailed the answer key because its probabilities became overconfident on that distribution. The team pre-registered six hypotheses, each tested with a falsification control.
What it does not show
- All figures come from the paper's own abstract; no independent replication is reported.
Sources
Source-support review completed 21 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 21 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.