Researchers propose reference-based rewards as a cheaper signal for LLM reasoning and alignment
An arXiv paper introduces RL2R, a reward design that scores how semantically close a model's output is to a reference answer, replacing costlier verification and reward-model pipelines.
In plain words
Training language models with reinforcement learning needs a way to score outputs, and the authors say existing scoring methods are expensive or too coarse. Their method, RL2R, instead compares a generated answer with a reference using lightweight similarity metrics like BERTScore, giving partial credit to responses that are close to the reference. They report this works for both math reasoning and alignment tasks at lower cost.
What was announced
The paper, listed on arXiv under the title "Reinforcement Learning with Reference-Based Rewards for Language Model Reasoning and Alignment," frames reward design as the shared bottleneck of two post-training lines: reasoning, where rule-based checks of final answers give binary feedback blind to intermediate derivations, and alignment, where reward models trained on preference pairs make data collection and training resource-intensive. Process reward models, which could grade intermediate steps, need step-level annotations the authors call too costly to scale.
RL2R measures semantic similarity, using metrics such as BERTScore, between a sampled generation and a pre-existing or distilled reference answer or solution. Because similarity is graded rather than binary, responses nearer the reference earn higher rewards, and the authors say sound derivations get credit even when the final conclusion is wrong. For math reasoning, adding answer- and step-level similarity to rule-based verification reportedly outperforms answer-only verification. For alignment, a critic-free REINFORCE-style variant covered general preference, safety, and confidence tasks on par with reward-model-based methods, according to the abstract.
What it does not show
- The evidence is the paper's abstract only; no full results, benchmarks, baselines, or peer review are available to verify the reported performance claims.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.