PASS reshapes process scores into policy advantages for GRPO reasoning
A new arXiv paper proposes PASS, a method for turning dense process supervision scores into advantages in group-relative policy optimization, reporting gains on math and multi-hop QA benchmarks.
In plain words
Reinforcement learning for language models often uses dense process supervision, meaning the model gets scores for intermediate reasoning steps rather than only a final answer. The PASS method defines how those step scores are converted into the advantage signals that guide policy updates. The authors report accuracy gains on math and question-answering benchmarks compared with outcome-only training.
What was announced
The paper, an arXiv abstract by the PASS authors, identifies a gap in process-supervised reinforcement learning: dense process supervision provides local evaluations, but how those scores become policy advantages is unresolved. PASS normalizes and fuses reward channels separately, builds batch-shared credit units from variation in the fused score trace, and normalizes return-to-go by the remaining unit count, linking feedback scale, credit resolution, and accumulation horizon in one computation.
For learned process reward model feedback in mathematical reasoning, the authors report a 4.62 percentage point average pass@1 improvement over outcome-only GRPO across seven mathematics benchmarks, with responses about 2% shorter. Matched comparisons with Step-GRPO and component ablations support the shared credit units and residual normalization choices. On multi-hop QA with teacher-derived token feedback, PASS improves over process-signal controls across two normalizers. Token-level ablations show an accuracy-length trade-off: token-level accumulation slightly favors pass@1, while PASS reaches higher pass@8/16 with shorter responses. The authors conclude that converting local process scores into advantages is a consequential design choice across supervision sources.
What it does not show
- The evidence is the paper's abstract only; full methods, experimental setup, and peer review are not available in the supplied passages.
- All reported gains and comparisons are the authors' own claims and have not been independently confirmed.
- The abstract does not specify model sizes, training costs, or which models were used, so generalizability cannot be assessed from the supplied text.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.