From Bradley-Terry preference modeling and KL-regularized PPO to closed-form reward reparameterization (DPO), critic-free group advantage estimation (GRPO), and Constitutional AI (RLAIF).
Markdown
1. Pre-Training and Preference Objectives
Causal language model pre-training optimizes next-token maximum likelihood estimation (MLE) over internet-scale corpora:
L_pretrain(θ) = -∑_t log P_θ(x_t | x_{<t}). While MLE builds broad world knowledge and in-context learning capabilities, a base model predicts statistically plausible continuations rather than helpful, truthful, or safe responses—for instance, completing a user question with a list of similar questions rather than answering it [3].
Post-training alignment bridges this gap through two sequential phases:
- Supervised Fine-Tuning (SFT): Adapts the base model on curated human or synthetic instruction-response demonstrations
(x, y) ~ D_SFT, yielding a reference policyπ_SFT(commonly denotedπ_ref) that understands dialogue formatting and task intent [3]. - Preference & Outcome Optimization: Refines
π_refusing pairwise human/AI rankings or verifiable outcome rewards so the policy actively prefers high-quality completions over plausible but flawed ones [2], [3], [6].
r_φ(x, y) is an imperfect proxy for true human preference R_gold(x, y), optimizing aggressively against r_φ causes proxy scores to climb continuously while true ground-truth quality peaks and then degrades sharply. Every modern alignment algorithm therefore enforces a reverse-KL divergence penalty against π_ref to prevent distribution collapse and reward hacking [3], [8].
Markdown
2. Classical 3-Stage RLHF: Bradley-Terry Reward Modeling & PPO
Popularized for language models by Christiano et al. (2017) [2] and scaled in InstructGPT by Ouyang et al. (2022) [3], classical Reinforcement Learning from Human Feedback (RLHF) trains an explicit scalar reward model before running online Proximal Policy Optimization (PPO) [4].
Stage A: Bradley-Terry Pairwise Reward Model
Because human annotators exhibit high variance when assigning absolute scalar scores (e.g., 1–10), datasets collect pairwise comparisons (x, y_w, y_l) where completion y_w is preferred over y_l (y_w ≻ y_l). Under the Bradley-Terry model [1], preference probability is modeled via the logistic sigmoid σ(z) = 1 / (1 + exp(-z)) over latent reward differences:
L_RM(φ) = -E_{(x, y_w, y_l) ~ D} [ log σ( r_φ(x, y_w) - r_φ(x, y_l) ) ]
Stage B: KL-Regularized RL Objective & PPO Surrogate
Starting from π_θ = π_ref, PPO optimizes the policy to maximize reward minus a per-token KL divergence penalty controlled by hyperparameter β > 0 [3], [4]:
PPO Clipped Surrogate Objective:
L_CLIP(θ) = E_t [ min( ρ_t(θ) · Â_t, clip(ρ_t(θ), 1 - ε, 1 + ε) · Â_t ) ]
where ρ_t(θ) = π_θ(a_t | s_t) / π_{θ_old}(a_t | s_t)
Systems Bottleneck: Online PPO-RLHF requires holding four LLM-scale models in accelerator memory simultaneously during training: the trainable Actor (π_θ), frozen Reference (π_ref), frozen Reward Model (r_φ), and a trainable Critic / Value Network (V_ψ) used to compute Generalized Advantage Estimation (GAE) [4], [6]. To reduce human annotation bottlenecks in Stage A, Constitutional AI / RLAIF (Bai et al., 2022) [7] replaces human preference labels with AI-generated critiques and rankings conditioned on an explicit set of normative principles.
Diagram
3. Evolution of Alignment: PPO-RLHF vs. DPO vs. GRPO
Subsequent research eliminated the heaviest bottlenecks of PPO-RLHF through two complementary breakthroughs: Direct Preference Optimization (DPO) [5] for offline preference pairs, and Group Relative Policy Optimization (GRPO) [6] for online reinforcement learning without a value critic.
| Method | Learning signal | Components and limitations |
|---|---|---|
| PPO-based RLHF | Rollouts scored with a reward model | Policy, reference policy and value estimation are typical roles. Actual model sharing, storage and memory depend on the implementation. |
| DPO | Chosen/rejected response pairs | Direct preference loss; no separate learned reward model or critic is required by the original formulation. A reference policy is part of that formulation. |
| GRPO | Rewards for a group of sampled responses | Group-relative advantages replace a learned value critic in the original method. Reward computation may use a verifier or model; memory savings are workload-dependent. |
Mathematical Derivation of DPO (Rafailov et al., 2023)
Rafailov et al. [5] proved that the KL-regularized RL objective admits a closed-form optimal policy π*(y | x) = (1 / Z(x)) · π_ref(y | x) · exp(r(x, y) / β), where Z(x) is the intractable partition function summing over all sequences. Rearranging for r(x, y) yields:
Substituting into Bradley-Terry P(y_w ≻ y_l | x) = σ(r(x, y_w) - r(x, y_l)), the β · log Z(x) terms cancel:
L_DPO(π_θ; π_ref) = -E_{(x, y_w, y_l) ~ D} [ log σ( β · log(π_θ(y_w|x)/π_ref(y_w|x)) - β · log(π_θ(y_l|x)/π_ref(y_l|x)) ) ]
Group Relative Policy Optimization (GRPO — Shao et al., 2024)
In reasoning domains (e.g., DeepSeekMath and DeepSeek-R1 [6]), rewards are typically sparse and evaluated only at the final answer token, making a learned token-level Critic V_ψ noisy and expensive. GRPO removes the Critic entirely: for each prompt q, it samples a group of G completions {o_1, ..., o_G} from π_{θ_old}, scores each completion with reward r_i, and normalizes rewards across the group to form the advantage Â_{i,t}:
Unbiased Low-Variance KL Estimator (Schulman k3):
D_KL(π_θ || π_ref) = (π_ref(o_{i,t}|q, o_{i,<t}) / π_θ(o_{i,t}|q, o_{i,<t})) - log(π_ref(o_{i,t}|q, o_{i,<t}) / π_θ(o_{i,t}|q, o_{i,<t})) - 1
| Dimension | PPO-RLHF | DPO | GRPO |
|---|---|---|---|
| Sampling Regime | On-policy rollouts | Offline static preference pairs | On-policy group rollouts (G per prompt) |
| Models in Memory | 4 (Actor, Ref, Critic, RM) | 2 (Policy, Ref) | 2 with rule verifier; 3 with neural RM |
| Advantage Baseline | Learned Value Network V_ψ(s_t) (GAE) |
N/A (implicit reward margin) | Group mean & std z-score normalization |
| KL Regularization | Subtracted from per-token reward | Implicit via closed-form reparameterization | Explicit k3 KL term added directly to loss |
| Primary Failure Mode | Reward hacking; Critic instability | Off-policy drift; likelihood displacement | Zero-variance groups (all r_i identical) |
Python
4. Reference Implementation: Direct Preference Optimization (DPO) Loss
Below is a self-contained PyTorch implementation of the DPO objective [5]. Notice how sequence log-probabilities are summed across response tokens (masking out prompt tokens) and how gradient updates dynamically weight pairs by σ(-u)—amplifying gradients when the policy assigns higher implicit reward to the rejected completion y_l than the chosen completion y_w.
import torch
import torch.nn.functional as F
def dpo_loss(
policy_chosen_logps: torch.Tensor,
policy_rejected_logps: torch.Tensor,
ref_chosen_logps: torch.Tensor,
ref_rejected_logps: torch.Tensor,
beta: float = 0.1,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
"""
Computes Direct Preference Optimization (DPO) loss (Rafailov et al., 2023).
All inputs are sequence-summed log-probabilities of shape (batch_size,).
"""
# Implicit log-probability ratios relative to frozen reference policy
chosen_logratios = policy_chosen_logps - ref_chosen_logps
rejected_logratios = policy_rejected_logps - ref_rejected_logps
# Preference margin logits: beta * (log(pi/pi_ref)(y_w) - log(pi/pi_ref)(y_l))
logits = beta * (chosen_logratios - rejected_logratios)
# Binary cross-entropy over Bradley-Terry preference probability
losses = -F.logsigmoid(logits)
# Diagnostic implicit rewards for telemetry monitoring
chosen_rewards = beta * chosen_logratios.detach()
rejected_rewards = beta * rejected_logratios.detach()
return losses.mean(), chosen_rewards.mean(), rejected_rewards.mean()
# Simulated batch of 4 preference pairs
torch.manual_seed(42)
pi_w = torch.tensor([-12.4, -18.1, -9.5, -15.0], requires_grad=True)
pi_l = torch.tensor([-14.2, -16.0, -11.8, -14.8], requires_grad=True)
ref_w = torch.tensor([-13.0, -18.0, -10.2, -15.1])
ref_l = torch.tensor([-13.5, -16.5, -11.0, -15.0])
loss, r_w, r_l = dpo_loss(pi_w, pi_l, ref_w, ref_l, beta=0.1)
loss.backward()
print(f"DPO Loss: {loss.item():.4f}")
print(f"Mean Implicit Chosen Reward: {r_w.item():+.4f}")
print(f"Mean Implicit Rejected Reward: {r_l.item():+.4f}")
print(f"Reward Accuracy (r_w > r_l): {((pi_w - ref_w) > (pi_l - ref_l)).float().mean().item()*100:.1f}%")
Python
5. Reference Implementation: GRPO Group Advantage & Surrogate Loss
In GRPO [6], each prompt spawns a group of G rollouts scored by a verifiable reward function (e.g., exact mathematical answer match + formatting check). Notice that if all G rollouts in a group receive the exact same reward (e.g., all 0.0 or all 1.0), the group standard deviation is zero and the prompt produces zero policy gradient—motivating dynamic curriculum filtering in production RLVR pipelines.
import torch
def grpo_step_loss(
policy_token_logps: torch.Tensor, # shape: (G, T)
old_policy_token_logps: torch.Tensor, # shape: (G, T)
ref_token_logps: torch.Tensor, # shape: (G, T)
group_rewards: torch.Tensor, # shape: (G,)
response_mask: torch.Tensor, # shape: (G, T) binary mask
clip_eps: float = 0.2,
kl_beta: float = 0.04,
) -> tuple[torch.Tensor, torch.Tensor]:
"""
Computes GRPO clipped surrogate + explicit k3 KL penalty for a single prompt group of size G.
"""
# 1. Group-normalized outcome advantage (broadcast across all response tokens)
mean_r = group_rewards.mean()
std_r = group_rewards.std(unbiased=False)
advantages = ((group_rewards - mean_r) / (std_r + 1e-8)).unsqueeze(-1) # (G, 1)
# 2. Importance sampling ratio rho_{i,t}(theta)
ratio = torch.exp(policy_token_logps - old_policy_token_logps)
unclipped = ratio * advantages
clipped = torch.clamp(ratio, 1.0 - clip_eps, 1.0 + clip_eps) * advantages
surrogate = torch.min(unclipped, clipped)
# 3. Non-negative Schulman k3 KL estimator: (pi_ref / pi_theta) - log(pi_ref / pi_theta) - 1
log_ref_over_policy = ref_token_logps - policy_token_logps
kl_div = torch.exp(log_ref_over_policy) - log_ref_over_policy - 1.0
# 4. Masked per-token objective averaged over valid completion tokens
per_token_loss = -(surrogate - kl_beta * kl_div) * response_mask
loss = per_token_loss.sum() / response_mask.sum().clamp(min=1.0)
mean_kl = (kl_div * response_mask).sum() / response_mask.sum().clamp(min=1.0)
return loss, mean_kl
# Simulated group of G=4 rollouts, sequence length T=6
torch.manual_seed(7)
G, T = 4, 6
old_logps = torch.randn(G, T) - 1.5
cur_logps = (old_logps + 0.05 * torch.randn(G, T)).detach().requires_grad_(True)
ref_logps = old_logps - 0.03 * torch.randn(G, T)
rewards = torch.tensor([1.0, 0.0, 1.0, 0.0]) # 2 correct, 2 incorrect rollouts
mask = torch.ones(G, T)
loss, mean_kl = grpo_step_loss(cur_logps, old_logps, ref_logps, rewards, mask)
loss.backward()
print(f"Normalized Advantages: {((rewards - rewards.mean()) / (rewards.std(unbiased=False) + 1e-8)).tolist()}")
print(f"GRPO Step Loss: {loss.item():.4f}")
print(f"Mean Token KL (k3): {mean_kl.item():.5f}")
Quiz
6. Questions and Explanations
Select the best answer for each of the 8 questions below, then click Submit & Grade Quiz to evaluate your understanding with source-cited explanations.
Q1 · Knowledge checkWhy do RLHF, DPO, and GRPO all include a KL-divergence regularization term relative to the SFT reference policy π_ref?
Q2 · Knowledge checkHow does Direct Preference Optimization (DPO) avoid training an explicit Reward Model and running online RL rollouts?
β log Z(x) terms in r(x, y_w) - r(x, y_l) cancel identically in the Bradley-Terry preference probability, turning RLHF into a supervised binary cross-entropy loss using only the policy and frozen reference model.
Q3 · Knowledge checkHow many LLM-scale models must typically reside in GPU memory simultaneously during standard online PPO-RLHF training?
π_θ, frozen Reference π_ref, frozen Reward Model r_φ, and a trainable Critic V_ψ of comparable size to the Actor.
Q4 · Knowledge checkHow does Group Relative Policy Optimization (GRPO) eliminate the Critic (Value) network required by PPO?
G sampled completions for the same prompt, eliminating the Critic network and cutting RL memory overhead roughly in half.
Q5 · Knowledge checkWhat happens during a GRPO training step if a prompt group of G = 8 rollouts receives identical rewards (e.g., all 0.0 on a very hard math problem)?
r_i - mean({r_1..r_G}) = 0 when all rollouts achieve identical rewards (and ε prevents division by zero), the surrogate advantage vanishes—leaving only the KL regularization term active for that prompt [6].
Q6 · Knowledge checkIn the Bradley-Terry preference model used to train RLHF reward models, how is the probability that completion y_w is preferred over y_l expressed?
P(y_w ≻ y_l | x) = 1 / (1 + exp(-(r(x, y_w) - r(x, y_l)))).
Q7 · Knowledge checkWhat is the key innovation of Constitutional AI (Bai et al., 2022)?
Q8 · Knowledge checkWhy does GRPO use Schulman's k3 estimator (π_ref / π_θ) - log(π_ref / π_θ) - 1 directly in the loss function rather than subtracting β log(π_θ / π_ref) inside the reward?
exp(x) - x - 1 ≥ 0), the k3 estimator is guaranteed non-negative on every token, is unbiased in expectation under π_θ, and keeps group reward normalization clean by decoupling KL regularization from r_i [6].
References
References and source links
-
Bradley, R. A., & Terry, M. E. (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4), 324–345.
https://doi.org/10.2307/2334029 -
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., & Amodei, D. (2017). Deep Reinforcement Learning from Human Preferences. Advances in Neural Information Processing Systems (NeurIPS 2017) / arXiv:1706.03741.
https://arxiv.org/abs/1706.03741 -
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback (InstructGPT). NeurIPS 2022 / arXiv:2203.02155.
https://arxiv.org/abs/2203.02155 -
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.
https://arxiv.org/abs/1707.06347 -
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023 / arXiv:2305.18290.
https://arxiv.org/abs/2305.18290 -
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., & Guo, D. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (Group Relative Policy Optimization). arXiv:2402.03300.
https://arxiv.org/abs/2402.03300 -
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073.
https://arxiv.org/abs/2212.08073 -
Gao, L., Schulman, J., & Hilton, J. (2023). Scaling Laws for Reward Model Overoptimization. International Conference on Machine Learning (ICML 2023) / arXiv:2210.10760.
https://arxiv.org/abs/2210.10760