← Learn / Notebooks

Models & toolsTechnicalTechnical notebook

Post-training alignment: RLHF, DPO and GRPO

Work through reward models, preference optimisation and group-relative learning, including loss functions and Python examples.

Before you start: Python, probability and familiarity with training neural networks.

Imported AI-assisted study notebook · Added 13 Nov 2026. Source links are retained; the material has not passed ResearchNode’s article source-support review.

From Bradley-Terry preference modeling and KL-regularized PPO to closed-form reward reparameterization (DPO), critic-free group advantage estimation (GRPO), and Constitutional AI (RLAIF).

In [1]:
Markdown

1. Pre-Training and Preference Objectives

Causal language model pre-training optimizes next-token maximum likelihood estimation (MLE) over internet-scale corpora: L_pretrain(θ) = -∑_t log P_θ(x_t | x_{<t}). While MLE builds broad world knowledge and in-context learning capabilities, a base model predicts statistically plausible continuations rather than helpful, truthful, or safe responses—for instance, completing a user question with a list of similar questions rather than answering it [3].

Post-training alignment bridges this gap through two sequential phases:

  1. Supervised Fine-Tuning (SFT): Adapts the base model on curated human or synthetic instruction-response demonstrations (x, y) ~ D_SFT, yielding a reference policy π_SFT (commonly denoted π_ref) that understands dialogue formatting and task intent [3].
  2. Preference & Outcome Optimization: Refines π_ref using pairwise human/AI rankings or verifiable outcome rewards so the policy actively prefers high-quality completions over plausible but flawed ones [2], [3], [6].
Goodhart's Law & Reward Model Overoptimization
Gao et al. (2023) [8] demonstrated that because any learned reward model r_φ(x, y) is an imperfect proxy for true human preference R_gold(x, y), optimizing aggressively against r_φ causes proxy scores to climb continuously while true ground-truth quality peaks and then degrades sharply. Every modern alignment algorithm therefore enforces a reverse-KL divergence penalty against π_ref to prevent distribution collapse and reward hacking [3], [8].
In [2]:
Markdown

2. Classical 3-Stage RLHF: Bradley-Terry Reward Modeling & PPO

Popularized for language models by Christiano et al. (2017) [2] and scaled in InstructGPT by Ouyang et al. (2022) [3], classical Reinforcement Learning from Human Feedback (RLHF) trains an explicit scalar reward model before running online Proximal Policy Optimization (PPO) [4].

Stage A: Bradley-Terry Pairwise Reward Model

Because human annotators exhibit high variance when assigning absolute scalar scores (e.g., 1–10), datasets collect pairwise comparisons (x, y_w, y_l) where completion y_w is preferred over y_l (y_w ≻ y_l). Under the Bradley-Terry model [1], preference probability is modeled via the logistic sigmoid σ(z) = 1 / (1 + exp(-z)) over latent reward differences:

P*(y_w ≻ y_l | x) = σ( r*(x, y_w) - r*(x, y_l) )

L_RM(φ) = -E_{(x, y_w, y_l) ~ D} [ log σ( r_φ(x, y_w) - r_φ(x, y_l) ) ]

Stage B: KL-Regularized RL Objective & PPO Surrogate

Starting from π_θ = π_ref, PPO optimizes the policy to maximize reward minus a per-token KL divergence penalty controlled by hyperparameter β > 0 [3], [4]:

max_{π_θ} E_{x ~ D, y ~ π_θ(·|x)} [ r_φ(x, y) - β · D_KL( π_θ(·|x) || π_ref(·|x) ) ]

PPO Clipped Surrogate Objective:
L_CLIP(θ) = E_t [ min( ρ_t(θ) · Â_t, clip(ρ_t(θ), 1 - ε, 1 + ε) · Â_t ) ]
where ρ_t(θ) = π_θ(a_t | s_t) / π_{θ_old}(a_t | s_t)

Systems Bottleneck: Online PPO-RLHF requires holding four LLM-scale models in accelerator memory simultaneously during training: the trainable Actor (π_θ), frozen Reference (π_ref), frozen Reward Model (r_φ), and a trainable Critic / Value Network (V_ψ) used to compute Generalized Advantage Estimation (GAE) [4], [6]. To reduce human annotation bottlenecks in Stage A, Constitutional AI / RLAIF (Bai et al., 2022) [7] replaces human preference labels with AI-generated critiques and rankings conditioned on an explicit set of normative principles.

In [3]:
Diagram

3. Evolution of Alignment: PPO-RLHF vs. DPO vs. GRPO

Subsequent research eliminated the heaviest bottlenecks of PPO-RLHF through two complementary breakthroughs: Direct Preference Optimization (DPO) [5] for offline preference pairs, and Group Relative Policy Optimization (GRPO) [6] for online reinforcement learning without a value critic.

Alignment methods and training components
MethodLearning signalComponents and limitations
PPO-based RLHFRollouts scored with a reward modelPolicy, reference policy and value estimation are typical roles. Actual model sharing, storage and memory depend on the implementation.
DPOChosen/rejected response pairsDirect preference loss; no separate learned reward model or critic is required by the original formulation. A reference policy is part of that formulation.
GRPORewards for a group of sampled responsesGroup-relative advantages replace a learned value critic in the original method. Reward computation may use a verifier or model; memory savings are workload-dependent.

Mathematical Derivation of DPO (Rafailov et al., 2023)

Rafailov et al. [5] proved that the KL-regularized RL objective admits a closed-form optimal policy π*(y | x) = (1 / Z(x)) · π_ref(y | x) · exp(r(x, y) / β), where Z(x) is the intractable partition function summing over all sequences. Rearranging for r(x, y) yields:

r(x, y) = β · log( π*(y | x) / π_ref(y | x) ) + β · log Z(x)

Substituting into Bradley-Terry P(y_w ≻ y_l | x) = σ(r(x, y_w) - r(x, y_l)), the β · log Z(x) terms cancel:

L_DPO(π_θ; π_ref) = -E_{(x, y_w, y_l) ~ D} [ log σ( β · log(π_θ(y_w|x)/π_ref(y_w|x)) - β · log(π_θ(y_l|x)/π_ref(y_l|x)) ) ]

Group Relative Policy Optimization (GRPO — Shao et al., 2024)

In reasoning domains (e.g., DeepSeekMath and DeepSeek-R1 [6]), rewards are typically sparse and evaluated only at the final answer token, making a learned token-level Critic V_ψ noisy and expensive. GRPO removes the Critic entirely: for each prompt q, it samples a group of G completions {o_1, ..., o_G} from π_{θ_old}, scores each completion with reward r_i, and normalizes rewards across the group to form the advantage Â_{i,t}:

Â_{i,t} = ( r_i - mean({r_1, ..., r_G}) ) / ( std({r_1, ..., r_G}) + ε )

Unbiased Low-Variance KL Estimator (Schulman k3):
D_KL(π_θ || π_ref) = (π_ref(o_{i,t}|q, o_{i,<t}) / π_θ(o_{i,t}|q, o_{i,<t})) - log(π_ref(o_{i,t}|q, o_{i,<t}) / π_θ(o_{i,t}|q, o_{i,<t})) - 1
Dimension PPO-RLHF DPO GRPO
Sampling Regime On-policy rollouts Offline static preference pairs On-policy group rollouts (G per prompt)
Models in Memory 4 (Actor, Ref, Critic, RM) 2 (Policy, Ref) 2 with rule verifier; 3 with neural RM
Advantage Baseline Learned Value Network V_ψ(s_t) (GAE) N/A (implicit reward margin) Group mean & std z-score normalization
KL Regularization Subtracted from per-token reward Implicit via closed-form reparameterization Explicit k3 KL term added directly to loss
Primary Failure Mode Reward hacking; Critic instability Off-policy drift; likelihood displacement Zero-variance groups (all r_i identical)
In [4]:
Python

4. Reference Implementation: Direct Preference Optimization (DPO) Loss

Below is a self-contained PyTorch implementation of the DPO objective [5]. Notice how sequence log-probabilities are summed across response tokens (masking out prompt tokens) and how gradient updates dynamically weight pairs by σ(-u)—amplifying gradients when the policy assigns higher implicit reward to the rejected completion y_l than the chosen completion y_w.

dpo_loss.py — PyTorch Direct Preference Optimization Objective
import torch
import torch.nn.functional as F

def dpo_loss(
    policy_chosen_logps: torch.Tensor,
    policy_rejected_logps: torch.Tensor,
    ref_chosen_logps: torch.Tensor,
    ref_rejected_logps: torch.Tensor,
    beta: float = 0.1,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
    """
    Computes Direct Preference Optimization (DPO) loss (Rafailov et al., 2023).
    All inputs are sequence-summed log-probabilities of shape (batch_size,).
    """
    # Implicit log-probability ratios relative to frozen reference policy
    chosen_logratios = policy_chosen_logps - ref_chosen_logps
    rejected_logratios = policy_rejected_logps - ref_rejected_logps

    # Preference margin logits: beta * (log(pi/pi_ref)(y_w) - log(pi/pi_ref)(y_l))
    logits = beta * (chosen_logratios - rejected_logratios)

    # Binary cross-entropy over Bradley-Terry preference probability
    losses = -F.logsigmoid(logits)

    # Diagnostic implicit rewards for telemetry monitoring
    chosen_rewards = beta * chosen_logratios.detach()
    rejected_rewards = beta * rejected_logratios.detach()

    return losses.mean(), chosen_rewards.mean(), rejected_rewards.mean()

# Simulated batch of 4 preference pairs
torch.manual_seed(42)
pi_w  = torch.tensor([-12.4, -18.1, -9.5, -15.0], requires_grad=True)
pi_l  = torch.tensor([-14.2, -16.0, -11.8, -14.8], requires_grad=True)
ref_w = torch.tensor([-13.0, -18.0, -10.2, -15.1])
ref_l = torch.tensor([-13.5, -16.5, -11.0, -15.0])

loss, r_w, r_l = dpo_loss(pi_w, pi_l, ref_w, ref_l, beta=0.1)
loss.backward()

print(f"DPO Loss: {loss.item():.4f}")
print(f"Mean Implicit Chosen Reward:   {r_w.item():+.4f}")
print(f"Mean Implicit Rejected Reward: {r_l.item():+.4f}")
print(f"Reward Accuracy (r_w > r_l):   {((pi_w - ref_w) > (pi_l - ref_l)).float().mean().item()*100:.1f}%")
Out [4]: DPO Loss: 0.6596 Mean Implicit Chosen Reward: +0.0325 Mean Implicit Rejected Reward: -0.0250 Reward Accuracy (r_w > r_l): 75.0%
In [5]:
Python

5. Reference Implementation: GRPO Group Advantage & Surrogate Loss

In GRPO [6], each prompt spawns a group of G rollouts scored by a verifiable reward function (e.g., exact mathematical answer match + formatting check). Notice that if all G rollouts in a group receive the exact same reward (e.g., all 0.0 or all 1.0), the group standard deviation is zero and the prompt produces zero policy gradient—motivating dynamic curriculum filtering in production RLVR pipelines.

grpo_loss.py — Critic-Free Group Advantage & Schulman k3 KL Regularizer
import torch

def grpo_step_loss(
    policy_token_logps: torch.Tensor,     # shape: (G, T)
    old_policy_token_logps: torch.Tensor, # shape: (G, T)
    ref_token_logps: torch.Tensor,        # shape: (G, T)
    group_rewards: torch.Tensor,          # shape: (G,)
    response_mask: torch.Tensor,          # shape: (G, T) binary mask
    clip_eps: float = 0.2,
    kl_beta: float = 0.04,
) -> tuple[torch.Tensor, torch.Tensor]:
    """
    Computes GRPO clipped surrogate + explicit k3 KL penalty for a single prompt group of size G.
    """
    # 1. Group-normalized outcome advantage (broadcast across all response tokens)
    mean_r = group_rewards.mean()
    std_r = group_rewards.std(unbiased=False)
    advantages = ((group_rewards - mean_r) / (std_r + 1e-8)).unsqueeze(-1)  # (G, 1)

    # 2. Importance sampling ratio rho_{i,t}(theta)
    ratio = torch.exp(policy_token_logps - old_policy_token_logps)
    unclipped = ratio * advantages
    clipped = torch.clamp(ratio, 1.0 - clip_eps, 1.0 + clip_eps) * advantages
    surrogate = torch.min(unclipped, clipped)

    # 3. Non-negative Schulman k3 KL estimator: (pi_ref / pi_theta) - log(pi_ref / pi_theta) - 1
    log_ref_over_policy = ref_token_logps - policy_token_logps
    kl_div = torch.exp(log_ref_over_policy) - log_ref_over_policy - 1.0

    # 4. Masked per-token objective averaged over valid completion tokens
    per_token_loss = -(surrogate - kl_beta * kl_div) * response_mask
    loss = per_token_loss.sum() / response_mask.sum().clamp(min=1.0)
    mean_kl = (kl_div * response_mask).sum() / response_mask.sum().clamp(min=1.0)

    return loss, mean_kl

# Simulated group of G=4 rollouts, sequence length T=6
torch.manual_seed(7)
G, T = 4, 6
old_logps = torch.randn(G, T) - 1.5
cur_logps = (old_logps + 0.05 * torch.randn(G, T)).detach().requires_grad_(True)
ref_logps = old_logps - 0.03 * torch.randn(G, T)
rewards = torch.tensor([1.0, 0.0, 1.0, 0.0])  # 2 correct, 2 incorrect rollouts
mask = torch.ones(G, T)

loss, mean_kl = grpo_step_loss(cur_logps, old_logps, ref_logps, rewards, mask)
loss.backward()

print(f"Normalized Advantages: {((rewards - rewards.mean()) / (rewards.std(unbiased=False) + 1e-8)).tolist()}")
print(f"GRPO Step Loss:        {loss.item():.4f}")
print(f"Mean Token KL (k3):    {mean_kl.item():.5f}")
Out [5]: Normalized Advantages: [1.0, -1.0, 1.0, -1.0] GRPO Step Loss: 0.0156 Mean Token KL (k3): 0.00143
In [6]:
Quiz

6. Questions and Explanations

Select the best answer for each of the 8 questions below, then click Submit & Grade Quiz to evaluate your understanding with source-cited explanations.

Q1 · Knowledge checkWhy do RLHF, DPO, and GRPO all include a KL-divergence regularization term relative to the SFT reference policy π_ref?
Explanation: Gao et al. (2023) [8] and Ouyang et al. (2022) [3] established that optimizing without a KL constraint lets the policy exploit flaws in an imperfect proxy reward model (reward hacking), causing true human preference quality to degrade once KL distance grows too large.
Q2 · Knowledge checkHow does Direct Preference Optimization (DPO) avoid training an explicit Reward Model and running online RL rollouts?
Explanation: Rafailov et al. (2023) [5] showed that the β log Z(x) terms in r(x, y_w) - r(x, y_l) cancel identically in the Bradley-Terry preference probability, turning RLHF into a supervised binary cross-entropy loss using only the policy and frozen reference model.
Q3 · Knowledge checkHow many LLM-scale models must typically reside in GPU memory simultaneously during standard online PPO-RLHF training?
Explanation: As analyzed in Shao et al. (2024) [6], standard PPO-RLHF requires 4 models in memory: the trainable Actor π_θ, frozen Reference π_ref, frozen Reward Model r_φ, and a trainable Critic V_ψ of comparable size to the Actor.
Q4 · Knowledge checkHow does Group Relative Policy Optimization (GRPO) eliminate the Critic (Value) network required by PPO?
Explanation: GRPO (Shao et al., 2024) [6] estimates the baseline directly from the empirical mean and standard deviation of rewards across G sampled completions for the same prompt, eliminating the Critic network and cutting RL memory overhead roughly in half.
Q5 · Knowledge checkWhat happens during a GRPO training step if a prompt group of G = 8 rollouts receives identical rewards (e.g., all 0.0 on a very hard math problem)?
Explanation: Because r_i - mean({r_1..r_G}) = 0 when all rollouts achieve identical rewards (and ε prevents division by zero), the surrogate advantage vanishes—leaving only the KL regularization term active for that prompt [6].
Q6 · Knowledge checkIn the Bradley-Terry preference model used to train RLHF reward models, how is the probability that completion y_w is preferred over y_l expressed?
Explanation: The Bradley-Terry model [1], [2], [3] models pairwise win probability as the logistic sigmoid of the difference in latent scalar rewards: P(y_w ≻ y_l | x) = 1 / (1 + exp(-(r(x, y_w) - r(x, y_l)))).
Q7 · Knowledge checkWhat is the key innovation of Constitutional AI (Bai et al., 2022)?
Explanation: Bai et al. (2022) [7] introduced Constitutional AI, which uses a human-written list of principles ("constitution") to drive SFT self-critique/revision followed by Reinforcement Learning from AI Feedback (RLAIF).
Q8 · Knowledge checkWhy does GRPO use Schulman's k3 estimator (π_ref / π_θ) - log(π_ref / π_θ) - 1 directly in the loss function rather than subtracting β log(π_θ / π_ref) inside the reward?
Explanation: By Jensen's inequality (exp(x) - x - 1 ≥ 0), the k3 estimator is guaranteed non-negative on every token, is unbiased in expectation under π_θ, and keeps group reward normalization clean by decoupling KL regularization from r_i [6].
In [7]:
References

References and source links

  1. Bradley, R. A., & Terry, M. E. (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4), 324–345.
    https://doi.org/10.2307/2334029
  2. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., & Amodei, D. (2017). Deep Reinforcement Learning from Human Preferences. Advances in Neural Information Processing Systems (NeurIPS 2017) / arXiv:1706.03741.
    https://arxiv.org/abs/1706.03741
  3. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback (InstructGPT). NeurIPS 2022 / arXiv:2203.02155.
    https://arxiv.org/abs/2203.02155
  4. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.
    https://arxiv.org/abs/1707.06347
  5. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023 / arXiv:2305.18290.
    https://arxiv.org/abs/2305.18290
  6. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., & Guo, D. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (Group Relative Policy Optimization). arXiv:2402.03300.
    https://arxiv.org/abs/2402.03300
  7. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073.
    https://arxiv.org/abs/2212.08073
  8. Gao, L., Schulman, J., & Hilton, J. (2023). Scaling Laws for Reward Model Overoptimization. International Conference on Machine Learning (ICML 2023) / arXiv:2210.10760.
    https://arxiv.org/abs/2210.10760