← News

ResearchSources: arXiv · 13 Nov 2026

LGPO drops importance ratios from LLM reinforcement learning, cutting training time

Researchers propose Likelihood-Gated Policy Optimization, a reinforcement learning method for post-training large language models that removes old-policy likelihood computation while keeping training stable, according to their arXiv abstract.

In plain words

Training large language models with reinforcement learning usually requires computing likelihoods twice—once for an older policy and once for the current one—to build a correction term called an importance ratio. The authors propose LGPO, which replaces that ratio with a likelihood-based gate. They report similar stability and performance with roughly 18% less training time.

What was announced

Reinforcement learning is now a core step in post-training large language models, but widely used methods such as GRPO pay an overhead: they compute both old-policy and current-policy likelihoods to form importance sampling ratios. LGPO instead enforces a soft trust region constraint through likelihood-based gating, so old-policy likelihoods never need to be computed, the paper's abstract states.

The authors report two empirical findings. Removing the importance sampling correction did not harm training stability, while removing the trust region mechanism could cause training to collapse. They also note that ratio-based clipping can fail in fully on-policy training, where the importance ratio stays at 1 and the constraint never activates. Under standard settings where GRPO is stable, LGPO matched its stability and peak performance while cutting training time by about 18% on average versus an implementation that recomputes old-policy likelihoods; in fully on-policy training, where they say GRPO fails, LGPO remained stable.

What it does not show

  • The evidence is the paper's abstract only; full experimental details, model sizes, benchmarks and peer review status are not available in the supplied passage.
  • The ~18% training-time reduction and stability comparisons are the authors' own reported results and have not been independently confirmed.
  • The abstract does not specify which models, datasets or training regimes beyond fully on-policy and standard settings were tested.

For security and IT teams

Scope
The method targets reinforcement learning post-training of large language models and is described in a cross-listed arXiv abstract, not a full published paper.

Sources

Source-support review completed 14 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 14 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.