Bradley-Terry training can distort accurate language world model probabilities, study finds
An arXiv paper reports that a common way of training language world models from comparative feedback can shift probability away from outcomes the model previously handled well.
In plain words
Language world models predict how likely different results of an action are. The paper's authors show that training such a model with a judge that picks which of two candidate results is closer to reality can make its probabilities less accurate, even when they started out good. In their simulated tests, a large share of probability moved to results that never occur.
What was announced
The authors, writing on arXiv, tested a Bradley-Terry objective, a fitting method for learning from comparative feedback, on a simulated robot push task with enumerated outcome distributions. A model that was already accurate on unseen layouts, with a hazard error of 2.4 percentage points, lost 33 to 43 percentage points of probability to outcomes that no branch of a condition's enumerated list produces. They attribute the problem to the objective itself rather than the comparisons: its optimum, calculated before training, predicts where the model ends up, and a likelihood method that knows the judge's rule learns from the same comparisons without the shift.
The distortion persisted across unseen conditions, five candidate laws and three-fold denser enumeration, with roughly 29 percentage points remaining after excluding physically impossible events. Its practical cost depends on those events: with them, a decision threshold carried over from the pre-feedback model lost about 3 percentage points of utility per decision on trained conditions and 1 to 2 on unseen ones, via false rejects; without them, average utility loss stayed below 1 percentage point over six seeds, though a small action-choice regret remained. Audits of readout, response and decision stages traced the effect from probabilities through to actions.
What it does not show
- Evidence comes from the paper's abstract only; the full article was not reviewed.
- Findings rest on a single simulated robot push task with enumerated outcomes, not on real-world deployments or other model types.
- The abstract is truncated mid-sentence, so some reported results may be incomplete.
Sources
Source-support review completed 21 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 21 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.