arXiv study finds overnight learning from user feedback helps small models only with careful judging
A new arXiv paper reports that a small language model fine-tuned nightly on user-approved attempts improved, then decayed, and that how feedback is judged decides whether learning holds.
In plain words
Researchers tested whether a small language model could improve by retraining each night on the attempts its users seemed to accept. Because nobody checks every attempt, something must guess which lessons are valid, and that guess can be wrong. The study found learning worked only when bad lessons were filtered out and users could actually see the results.
What was announced
The authors, writing on arXiv, ran experiments with small Qwen3 models of 0.6B and 1.7B parameters fitted with a LoRA adapter, fine-tuned each night on the attempts a judge accepted. They simulated 127 lives of up to a year on the WebShop and tau2-bench environments, comparing three signals: the true outcome, the user's stated verdict, and the model's own judgment, plus a separate frozen "feedback sense" module that reads the user's reaction.
The results were mixed. A sense that mistook polite give-ups for success collapsed three of five lives. A more discerning version, which first asks whether the user gave a clear verdict, kept four of five lives near the oracle for two weeks before fading to the level of self-judging by day 60. On tau2-bench, where users cannot see the outcome, the sense read reactions correctly, yet 80 percent of its lessons were failures the customer had thanked. Across 97 lives, the share of failed attempts among the lessons predicted the outcome, with a Spearman rho of -0.81. The authors conclude that learning from users held up only where the judge kept failures out and the user could see the outcome, and they note that lopsided lessons may also narrow the model.
What it does not show
- The evidence is the paper's abstract only, so methods, full results and peer review status cannot be verified from the supplied passages.
- All findings come from the authors' own experiments on WebShop and tau2-bench; no independent confirmation is available.
- The abstract reports aggregate results across simulated lives and does not establish how the approach performs in real deployments.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.