Controlled benchmark shows RL gains for mobile GUI agents shrink as tasks drift from training
A new arXiv study introduces AndroidWorld-Generalization, a benchmark measuring how online reinforcement learning transfers across increasingly distant mobile task contexts.
In plain words
Reinforcement learning trains AI agents by rewarding successful actions, and it has pushed mobile phone agents to high success rates on a benchmark called AndroidWorld. The paper's authors built a harder test with three levels of distance from training tasks. They found the learning helps most on familiar tasks and least on new apps.
What was announced
The authors, writing on arXiv, note that RL-trained mobile GUI agents have exceeded 80% success on AndroidWorld, but typically train and evaluate in the same environment, and their training tasks are not released for auditing. They formalize the mobile setting as a Contextual MDP and define three progressively distant context shifts: Unseen Instance, Unseen Template, and Unseen Application, with separation verified manually. They also release an open RL training system intended to make studies at this scale feasible.
Applying the same RL algorithm to two starting models of different origins and competence, UI-TARS-7B-SFT at 23.0% and Qwen3-VL-8B at 52.6%, the researchers report that RL improves both in distribution, but gains fall monotonically up the ladder: +17.5/+11.1/+4.2 and +10.3/+7.4/+0. Training performance rises throughout while evaluation performance saturates, and the decay is algorithm-agnostic, reproduced under PPO. RL contributes less as the prior strengthens. Few-shot adaptation in the target application recovers +22.9, which the authors say motivates test-time adaptation alongside further RL training. The benchmark, training system, recipes, and model weights are released.
What it does not show
- The source is the paper's abstract only; full methodology, task details, and evaluation conditions are not available in the supplied passages.
- Generalization beyond the two tested starting models and the AndroidWorld environment is not established by the evidence.
- The reported gains are the authors' own measurements and have not been independently confirmed.
For security and IT teams
- Scope
- The benchmark covers three context-shift levels on AndroidWorld (Unseen Instance, Unseen Template, Unseen Application), not general mobile agent performance.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.