Computer-use agents judge apps by interacting, not just reading code
An arXiv paper proposes Interaction-as-a-Judge, where computer-use agents operate applications directly, reportedly improving evaluation and generation of interactive software.
In plain words
Automated checks of AI-written apps usually read code or run unit tests, so they can miss problems a human would spot by clicking around. This paper has computer-use agents — AI agents that operate software like a person — actually use the app and judge it. The authors report this improves both scoring apps and fixing broken ones.
What was announced
The paper, an arXiv abstract, argues that because humans assess interactive applications such as websites and games by using them, while automated LLM evaluations focus on code analysis and unit tests, judges can miss failures obvious to human users. Interaction-as-a-Judge instead has computer-use agents interact with the application directly. The authors released two datasets: 6,858 application pairs with human preferences and agent traces, and 222 applications with human-verified requirement labels.
On reported results, interactive judging improved prediction of human preferences from 65.6% to 69.1% and requirement verification from 77.0% to 81.6% versus non-interactive approaches, with higher-quality interactions performing better. For generation, it repaired 75.5% of broken requirements within five attempts at inference time versus 55.4% for code-only self-feedback, and used as a training reward it raised win-rate by 6.7 percentage points while cutting code length by 35.7% and build failures by 7.2 points. These figures come from the authors' own experiments and are not independently confirmed.
What it does not show
- Evidence is limited to the paper's abstract; full methods, benchmarks and caveats are not visible, and the reported numbers are the authors' own results without independent confirmation.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.