WebArena-Pro benchmark tests web agents across 20 apps with audio and video tasks
The arXiv paper introduces 300 tasks on self-hosted applications, adding audio and video observations while keeping evaluation reproducible, and reports results for several agent configurations.
In plain words
Web agents are AI systems that operate a browser to complete tasks. The WebArena-Pro authors built 300 tasks across 20 self-hosted web applications so these agents can be tested on realistic, multimodal work. Because the applications are self-hosted, runs stay reproducible, unlike live benchmarks where pages drift.
What was announced
The paper argues that existing reproducible benchmarks cover few applications and restrict evaluation to text and vision, while live benchmarks broaden site coverage but lose reproducibility as pages and data drift between runs. WebArena-Pro addresses this with 300 tasks across 20 self-hosted web applications in six domain categories, treating audio and video as core observations alongside text and vision, with dedicated actions for extracting information from each. Tasks run in isolation and support parallel evaluation.
On results reported by the authors, Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5 tie for the best performance at 62.3%, with OpenCode using Kimi 3.1 Pro at 60.7%. The strongest GenericAgent configurations, Gemini 3.1 Pro omni and Gemma4 31B, both reach 30.3%. Tasks were authored through a dedicated annotator interface, filtered by LLM-assisted triage, and validated by humans before release.
What it does not show
- The evidence is the paper's abstract only; the full paper's methods, task details and evaluation setup were not available for review.
- Reported scores come from the authors' own evaluations and are not independently confirmed.
- The abstract does not state how many agents were evaluated in total or the evaluation conditions behind each score.
For security and IT teams
- Scope
- The benchmark covers 20 self-hosted web applications in six domain categories, with tasks run in isolation for reproducible, parallel evaluation.
Sources
Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 17 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.