← News

Agents & MCPSources: arXiv · 16 Nov 2026

WebArena-Pro benchmark tests web agents across 20 apps with audio and video tasks

The arXiv paper introduces 300 tasks on self-hosted applications, adding audio and video observations while keeping evaluation reproducible, and reports results for several agent configurations.

In plain words

Web agents are AI systems that operate a browser to complete tasks. The WebArena-Pro authors built 300 tasks across 20 self-hosted web applications so these agents can be tested on realistic, multimodal work. Because the applications are self-hosted, runs stay reproducible, unlike live benchmarks where pages drift.

What was announced

The paper argues that existing reproducible benchmarks cover few applications and restrict evaluation to text and vision, while live benchmarks broaden site coverage but lose reproducibility as pages and data drift between runs. WebArena-Pro addresses this with 300 tasks across 20 self-hosted web applications in six domain categories, treating audio and video as core observations alongside text and vision, with dedicated actions for extracting information from each. Tasks run in isolation and support parallel evaluation.

On results reported by the authors, Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5 tie for the best performance at 62.3%, with OpenCode using Kimi 3.1 Pro at 60.7%. The strongest GenericAgent configurations, Gemini 3.1 Pro omni and Gemma4 31B, both reach 30.3%. Tasks were authored through a dedicated annotator interface, filtered by LLM-assisted triage, and validated by humans before release.

What it does not show

  • The evidence is the paper's abstract only; the full paper's methods, task details and evaluation setup were not available for review.
  • Reported scores come from the authors' own evaluations and are not independently confirmed.
  • The abstract does not state how many agents were evaluated in total or the evaluation conditions behind each score.

For security and IT teams

Scope
The benchmark covers 20 self-hosted web applications in six domain categories, with tasks run in isolation for reproducible, parallel evaluation.

Sources

Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 17 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.