← News

Agents & MCPSources: arXiv · 19 Nov 2026

WorkWorlds benchmark finds agent scores shift when task environments change

A new arXiv paper introduces WorkWorlds, an infrastructure that rebuilds a real pharmaceutical company so researchers can measure how environment construction affects AI agent benchmark results.

In plain words

Benchmarks that test AI agents on office work usually pick the files and context for each task after the task is written. The WorkWorlds team rebuilt a real company and found that when agents saw all materials an employee would actually see, their scores dropped. Environment choices, in other words, can make agents look better than they are.

What was announced

Many knowledge work benchmarks define a task first and select its context and files at the same time or afterwards, a construction method researchers have argued does not reflect real workplace work, but whose impact had not been measured under controlled conditions. The WorkWorlds infrastructure modularizes company components into revision, date, and employee seat so each can be adjusted separately. Using historical data donated by a real company, the authors reconstructed a pharmaceutical company called PharmaCo. Two experts, each with over ten years of industry experience, wrote eight tasks based on real events at the company, and each independently built a task-specific environment for every task.

Across 192 controlled evaluations, switching the agent from the task-specific environment to all materials truly visible to the employee dropped evidence access from 90.4 percent to 72.8 percent and the criterion pass rate from 76.7 percent to 68.0 percent. When agents did find the required evidence, pass rates were almost identical under both conditions. Swapping between the two experts' environments for the same task produced an average pass rate difference of 6.6 percentage points, and three agents gained by different amounts, changing the gaps between them. The authors argue benchmarks should report how environment materials were selected and, where possible, test score changes after changing the environment builder.

What it does not show

  • The source is the paper's abstract only, so details of the three agents, the criteria, and the reconstruction method are not available to verify.

Sources

Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 20 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.