OfficePrism benchmark tests AI agents on single-domain enterprise tasks
A new arXiv paper introduces a benchmark of 420 tasks across seven business functions, finding wide pass-rate gaps among 124 model-harness-package configurations.
In plain words
The authors built a benchmark that checks whether AI agents can complete office tasks within one business function, such as handling a ticket, rather than juggling many departments at once. Each task is verified automatically with a hidden executable reference and human review. In their tests, success rates varied widely, from about 23% to 76%.
What was announced
The paper argues that broad workplace scores can blend genuine competence in one function with the added cost of coordinating across systems, which makes failures hard to localize. OfficePrism therefore measures single-domain performance using 420 tasks across seven business functions, with benchmark-defined L1-L3 task-structure levels held constant. Each task runs in a fresh environment pairing a shared synthetic organization with one function-specific domain and typed state access.
The construction pipeline combines grounded records, business rules, and tool schemas to produce a Chinese-language ticket, a hidden executable reference witness, and an expected response, followed by automated filtering and human curation. Evaluating 124 model-harness-package configurations, the authors report per-case joint natural-language-and-database pass rates between 22.6% and 76.4%, with a median of 67.9%. L1 tasks scored highest in all configurations, and L2 exceeded L3 in 122 of 124. Comparing two interface packages on 50 matched configurations, the search-and-dispatch package scored higher in 30, tied in one, and scored lower in 19.
What it does not show
- The evidence is limited to the paper's abstract; full methodology, task examples, and model identities are not available in the supplied passages.
- Pass rates reflect the authors' own benchmark setup and have not been independently confirmed.
For security and IT teams
- Scope
- The benchmark covers single-domain enterprise tasks in seven business functions, not cross-function coordination, and tasks are in Chinese.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.