SciWorkBench tests whether AI agents can handle real scientific work
A new arXiv benchmark from the SciWorkBench authors evaluates agents on science tasks drawn from peer-reviewed literature, and the best model still scores well below perfect.
In plain words
Researchers released a benchmark called SciWorkBench that checks whether AI agents — models that plan and carry out multi-step tasks — can do the kind of work real scientists do. The best agent scored 57.38 on average, so current systems still fall short on realistic scientific workflows.
What was announced
The benchmark's creators built SciWorkBench from peer-reviewed literature and expert-contributed research scenarios, covering biology, chemistry, materials science, and physics. They also propose a unified scientific analysis framework, validated by domain experts, that categorizes the tasks; each task maps to one or more stages of that framework.
Testing recent models inside the Claude Code and Codex agent frameworks under a single execution and grading protocol, the authors found the top agent reached a mean score of 57.38. Their failure analysis points to three recurring weaknesses: turning scientific knowledge into task-specific procedures, combining evidence across different modalities, and self-correcting during scientific work. The authors argue that workflow-based evaluation matters for measuring progress toward reliable AI research systems.
What it does not show
- The source is the paper's abstract only, so task counts, grading details, and per-model results are not available to verify.
- Scores come from the benchmark authors' own evaluation and have not been independently confirmed.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.