← News

Agents & MCPSources: arXiv · 19 Nov 2026

Openhagen Worlds: 27 calibrated environments for testing scientific discovery agents

The arXiv paper introduces procedurally generated environments where agents run budgeted experiments and commit answers checked by a deterministic verifier against hidden truth.

In plain words

Researchers released a set of 27 test environments for AI agents that do science by running experiments and reasoning from results. Each environment states its scoring rules openly and is calibrated so that a scripted reference method passes while simple shortcut strategies fail. Testing one frontier model showed it solved many but not all withheld tasks, leaving measurable headroom.

What was announced

The authors argue that most existing discovery benchmarks cannot separate genuine experimental reasoning from recall, unstated scoring rules or shortcut policies. Openhagen Worlds addresses this by having every observation state the complete scoring rule, and by admitting each of the 178 difficulty levels only if a scripted oracle succeeds on at least 0.75 of 64 development instances, while the strongest plausible non-deductive policy stays at or below 0.25 and each gated shortcut at or below 0.35. The suite spans classic experimental-reasoning puzzles and simulator-grounded physics and chemistry, with agents working under experiment budgets.

On 84 withheld frontier instances, one per family and rung, DeepSeek V4.1 Flash solved 32 at default reasoning effort with 16,384 tokens per call, and 57 at maximum effort with up to 393,216 tokens per call, against an oracle score of 79. The authors report remaining headroom is largest in eight-stage campaigns. Training also appears possible: on-policy expert iteration raised a 9B model's core-tier success on trained families from 0.090 to 0.148, partly through better action formatting, with no detectable loss on GPQA Diamond. A hosted service plays the core tier's withheld instances.

What it does not show

  • Evidence is limited to the paper's abstract; full methodology, baselines and caveats are not visible in the supplied excerpt.
  • The DeepSeek V4.1 Flash results come from the paper's authors and are not independently confirmed.
  • The 9B model training gains are modest and attributed partly to action formatting, so the extent of genuine reasoning improvement is unclear from the abstract.

For security and IT teams

Access
A hosted service plays the core tier's withheld instances, per the paper's abstract.

Sources

Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 20 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.