Code4Scene benchmark tests whether coding agents can build and edit 3D scenes reliably
A new arXiv benchmark evaluates 14 coding-agent systems on 170 Unreal Engine scene tasks, finding plausible-looking 3D output often hides flawed spatial reasoning and unintended edits.
In plain words
Coding agents are AI systems that write and run code. The Code4Scene benchmark asks them to build 3D scenes from text descriptions and to edit existing scenes so they match reference images. Instead of judging code or pictures, it checks the actual 3D scene files for errors like overlapping objects or unwanted changes.
What was announced
The benchmark's authors argue that a rendered image can look convincing while the underlying scene contains incorrect spatial relations, intersecting objects, or unintended modifications, so Code4Scene scores the generated engine-native scene itself. It measures task fulfillment, artifact integrity, and static physical validity, with editing results additionally compared against withheld ground truth. Construction tasks start from open-ended language specifications where many valid realizations exist; editing tasks require recovering a target scene from reference images while preserving everything else.
Across 14 coding-agent systems on the 83-case public set, construction and editing performance were strongly correlated but not interchangeable, with a Spearman correlation of 0.67. Claude Fable 5.1 led construction while Gemini 3.8 Flash led editing. Spatial Composition was the weakest construction category for every agent, and editing stayed imprecise: the best Repair F1 was 0.562, and 34.2% of edits that fully recovered the target still introduced unintended changes elsewhere. The authors say this exposes a gap between plausible 3D generation and reliable spatial reasoning and state control.
What it does not show
- The source is the paper's abstract only, so methodology details, evaluation procedures, and full results are not available for verification.
- Reported scores cover the 83-case public set, not the full 170-case benchmark, and the abstract does not state results for the withheld cases.
- The benchmark comes from a single paper; no independent evaluation of the agents or scores is reported.
For security and IT teams
- Scope
- The benchmark covers 170 Unreal Engine cases with an 83-case public set, and a project webpage is available at https://simworld.org/code4scene.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.