OCTAVE benchmark tests whether AI agents can build complete HTML5 games
A new arXiv paper introduces OCTAVE, a benchmark that grades end-to-end HTML5 game generation across eight capability dimensions instead of a single score.
In plain words
OCTAVE is a test that checks whether AI systems can build a whole working game from a request, not just parts of one. It breaks each game task into many small requirements and reports which ones the AI met. The paper's authors found the best systems still fail more than half of the requirements.
What was announced
The benchmark contains 180 tasks with 1,800 evaluation requirements, drawn from real player requests, reconstruction of commercial game mechanics, and human expert design, and spanning PC Web, Mobile Web, 2D and 3D. Requirements are split into Completeness, covering publicly specified core implementation, and Richness, covering hidden, deeper system relationships. Static source-code judgments let the authors diagnose performance requirement by requirement rather than through one aggregate number.
Across 11 model-harness combinations, the paper reports 33.2% Completeness and 32.5% Richness, with 19.9% of generation records failing static gates or missing artifacts. Even excluding same-model self-evaluation, the best combination stayed below 50% on both requirement types. Systems did relatively better on Core Rules and Numerical Design, while AI/NPC behaviour and Feedback and Game Feel showed larger gaps. A limited human cross-check found the main disagreement ran toward requirements humans judged satisfied but the automated evaluator failed, suggesting the scores may understate performance in some cases.
What it does not show
- The evidence is the paper's abstract only; the full methodology, task examples and per-model results are not available for verification.
- The human cross-check is described as limited, and requirement difficulty was only weakly correlated with pass rates, so difficulty stratification is coarse.
- No independent party has reproduced the benchmark results.
Sources
Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 17 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.