← News

Agents & MCPSources: arXiv · 19 Nov 2026

MARBLE benchmark shows frontier multimodal models struggle to turn images into spatial states

A new diagnostic benchmark, MARBLE, tests whether multimodal language models can infer spatial states from images and plan over them, and results show frontier models perform poorly.

In plain words

Researchers released MARBLE, a test made of 2D and 3D puzzles that checks whether AI models can look at a picture, work out the spatial arrangement it shows, and plan moves from there. The strongest model tested solved only 7.3% of the hardest tasks, and models did far worse when they had to read the layout from images themselves.

What was announced

The benchmark's authors, writing on arXiv, describe MARBLE as a diagnostic benchmark spanning three agent environments, CUBE, MAZE and BLOX, covering 2D and 3D spatial puzzles with dynamic layouts and transformations. Difficulty is controlled along two axes: how observations are presented and how long a planning horizon is required. In the hardest setting, the strongest model, GPT-5.4, achieved only a 7.3% average success rate across the three environments, according to the paper.

Ablation tests on open-weight models point to visual-to-spatial abstraction as the main bottleneck. When Gemma-4-31B had to infer spatial states from images rather than receive human-annotated states, its success rate fell from 50.7% to 3.7%; Kimi-K2.6 dropped from 87.9% to 6.5%. The authors conclude that models often fail to build precise spatial states from visual observations before planning, a step they describe as straightforward for humans, and position MARBLE as a controlled testbed for measuring progress.

What it does not show

  • The evidence is the paper's abstract only, so evaluation details, methods and full results are not available for verification.
  • The reported scores come from the authors' own evaluations and have not been independently confirmed.
  • The abstract does not specify which models beyond GPT-5.4, Gemma-4-31B and Kimi-K2.6 were tested, nor how many tasks each environment contains.

Sources

Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 20 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.