MentalGrid benchmark shows multimodal models struggle to simulate changing scenes
A new arXiv paper introduces MentalGrid, a benchmark testing whether multimodal language models can track spatial states that evolve without visual feedback, and proposes a training method that improves results.
In plain words
Multimodal models can describe objects in a picture, but the authors tested something harder: imagining how a scene changes step by step without seeing intermediate images. Their benchmark, MentalGrid, found current models perform poorly at this. The team then trained a model, SpatialWalker, that does much better on these tasks.
What was announced
The paper, posted to arXiv by the MentalGrid authors, argues that prior work has mostly advanced how models perceive spatial relations in static scenes, while dynamic spatial simulation remains largely unexplored. Their benchmark asks models to simulate evolving spatial states without intermediate visual feedback. Evaluation found a significant deficiency in existing models: performance is constrained by perceptual limits in simple scenarios and by inadequate multi-step reasoning as complexity grows, according to the abstract.
As a remedy, the authors propose a three-stage curriculum learning framework built on 16k specialized spatial reasoning samples, producing a model called SpatialWalker. They report a 33.8% average improvement over its base model, surpassing proprietary models such as Gemini-3-Flash in spatial simulation, plus a 4.9% average gain on out-of-domain benchmarks. These figures come from the authors' own evaluation and have not been independently verified.
What it does not show
- The source is the paper's abstract only; full methodology, task details and evaluation setup are not available for verification.
- All reported improvements, including the 33.8% and 4.9% gains, are author-reported and lack independent confirmation.
- The abstract does not specify which models were evaluated, how many, or the benchmark's size and scoring methods.
For security and IT teams
- Scope
- The findings concern spatial simulation in multimodal large language models as described in a research abstract; no deployment, security or product-availability information is included.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.