EMRD benchmark tests whether vision-language models can explore, map, remember and decide safely
Researchers introduce EMRD, an evaluation framework that breaks spatial reasoning into four stages and scores eleven vision-language models across 16 buildings under normal and stressed conditions.
In plain words
Vision-language models are AI systems that interpret images and text, and they are increasingly used to control robots. The EMRD framework checks how well they explore unknown rooms, record what they see, retain that information and make a safe final choice. In tests, the models often missed rooms and objects, and their decisions were swayed by room labels.
What was announced
The authors, writing on arXiv, present EMRD as a way to decouple spatial reasoning into four stages: Explore, covering what the agent observes; Map, what it records and where; Remember, what it retains; and Decide, whether the final choice is justified. The framework scores performance across 18 stage-specific metrics. Eleven open- and closed-weight models were evaluated across 16 buildings, with tasks including exploring unknown rooms, mapping observed objects and selecting the safest gathering place.
The reported results suggest substantial weaknesses. Open-weight models left rooms unvisited in 95% of episodes and detected at most 51% of objects. Closed-weight models explored more, but with an external memory aid, two of the three stopped prematurely. Without external memory, most models retained fewer of the objects they had seen, especially in larger buildings. Starting location shaped which rooms were explored, and open-weight models starting in different rooms agreed on the final room little more than chance. Choices also followed semantic priors: relabelling a room was often enough to change the decision. Environmental stresses mattered too. Motion noise raised object position error by 12-60% for closed-weight models, strong darkness cut map recall by 28-31 percentage points, and dense smoke reduced seen objects by 25-39 points, though the choice among entered rooms did not change significantly.
What it does not show
- The evidence is the paper's abstract only; full methodology, experimental setup and author responses to critique are not covered.
For security and IT teams
- Scope
- The findings concern vision-language models used as embodied controllers in simulated safety-critical navigation tasks, evaluated under nominal conditions plus darkness, smoke and motion noise.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.