CueClash benchmark shows frontier vision-language models lean on picture-plane cues over 3-D layout
A new arXiv paper, CueClash, separates what vision-language models see in a picture from the underlying scene layout, and finds the models often trust the picture.
In plain words
Vision-language models answer questions about 3-D relations, such as whether one object fits through an opening, by reading either the flat image or the real spatial layout behind it. The CueClash authors built rendered scenes where these two viewpoints disagree, and found the tested models score much higher when the image alone gives the right answer. The weakness sits in converting the picture into a mental map of the scene.
What was announced
The paper's authors introduce CueClash, a set of rendered scenes in which, for each of five spatial relations, the 2-D cue agrees with the correct answer on exactly half the statements. They measure a "shortcut gap": accuracy when the picture cue is right, minus accuracy when it is wrong. At low reasoning effort, Claude Opus 5.5, GPT-5.5 and Gemini 3.6 Flash show gaps of +18 to +37 percentage points, still +7 to +29 when two relations with confounded items are excluded. Given ground coordinates and relation definitions, the same models answer 99 to 100% correctly, which the authors say locates the failure in the step from picture to layout.
The authors report that overhead maps drawn by the models mostly preserve depth order but compress depth like the picture itself. GPT-5.5's maps stay close to a copy of the image, while Opus 5.5's and Gemini's undo about 0.6 of that shortening, with Gemini then shrinking all depths uniformly. The three models also lose accuracy on CV-Bench photographs where a reliable 2-D cue misleads, and in two open models the layout information reaches the language model but much of it is lost in their maps.
What it does not show
- The evidence is the paper's abstract only, so method details, full results and peer review status are not available from this source.
- All reported figures come from the paper's authors; no independent confirmation is implied.
- The abstract does not state which two open models were examined or how they were selected.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.