← News

ResearchSources: arXiv · 16 Nov 2026

Researchers propose regression-free layout matching to curb coordinate hallucinations in GUI grounding

A paper on arXiv describes a two-model pipeline that separates instruction understanding from element localization, aiming to reduce hallucinated coordinates without fine-tuning.

In plain words

Software agents that control a computer screen must turn a user's request into exact click coordinates. The paper reports that end-to-end multimodal models often guess wrong positions, a problem the authors call coordinate hallucination. Their fix splits the job: one model describes the screen, and a second matches that description against layout candidates instead of predicting coordinates directly.

What was announced

The authors, writing in an abstract for the arXiv preprint, frame the problem as a dual obstacle: conventional grounding models lack the semantic depth to interpret abstract instructions, while end-to-end multimodal large language models produce inaccurate coordinates because of weak fine-grained perception. Their framework keeps a frozen MLLM for instruction parsing, which expands a user request into a structured visual description containing layout cues. A dedicated Layout-Aware GUI Grounding Model then localizes the element by matching against layout-prior candidates, with no coordinate regression learned at any stage. The grounding model is trained only on binary text-versus-icon labels, which the authors say avoids expensive fine-tuning.

On the ScreenSpot-Pro benchmark, the paper claims more than a 20% improvement in grounding accuracy over end-to-end systems, and on Mind2Web it reports gains of more than 15% in success rate and element selection rate. The authors attribute these results to decoupling instruction understanding from layout-aware localization. The figures come solely from the paper's abstract; the full study, its experimental setup, and any independent replication are not covered by the supplied excerpt.

What it does not show

  • Evidence is limited to the paper's abstract; benchmark numbers are author claims from the preprint and are not independently confirmed.

Sources

Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 17 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.