StreamGaze benchmark tests whether video AI models can use human gaze
Researchers introduce StreamGaze, a benchmark measuring how well multimodal large language models interpret human gaze signals in streaming video, and report substantial gaps between state-of-the-art models and human performance.
In plain words
Multimodal large language models are AI systems that process video and other inputs together. The StreamGaze benchmark checks whether such models can use where a person is looking, recorded as gaze data, to understand streaming video and decide when to respond. The authors report that current models still lag well behind humans on these tasks.
What was announced
StreamGaze, described in an arXiv paper, is presented as the first benchmark evaluating how effectively multimodal large language models (MLLMs) use gaze for temporal and proactive reasoning in streaming videos. The authors motivate the work with applications such as augmented reality glasses, where a system must process incoming frames and anticipate user intent. Prior streaming benchmarks, they write, evaluate temporal reasoning but do not measure whether models can interpret or leverage human gaze signals.
The benchmark defines gaze-guided past, present, and proactive tasks. Past and present tasks demand immediate answers from observations available when the query is made, while proactive tasks require models to keep processing incoming frames and respond at the right future moment. To build the dataset, the team created a gaze-video question answering pipeline that aligns egocentric videos with raw gaze trajectories using fixation extraction, region-specific visual prompting, and scanpath construction, producing spatio-temporally grounded QA pairs. Across all tasks, the authors report substantial performance gaps between state-of-the-art MLLMs and human performance, pointing to weaknesses in gaze-based temporal reasoning, intent modeling, and proactive prediction. They also analyze gaze prompting strategies and failure modes, and state that all data and code are publicly available.
What it does not show
- The evidence is the paper's abstract only; the full article was not reviewed, so benchmark construction details, model lists, and exact results cannot be verified from this excerpt.
- Performance gaps between models and humans are reported by the paper's authors and have not been independently confirmed.
For security and IT teams
- Scope
- StreamGaze targets gaze-guided temporal and proactive reasoning in streaming video for multimodal large language models, motivated by applications such as AR glasses.
- Access
- The authors state that all data and code for StreamGaze are publicly available.
Sources
Source-support review completed 13 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 13 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.