Nunchi benchmark adds conversational cues to gaze tracking in group scenes
A new arXiv paper introduces SORBET, a gaze benchmark with speech and transcripts, and Nunchi, a framework that lets a vision-language model weigh what is said against where people look.
In plain words
When several people talk, figuring out who is looking at whom from camera geometry alone gets harder as the group grows. The researchers built SORBET, a test set pairing gaze labels with audio, transcripts and speaker information, and Nunchi, a method that feeds these cues to a vision-language model. Nunchi beat the strongest prior method by 4.8 percentage points on their benchmark.
What was announced
The authors argue that gaze geometry alone becomes ambiguous in multi-party scenes, and that conversations supply cues geometry lacks, such as who is speaking and what is said. They say no existing gaze dataset provides such cues for many multi-party scenes, so they built SORBET with 3,340 samples from 1,578 clips, each with synchronized speech, transcripts and attended-person labels.
Their Nunchi framework builds a text prompt from the transcript plus outputs of gaze and speaker models, links each cue to candidates through shared labels, and lets a vision-language model weigh geometry against conversational signals. Nunchi reportedly outperforms the strongest existing method by 4.8 percentage points, and the authors report gains across three VLMs, with automatic transcripts, and on two existing datasets. On SORBET, conversational cues helped while additional visual cues did not.
What it does not show
- The evidence is the paper's own abstract; results are author-reported and not independently confirmed.
- The abstract does not describe how samples were collected, class balance, or error cases, so dataset quality cannot be assessed from this source.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.