SHRINK edits sink-token keys to steer vision-language model attention
A training-free method from an arXiv paper reduces sink-aligned key components during generation and reports accuracy gains on nine benchmarks.
In plain words
Vision-language models have a few 'sink' tokens that soak up most of the attention. The paper's authors found that changing specific parts of these tokens' keys shifts how much attention image and question content receives. Their method, SHRINK, adjusts these parts during generation without any extra training and reports higher accuracy on the benchmarks they tested.
What was announced
In a paper posted to arXiv, the authors report that in vision-language models a small number of sink tokens attract a disproportionate share of attention. Probing four such models, they found that increasing the magnitude of positive components aligned with the sink-key direction raises attention on average, while increasing negative components lowers it, and that these responses persist through later layers. They present this as evidence that the native sink key can serve as a reference point for controlling attention beyond its own position.
Building on that finding, the team proposes SHRINK, a training-free technique that partially reduces sink-aligned components across visible keys while the model generates output. Under sampled reasoning across nine benchmarks, they report mean accuracy improvements of 5.1 and 2.7 percentage points on LLaVA-OneVision-7B and Qwen2.5-VL-3B, with smaller gains on Qwen2.5-VL-7B and Gemma-3-4B. They also state that rescaling sink-aligned key components can replace the attention operations of two existing attention-steering methods, GIFT and ACG, with comparable accuracy on hallucination benchmarks. The results are self-reported by the authors and have not been independently verified.
What it does not show
- The source is the paper's abstract only; full experimental details, model configurations and evaluation setups are not covered.
- All accuracy figures and comparisons are author-reported and not independently confirmed.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.