← News

Models & toolsSources: arXiv · 19 Nov 2026

SkimPrefill uses a small model to skim long image-text inputs and cut first-token wait

A new arXiv paper proposes letting a small hybrid model read a long multimodal input first, then pruning it down to the most relevant units before the target vision-language model starts.

In plain words

Vision-language models that answer questions about very long documents or large image collections spend most of their startup time reading the whole input. SkimPrefill instead has a small model quickly skim the input and identify the few images or paragraphs that matter, then hands only those to the big model. The authors report large speedups on first-token latency without losing accuracy.

What was announced

The paper's authors observe that when a vision-language model faces inputs of roughly 100K tokens, the prefill stage accounts for most of the wait before the first output token. Existing pruning methods score tokens using the target model itself or a small model from the same family, but that scoring consumes much of the time pruning saves. Their key finding is that the evidence answering a question sits in a few whole units, such as images or paragraphs, and that a single attention head in an early layer of a small hybrid model can locate them, even with images at half resolution.

SkimPrefill therefore runs the small hybrid model, Qwen3.5-0.8B, over the whole input up to that early layer with images at half resolution, then uses the head to rank every unit and passes the top units to the target model as a shorter ordinary prompt. Because most layers of the small model run in linear time, this pass is short, and one untrained selector serves targets from different families. On MMLongBench at 128K tokens across 19 datasets, keeping 10% of input tokens, the authors report prefill speedups of 3.7x for Qwen3.5-9B, 11.6x for Gemma-4-12B and 8.2x for Qwen3-VL-32B while maintaining average accuracy.

What it does not show

  • Evidence is limited to the paper's abstract; full experimental details, baselines and failure cases are not visible in the supplied passages.
  • Reported speedups and accuracy claims come from the authors' own experiments on MMLongBench and have not been independently confirmed.

For security and IT teams

Scope
The reported results cover three target models (Qwen3.5-9B, Gemma-4-12B, Qwen3-VL-32B) on MMLongBench at 128K tokens; behavior on other benchmarks or input lengths is not addressed in the abstract.

Sources

Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 20 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.