← News

ResearchSources: arXiv · 19 Nov 2026

NarrowSpec: draft models can run at half the hidden width in speculative decoding

The paper's authors report that a draft model's residual stream can be narrowed to a quarter of the target's width while largely preserving token acceptance, speeding end-to-end decoding on two GPUs.

In plain words

Speculative decoding uses a small draft model to propose text that a larger target model checks, and draft models usually copy the target's internal width. The authors show the draft can instead run at half or a quarter of that width with little loss in acceptance. In their tests this made generation faster end to end.

What was announced

Writing on arXiv, the NarrowSpec authors argue that draft models reading the target's hidden states, such as DFlash and EAGLE-3, need not inherit the target's hidden width. For DFlash, which reuses the target's vocabulary projection, they report that the trained output adapter folds into that projection, producing one narrower projection whose logits are unchanged up to floating-point rounding.

The reported evaluation covers two drafter families, DFlash and EAGLE-3, on two targets, Qwen3-8B (NVFP4) and GPT-OSS-20B (MXFP4), served with SGLang on a B200 and a DGX Spark (GB10). At batch size 1 and temperature 0, the authors say narrowing the draft to half or a quarter width sped up end-to-end decoding by 1.06-1.12x on B200 and 1.13-1.44x on GB10. They add that the speedup holds at larger batch sizes: from batch 32 upward, narrowing reached up to 1.21x on both GPUs, and no narrow draft was slower than full width.

What it does not show

  • The source is the paper's abstract only; full experimental setup, baselines and peer review are not visible, and the reported speedups are the authors' own measurements, not independently confirmed.

Sources

Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 20 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.