NarrowSpec: draft models can run at half the hidden width in speculative decoding
The paper's authors report that a draft model's residual stream can be narrowed to a quarter of the target's width while largely preserving token acceptance, speeding end-to-end decoding on two GPUs.
In plain words
Speculative decoding uses a small draft model to propose text that a larger target model checks, and draft models usually copy the target's internal width. The authors show the draft can instead run at half or a quarter of that width with little loss in acceptance. In their tests this made generation faster end to end.
What was announced
Writing on arXiv, the NarrowSpec authors argue that draft models reading the target's hidden states, such as DFlash and EAGLE-3, need not inherit the target's hidden width. For DFlash, which reuses the target's vocabulary projection, they report that the trained output adapter folds into that projection, producing one narrower projection whose logits are unchanged up to floating-point rounding.
The reported evaluation covers two drafter families, DFlash and EAGLE-3, on two targets, Qwen3-8B (NVFP4) and GPT-OSS-20B (MXFP4), served with SGLang on a B200 and a DGX Spark (GB10). At batch size 1 and temperature 0, the authors say narrowing the draft to half or a quarter width sped up end-to-end decoding by 1.06-1.12x on B200 and 1.13-1.44x on GB10. They add that the speedup holds at larger batch sizes: from batch 32 upward, narrowing reached up to 1.21x on both GPUs, and no narrow draft was slower than full width.
What it does not show
- The source is the paper's abstract only; full experimental setup, baselines and peer review are not visible, and the reported speedups are the authors' own measurements, not independently confirmed.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.