G-MTP drafting head speeds up LLM inference on servers and phones
A new arXiv paper describes a multi-token prediction architecture that turns existing LLMs into faster drafters for speculative decoding, with reported speedups on H100 GPUs and smartphone hardware.
In plain words
Large language models can be made faster with speculative decoding, where a small helper model quickly drafts text and the main model checks it in parallel. The paper introduces G-MTP, a drafting component built on multi-token prediction that attaches to existing LLMs. Its authors report it speeds up generation on servers and on phones, though the results come only from their own paper.
What was announced
The paper, posted to arXiv, targets speculative decoding, a technique that splits LLM generation into a drafting phase and a parallel verification phase. The authors argue the central design problem is building a drafter that is lightweight yet accurate enough to approximate the target model's predictions. Their answer is a highly optimized multi-token prediction architecture that extends existing backbone LLMs, built around a single shared drafting head.
Three optimizations are described: injecting target-model activations to anchor semantic planning, zero-copy KV-cache cross-attention to skip prefill compute and cut memory overhead, and a clustered vocabulary projection to lower the cost of full-vocabulary decoding. Tested with the Gemma 4 model family, the authors report speedups up to 3.1x for large server setups, up to 2.3x on H100 hardware, and up to 2.2x on a smartphone GPU for memory-constrained on-device use. They also present parameter-efficient fine-tuning recipes intended to keep the drafter aligned with the backbone under domain shift, requiring up to 1.83x fewer target forward passes than tuning only the backbone.
What it does not show
- The evidence is limited to the paper's abstract; the reported speedups and fine-tuning results are the authors' own claims and have not been independently confirmed.
- The abstract does not specify which Gemma 4 model sizes, workloads, or baselines underlie the reported speedup figures.
For security and IT teams
- Scope
- The reported results cover the Gemma 4 model family, spanning large server-side setups and smaller models aimed at memory-constrained on-device environments including smartphone GPUs.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.