Sampler overhead, not experts, slows quantised MoE diffusion decoding
Researchers profiled a 26B diffusion Mixture-of-Experts model on an edge accelerator and found the per-token sampling pipeline dominates decode time once weights are quantised.
In plain words
Masked diffusion language models generate text by repeatedly cleaning up a whole block of tokens, and each pass must sample from a huge table of scores. The authors found that after shrinking the model weights, this sampling step, not the model's expert math, takes the most time. They rebuilt it to run faster without losing accuracy.
What was announced
The team behind the arXiv paper profiled single-user decoding of a 26B-parameter diffusion Mixture-of-Experts model running on an NVIDIA GB10 accelerator, which they describe as bandwidth-bound. Their measurement: once weights are quantised, the sampling and stopping pipeline over the full canvas-by-vocabulary score tensor becomes the largest single share of decode time, ahead of the expert matrix multiplications. Diffusion decoding touches many more score entries per output token than autoregressive decoding, with the multiplier equal to the number of denoising steps.
As a fix, the authors propose two lossless tiers: one bit-exact, producing byte-identical output to the reference decoder, and one matching accuracy using a fused single-pass entropy kernel with Gumbel-max sampling. Paired with a W4A16 expert kernel, the stack reportedly reaches 2.4-2.7x end-to-end speedup over bf16 at unchanged accuracy on GSM8K, HumanEval, and MATH, on a single 273 GB/s device. A second, independently built diffusion MoE, LLaDA2.0-mini, showed the pipeline at 28.8% of decode and a 3.3x fused speedup at accuracy parity, while a dense diffusion model showed only 3.6%, suggesting the cost is specific to MoE diffusion. Code and quantised weights are released.
What it does not show
- Evidence is limited to the paper's abstract; full methodology, hardware details, and benchmark configurations are not available for verification.
- The reported speedups and accuracy results are the authors' own measurements and have not been independently confirmed.
- The dense-model comparison appears only for LLaDA2.0-mini and one dense model, so the MoE-specific claim rests on few data points.
For security and IT teams
- Scope
- Findings apply to single-user decoding of MoE diffusion models on a bandwidth-bound edge accelerator (NVIDIA GB10, 273 GB/s), not to dense diffusion models or other hardware.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.