SharQ pairs FP4 quantization with activation sparsity for faster LLM inference
The training-free method decomposes activations into a sparse FP4 backbone and a dense residual, recovering much of the accuracy lost when compressing to FP4 alone, its authors report.
In plain words
Running large language models on cheaper, lower-precision number formats is fast but hurts accuracy, because a few unusually large values dominate the compression. SharQ splits each activation into a sparse, outlier-heavy part and a dense remainder, quantizing both to FP4 while correcting for the errors this introduces. The authors say it needs no calibration data, retraining, or per-model tuning.
What was announced
SharQ, described in an arXiv paper, addresses a known tension: modern accelerators support both low-bit FP4 formats and N:M sparsity, but combining them is difficult because input-dependent outliers dominate FP4 block scales, and naive sparsity masks discard moderate values, coupling sparsification loss with quantization error. The method performs an online sparse-dense decomposition, generating an input-adaptive N:M mask that extracts an outlier-dominated sparse backbone quantized to FP4, then defining a dense residual relative to the quantized backbone rather than the unquantized values. A sparse FP4 GEMM processes the backbone while a dense FP4 GEMM compensates for mask-induced loss and quantization error; both paths share one FP4 weight payload, and a fused kernel handles mask generation, residual construction, and layer normalization.
The authors report evaluations on Llama-3.1-8B, Qwen2.5-7B, Qwen3-30B-A3B, and Qwen3-VL-8B, in which SharQ recovers 43-63% of the NVFP4-to-FP16 accuracy gap across language and vision-language tasks, generalizing across NVFP4, HiF4, and MXFP4 formats. On an RTX 5090, they claim 2.2-2.4x latency reduction over FP16 and 1.2-1.4x throughput improvement over FP8 in language model serving, plus up to 1.58x speedup on Wan2.2-T2V-A14B video generation combined with SageAttention. Code is available on GitHub.
What it does not show
- Evidence is limited to the paper's abstract; the full methodology, experimental setup, and baselines were not reviewed, and the reported accuracy and speed figures are author claims without independent confirmation.
For security and IT teams
- Scope
- The reported results cover four specific models (Llama-3.1-8B, Qwen2.5-7B, Qwen3-30B-A3B, Qwen3-VL-8B), three FP4 formats, and an RTX 5090; behavior on other models or hardware is not established in the abstract.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.