← News

ResearchSources: arXiv · 19 Nov 2026

SharQ pairs FP4 quantization with activation sparsity for faster LLM inference

The training-free method decomposes activations into a sparse FP4 backbone and a dense residual, recovering much of the accuracy lost when compressing to FP4 alone, its authors report.

In plain words

Running large language models on cheaper, lower-precision number formats is fast but hurts accuracy, because a few unusually large values dominate the compression. SharQ splits each activation into a sparse, outlier-heavy part and a dense remainder, quantizing both to FP4 while correcting for the errors this introduces. The authors say it needs no calibration data, retraining, or per-model tuning.

What was announced

SharQ, described in an arXiv paper, addresses a known tension: modern accelerators support both low-bit FP4 formats and N:M sparsity, but combining them is difficult because input-dependent outliers dominate FP4 block scales, and naive sparsity masks discard moderate values, coupling sparsification loss with quantization error. The method performs an online sparse-dense decomposition, generating an input-adaptive N:M mask that extracts an outlier-dominated sparse backbone quantized to FP4, then defining a dense residual relative to the quantized backbone rather than the unquantized values. A sparse FP4 GEMM processes the backbone while a dense FP4 GEMM compensates for mask-induced loss and quantization error; both paths share one FP4 weight payload, and a fused kernel handles mask generation, residual construction, and layer normalization.

The authors report evaluations on Llama-3.1-8B, Qwen2.5-7B, Qwen3-30B-A3B, and Qwen3-VL-8B, in which SharQ recovers 43-63% of the NVFP4-to-FP16 accuracy gap across language and vision-language tasks, generalizing across NVFP4, HiF4, and MXFP4 formats. On an RTX 5090, they claim 2.2-2.4x latency reduction over FP16 and 1.2-1.4x throughput improvement over FP8 in language model serving, plus up to 1.58x speedup on Wan2.2-T2V-A14B video generation combined with SageAttention. Code is available on GitHub.

What it does not show

  • Evidence is limited to the paper's abstract; the full methodology, experimental setup, and baselines were not reviewed, and the reported accuracy and speed figures are author claims without independent confirmation.

For security and IT teams

Scope
The reported results cover four specific models (Llama-3.1-8B, Qwen2.5-7B, Qwen3-30B-A3B, Qwen3-VL-8B), three FP4 formats, and an RTX 5090; behavior on other models or hardware is not established in the abstract.

Sources

Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 20 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.