← News

ResearchSources: arXiv · 20 Nov 2026

HiSA sparsifies attention along the hidden dimension, cutting query-key work to about a third

Researchers propose Hidden Sparse Attention, which computes only part of each query-key dot product using a fixed rule based on token positions, reporting better accuracy and faster decoding for LLMs.

In plain words

Large language models normally compare every query against every key using a full dot product, a calculation over many feature dimensions. HiSA instead computes only a portion of that product, choosing which parts to skip with a fixed rule tied to token positions. The authors say models trained this way perform slightly better while doing roughly a third of the multiplications.

What was announced

The method, described in an arXiv paper, keeps the query and key projections unchanged but makes attention sparse along the hidden, or feature, dimension rather than across tokens, which the authors note is an axis most sparse attention methods leave aside. Building on the structure of rotary position embeddings (RoPE), they decompose keys and queries so that certain parts can be omitted from the attention computation under a static, parameter-free rule.

Training LLMs from scratch with HiSA, the authors report that the partial dot product improved rather than hurt performance relative to the complete product, while computing only about a third of the query-key products. They also state that inference decoding speeds up and the memory footprint shrinks, that HiSA has been extended to FlashAttention-3 kernels for training and inference, and that dense attention appears to learn to suppress the score parts HiSA drops. Code is available on GitHub.

What it does not show

  • The evidence is the paper's abstract only; the reported performance, speed and memory results have not been independently verified and the full paper was not reviewed.
  • The abstract does not specify which model sizes, benchmarks or baselines were used, so the generality of the reported gains is unclear.

For security and IT teams

Status
HiSA is a research paper with code published on GitHub; the abstract does not state any product integration or deployment availability.

Sources

Source-support review completed 21 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 21 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.