← News

ResearchSources: arXiv · 19 Nov 2026

SANTA++ samples attention keys to cut KV cache reads

A training-free method from a new arXiv paper estimates full attention by scoring representative keys, with reported accuracy and speed results on Qwen2.5-7B-Instruct.

In plain words

Large language models usually read every stored token when answering a question, which gets expensive at long context lengths. SANTA++ instead groups stored keys, scores one representative per group, and samples only promising groups, then statistically corrects the result to approximate full attention. The authors report it keeps most accuracy while reading far fewer cached entries.

What was announced

The paper's abstract describes attention as often concentrating on a small subset of context tokens, with the relevant subset changing per query. SANTA++ organizes cached keys into teams, has the query score one representative per team, and samples teams accordingly. Exact attention scores are computed within sampled teams, and each team's contribution is reweighted by the inverse of its inclusion probability, an importance-sampling correction meant to estimate attention over the full cache. The method is training-free, and a sampling budget trades memory reads for accuracy.

The reported measurements come from the authors' own experiments. With Qwen2.5-7B-Instruct at 32K context, contiguous SANTA++ at S=1024 retained 95-98% of dense aggregate scores on LongBench v2, HELMET RAG, and RULER while using 28-31% logical KV access. On RULER, the authors state it outperformed Quest across the tested accuracy-access sweep, using 48.7% fewer KV reads at roughly 79% accuracy, and also beat MagicPIG at low access. Their Triton implementation delivered a 1.38x speedup for the attention operator over PyTorch's Flash SDPA backend in FP16, and 1.23x faster full-model decoding at batch eight on an RTX 5090 Laptop GPU with INT4 weights and FP16 KV cache. Kernels are available on GitHub.

What it does not show

  • Evidence is limited to the paper's abstract (v2), so the reported accuracy and speed figures are author claims that have not been independently verified.
  • Benchmarks cover specific settings, including Qwen2.5-7B-Instruct at 32K context and one consumer GPU configuration; results may not generalize to other models, contexts, or hardware.

For security and IT teams

Access
The authors' Triton kernels are available at a public GitHub repository, per the abstract.

Sources

Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 20 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.