RoCoKV compresses reasoning-model KV caches for faster vLLM inference
An arXiv paper describes a KV cache compression method for large reasoning models, pairing uncertainty-guided pruning with redundancy-aware token selection and CUDA kernels integrated into vLLM.
In plain words
Reasoning models generate long chains of intermediate steps, and storing those steps in memory (the KV cache) becomes a major cost. RoCoKV shrinks this cache by cutting tokens the model is confident about and ones that repeat recent content. The authors report faster serving in vLLM with accuracy held up under tight memory budgets.
What was announced
The paper argues that in large reasoning models most serving cost comes from long generated reasoning traces rather than input prompts, and that these traces contain redundant, low-utility tokens from looping, self-verification and corrections. Because such tokens can still attract high attention scores, removing them naively risks harming reasoning. RoCoKV addresses this with two components: uncertainty-guided adaptive pruning, which keeps more memory during uncertain phases and prunes harder when the model is confident, and redundancy-aware token selection, which downweights tokens highly similar to recent reasoning content so more diverse, informative tokens are retained.
Beyond the algorithm, the authors co-designed RoCoKV with the serving stack, building fused three-phase CUDA decode kernels integrated into vLLM as a drop-in plugin that preserves its paged KV layout and CUDA graph capture. In experiments across three reasoning benchmarks and two reasoning models, the paper reports higher accuracy than state-of-the-art KV cache compression methods under constrained cache budgets. For DeepSeek-R1-Distill-Llama-8B in vLLM, it claims a 1.27x speedup and 38% higher throughput on AIME25, and a 1.43x speedup and 45% higher throughput on a long synthetic benchmark, compared with Full-KV.
What it does not show
- Evidence is limited to the paper's abstract; experimental details, baselines and evaluation conditions cannot be verified from the excerpt.
- Reported speedups and accuracy gains are the authors' own claims and have not been independently confirmed.
- The abstract does not specify which two reasoning models or three benchmarks were used beyond AIME25 and a long synthetic benchmark.
For security and IT teams
- Scope
- The method targets large reasoning models with long generated traces and is integrated into vLLM as a drop-in plugin preserving its paged KV layout and CUDA graph capture.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.