LoopCache speeds long-context decoding for looped Transformers
The arXiv paper reports a self-speculative decoding framework that cuts KV cache memory traffic and updates drafts dynamically, claiming double-digit speedups over prior wave-style methods.
In plain words
Looped Transformers reuse one block of weights many times to build deep models cheaply, and each pass keeps its own key-value cache, the stored intermediate data used during text generation. LoopCache is a new way to run this process faster by refreshing draft predictions as they move through the loops and by rotating caches so only one loop's data is read per step. The authors report large speedups over two earlier methods.
What was announced
Looped Transformers gain effective depth by applying a single weight-shared block across R recurrent steps, with each step keeping a loop-specific key-value (KV) cache, the paper's abstract explains. Earlier wave-style self-speculative approaches such as WaveFront Decoding (WFD) and LoopSpec pipeline draft and target tokens across loops in one step, but the authors identify two drawbacks: drafting happens only at fixed loops and ignores intermediate prediction changes, and batching tokens from different loops can load up to R times the KV cache per step, which amplifies memory traffic in long-context settings.
LoopCache addresses both issues, according to the abstract. It uses confidence-guided dynamic draft updating, refining in-flight drafts across intermediate loops based on the correlation between prediction switches and output confidence. A cyclic loop KV cache strategy rotates through loop-specific caches, restricting per-step KV traffic to a single loop while batching tokens at the nearest matching cycle for what the authors describe as lossless verification. Selectively retaining independent caches for a few loop-sensitive shallow layers is said to improve acceptance rates with negligible overhead. In experiments, LoopCache achieved average speedups of 10.25x over WFD and 14.56x over LoopSpec.
What it does not show
- The evidence is the paper's abstract only; experimental setup, model details, and full results are not available in the supplied passages, and the speedup figures are the authors' own claims without independent confirmation.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.