SpecMem keeps tool-call memory live to speed agentic decoding
The arXiv paper reports that continuously updating per-user draft stores sustains speculative-decoding acceptance where frozen stores fade, with up to 1.74x wall-clock speedup.
In plain words
AI agents that use tools generate each tool call token by token, which is slow. SpecMem speeds this up by reusing verified past calls as drafts, and it keeps updating those drafts as users work. In the paper's tests, a store that is never updated loses effectiveness over time, while the live store keeps its speed advantage.
What was announced
Speculative decoding speeds up generation by drafting tokens and verifying them without changing outputs, and the authors note that retrieved past calls make strong drafts because user behavior tends to repeat. They argue existing systems either keep no cross-session memory or build a draft store once and never refresh it. SpecMem addresses this by writing verified calls back online into bounded per-user stores with LRU eviction, while a side-effect gate holds back irreversible calls until verification confirms them.
In tests on multi-session, multi-user query streams served by three open-weight models, the paper reports that a frozen store steadily loses acceptance while SpecMem holds it, with a 27% gap in mean accepted tokens, gains of +11% to +34% across benchmarks, and up to 1.74x wall-clock speedup. Ablations attribute the gain almost entirely to keeping the store live. Per-user scoping added no acceptance but closed a measured cross-user leakage channel, and gating on declared side effects, rather than retrieval confidence, is what eliminated irreversible mistakes.
What it does not show
- Evidence is limited to the paper's abstract; full experimental setup, baselines and caveats are not available in the supplied passages.
- Results come from the authors' own benchmarks with three open-weight models and have no independent confirmation.
- The abstract does not state whether code or a deployed system is publicly available.
For security and IT teams
- Scope
- The reported evaluation covers multi-session, multi-user query streams served by three open-weight models, per the paper's abstract.
Sources
Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 17 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.