KV-LTM proposal stores session KV caches as assistant memory
An arXiv paper describes KV-LTM, a memory design that keeps each conversation's raw text plus its KV cache under a fixed reader model, so questions are answered by restoring cached sessions rather than re-deriving evidence.
In plain words
Large language model assistants can forget details from months of past chats. KV-LTM is a proposed design that saves each session's internal state, called a KV cache, alongside the raw text, letting the model resume a session without recomputing it. The authors position it as an accelerator that complements, not replaces, retrieval and indexing.
What was announced
The paper argues that current long-term memory stacks insert derivation or selection steps between a question and its evidence. On LongMemEval, two commercial assistants scored well below the same model reading the history offline, which the authors say is consistent with, but does not isolate, content loss between writing and reading memories. KV-LTM instead stores each session's raw text plus its KV cache under a fixed reader model, answering queries by restoring cached sessions and appending the question. Text remains the memory of record, while the cache acts as a recompute-free prefill accelerator that degrades gracefully to text.
The authors size the design analytically rather than through a deployed system. Across pinned BF16 full-attention (GQA) and latent-cached MLA configurations, per-token state spans roughly 31K-328K bytes, implying about 0.34-3.6 TB per user-year at a declared 30K-token-per-day scenario, against roughly 49 MB of raw text; a pinned KDA+MLA hybrid differs at about 8.1K bytes per token plus a fixed per-session recurrent term. The evaluation protocol needs no persistent-cache implementation for logical QA: per-session prefill emulates cache reuse, and a separate minimal real-cache replay tests numerical equivalence and measures cost, with outcomes not assumed.
What it does not show
- The evidence is the paper's abstract only; the full article's details are not covered.
- The storage figures and per-query cost model are analytical estimates under declared scenarios, not measurements from a deployed system.
- The comparison with commercial assistants is a system-level observation the authors say does not isolate content loss as the cause.
- The evaluation protocol's real-cache replay has outcomes not assumed, so no end-to-end results are reported in the abstract.
For security and IT teams
- Scope
- KV-LTM is described as complementary to retrieval and indexing rather than a replacement, with text as the memory of record and the cache as an accelerator within a declared retention budget.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.