{"sources":[{"id":"S1","url":"https://huggingface.co/blog/kv-cache","title":"KV Cache from scratch in nanoVLM","publisher":"Hugging Face","published_at":"2025-06-04","accessed_at":"2026-11-14","version":"Live article and saved original revision 237e72b9c002317d6ed18ff426d397ff","document_id":"0476d36f217f03bbd39a9a0829660254","revision_id":"237e72b9c002317d6ed18ff426d397ff","limitations":"Generation tutorial; its displayed loop includes an additional final forward relative to this notebook's stop boundary. No speedup or source-code correctness claim is adopted."},{"id":"S2","url":"https://huggingface.co/docs/transformers/main/cache_explanation","title":"How caching works","publisher":"Hugging Face Transformers","published_at":null,"accessed_at":"2026-11-14","version":"main; no pinned commit","document_id":"","revision_id":"","limitations":"Moving documentation used for logical attention dependencies, per-layer cache semantics and forward-selection ordering. Not an API compatibility guarantee."},{"id":"S3","url":"https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/","title":"Mastering LLM Techniques: Inference Optimization","publisher":"NVIDIA Technical Blog","published_at":"2023-11-17","accessed_at":"2026-11-14","version":"Mutable live article; displayed original publication date","document_id":"","revision_id":"","limitations":"Original article body used, not its AI-generated summary. Payload formula is specialized to stored KV heads; no benchmark or hardware generalization is adopted."},{"id":"S4","url":"https://arxiv.org/abs/2305.13245","title":"GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints","publisher":"arXiv; Joshua Ainslie and collaborators","published_at":"2023-12-23","accessed_at":"2026-11-14","version":"2305.13245v3; first submitted 2023-05-22","document_id":"","revision_id":"","limitations":"Abstract read and used only for the distinction between query-head and KV-head counts. No experimental results or full-paper details are adopted."}],"claims":[{"id":"C1","text":"Prompt prefill builds initial cached state and supplies the prediction used for the first output selection; subsequent forwards consume new selected tokens.","location":"scope; trace; questions Q1","evidence":[{"source_id":"S1","passage":"Encode the full prompt and build the initial cache.","location":"Prefill vs Decode in the Generation Loop, prefill bullet; live lines 234–251 and saved section 3"}],"limitations":"Ordinary autoregressive causal decoder generation; no performance claim.","kind":"source","basis":"Read saved section 3 and live prompt-forward/sample ordering at lines 246–263."},{"id":"C2","text":"Causal reuse permits per-layer caching of past K/V; current attention uses the current query with past and current K/V. The seven diagram edges show those dependencies, without caching past queries.","location":"attention prose, figure, caption and alternative; decode-attention.drawio; questions Q2","evidence":[{"source_id":"S2","passage":"the mask prevents the model from attending to future tokens","location":"Attention matrices, line 137"},{"source_id":"S2","passage":"caching is done on a per-layer basis","location":"Attention matrices, line 143"}],"limitations":"Fixed causal model and full-history cache. Diagram omits other layer computations and physical allocation mechanics.","kind":"source","basis":"Read attention expression and concatenation at lines 138–142, then cache.update and attention call at lines 157–158. XML edges x→q, x→kv, kv→all, past→all, q→attn, all→attn and attn→out encode these relationships."},{"id":"C3","text":"Model weights and cached activations are distinct memory categories; dense KV payload counts K and V across layers, batch, cached positions, stored heads, head dimensions and element bytes.","location":"attention final paragraph; memory formula","evidence":[{"source_id":"S3","passage":"model weights and the KV cache","location":"LLM memory requirement, line 63"},{"source_id":"S3","passage":"The first factor of 2 accounts for the K and V matrices.","location":"LLM memory requirement, line 70"}],"limitations":"Tensor payload only; stored-head specialization uses C6. Not an allocation or admission estimate.","kind":"source","basis":"Read per-token and total-size equations at lines 69–73. Multiply the tensor dimensions with equal K/V head dimensions."},{"id":"C4","text":"Selection does not itself process a token into the cache. nanoVLM's displayed loop performs a forward after each selection, while the notebook stops after its last selection.","location":"trace; questions Q1","evidence":[{"source_id":"S1","passage":"next_token = sample_from(prompt_output)","location":"Prefill vs Decode generation code, line 255, followed by forward at lines 257–261"},{"source_id":"S2","passage":"leaving unprocessed tokens","location":"Generation example comment, line 201"}],"limitations":"Interpretation of the inspected ordering; final cached length depends on stopping boundary.","kind":"source","basis":"S1 forward occurs after sampling inside each iteration; S2 lines 196–205 forward first, select, then prepare new input."},{"id":"C5","text":"Synthetic prompt P0–P3 and selections A/B/C yield cached lengths 4/5/6, leaving C unprocessed. For G>=1 this loop has G forwards and P+G−1 cached positions.","location":"trace prose and table; lab trace and assertions; questions Q1 and Q3","evidence":[],"limitations":"Invented labels and prescribed outputs, not actual sampling or tensor cache contents.","kind":"original_example","basis":"lab.py trace extends cached with current_input before snapshotting, then assigns selected as the next input. Inputs have lengths 4,1,1. Final tuple is prompt+(A,B); a further forward of C would yield seven positions."},{"id":"C6","text":"GQA can store fewer KV heads than query heads; memory counting must use the number of stored KV heads.","location":"memory formula; questions Q4","evidence":[{"source_id":"S4","passage":"more than one, less than number of query heads","location":"Abstract, definition of grouped-query attention"}],"limitations":"Only the head-count distinction is used. No quality or speed claim.","kind":"source","basis":"Abstract definition plus counting the dimensions of stored K/V rather than queries."},{"id":"C7","text":"The synthetic dimensions require 49,152 bytes per position and 96 MiB at 2,048 positions; trace lengths 4/5/6 use 192/240/288 KiB. Seven positions use 336 KiB. Batch three gives 288 MiB and 32 stored heads gives 384 MiB at length 2,048.","location":"memory arithmetic; lab arithmetic assertions; questions Q4","evidence":[{"source_id":"S3","passage":"precision_in_bytes","location":"Per-token KV-size equation, line 69"}],"limitations":"Original arithmetic using synthetic uniform uncompressed dimensions. Not measured memory.","kind":"original_example","basis":"Premise C3 formula: 2*24*1*8*64*2=49152=48*1024. Multiplying by 2048 yields 100663296=96*1048576. Multiplying 48 KiB by 4,5,6,7 gives 192,240,288,336. Scaling batch by three or KV heads by four yields 288 or 384 MiB."},{"id":"C8","text":"The notebook's declared scope, snapshot convention and memory exclusions bound its conclusions; selected positions, processed positions and allocated capacity are deliberately distinguished.","location":"scope; attention alternative; memory limitations; lab limitations; questions Q5","evidence":[],"limitations":"Teaching assumptions and exclusions, not descriptions of all serving systems.","kind":"original_example","basis":"Inspectable scope paragraphs, diagram nodes and lab.py inputs. The calculation has no weight, temporary-storage, padding, allocator, compression or sharing terms."},{"id":"C9","text":"The original illustrative lab performs label bookkeeping and integer payload arithmetic, validates its stated input dimensions and specifies expected outcomes without implementing attention or allocating KV tensors.","location":"lab section; entire lab.py including comments and assertions; questions Q5","evidence":[],"limitations":"No execution report for this revised lab; no numerical attention, masks, positions, backend or performance validation is claimed.","kind":"original_example","basis":"Full lab.py inspection: math.prod is the only import; trace stores label tuples; payload_bytes multiplies integers; main contains assertions and print calls. No model, network or file operations."},{"id":"C10","text":"The disclosure accurately identifies the public original readings and limited evidence method; source reading is not independent expert verification.","location":"references source-check disclosure","evidence":[],"limitations":"Moving web sources; GQA abstract-only scope. No completed rendering, execution or separate acceptance review is asserted.","kind":"process","basis":"This turn's ResearchNode nanoVLM search and read_document calls returned document 0476d36f217f03bbd39a9a0829660254 with revision 237e72b9c002317d6ed18ff426d397ff; the offset-5000 read includes the full generation section. The refined_work nanoVLM query returned no items. Live web searches and opens read S1–S4, including focused S1 generation, S2 cache/generation and S3 original memory passages. draw.io XML guidance was read and create_diagram called with the supplied topology."}]}