← Learn / Notebooks

Models & toolsTechnicalTechnical notebook

Transformer architecture and attention

Explore attention, GQA, RoPE, KV-cache memory and FlashAttention, with mathematical explanations and a verification exercise.

Before you start: Python, linear algebra and familiarity with neural network training and inference.

Imported AI-assisted study notebook · Added 13 Nov 2026. Source links are retained; the material has not passed ResearchNode’s article source-support review.

A rigorous engineering breakdown of autoregressive Transformer internals—from variance-stabilized self-attention and Grouped-Query Attention (GQA) memory budgets to Rotary Position Embeddings (RoPE), Pre-RMSNorm/SwiGLU blocks, and SRAM-tiled online softmax in FlashAttention.

In [1]:
Markdown

1. Scaled Dot-Product Attention & Variance Stabilization

Introduced by Vaswani et al. (2017) [1], the Transformer replaces recurrent and convolutional sequence processing with Scaled Dot-Product Attention. Given projected Query (Q ∈ R^{N × d_k}), Key (K ∈ R^{N × d_k}), and Value (V ∈ R^{N × d_v}) matrices for sequence length N, attention computes a content-addressable weighted sum:

Attention(Q, K, V) = softmax( (Q K^T) / sqrt(d_k) + M ) V

Where M_{ij} = 0 if i >= j, and M_{ij} = -∞ if i < j (causal autoregressive mask)

Scaling Attention Logits by sqrt(d_k)

Suppose the scalar components of query vector q ∈ R^{d_k} and key vector k ∈ R^{d_k} are independent random variables with mean 0 and variance 1. The unscaled dot product q · k = ∑_{i=1}^{d_k} q_i k_i has mean 0, but its variance grows linearly with head dimension d_k [1]:

Var(q · k) = ∑_{i=1}^{d_k} Var(q_i) · Var(k_i) = ∑_{i=1}^{d_k} (1 · 1) = d_k

Scaling by 1 / sqrt(d_k) restores unit variance:
Var( (q · k) / sqrt(d_k) ) = (1 / d_k) · Var(q · k) = 1

Without the 1 / sqrt(d_k) factor, large head dimensions (such as d_k = 128, where standard deviation reaches sqrt(128) ≈ 11.31) push pre-softmax logits to extreme magnitudes. Because the Jacobian of softmax satisfies ∂p_i / ∂z_j = p_i (δ_{ij} - p_j), saturated one-hot probabilities (p_i → 1) drive gradients toward zero, stalling optimization.

In [2]:
Diagram

2. Attention Head Topologies (MHA, GQA, MQA) & KV-Cache Bottlenecks

During autoregressive inference, generating token t requires keys and values from all prior tokens 1..t-1. To avoid recomputing historical projections at every step, inference engines store past Key and Value tensors in GPU High Bandwidth Memory (HBM) as the KV-Cache.

In standard Multi-Head Attention (MHA) [1], every query head has a dedicated Key and Value head (H_q = H_{kv}), causing the KV-Cache to dominate GPU memory at long context lengths. Multi-Query Attention (MQA) (Shazeer, 2019) [2] shares a single KV head across all query heads (H_{kv} = 1), maximizing memory bandwidth savings at the cost of slight quality degradation. Grouped-Query Attention (GQA) (Ainslie et al., 2023) [3] interpolates between MHA and MQA by partitioning H_q query heads into G groups (H_{kv} = G), each sharing one Key and one Value head.

Attention heads: key/value sharing
Illustrative eight-query-head example. Holding layers, batch size, sequence length, head dimension and element size constant, KV tensor storage scales with the number of KV heads: 1×, 0.25× and 0.125× here. This does not establish total model memory, latency or equal quality. Source 1 · Source 2
Diagram as text and editable source
  • MHA: 8 query heads → 8 key/value headsOne pair per query head: 1:1
  • GQA: 8 query heads2 groups of 4 → 2 key/value headsOne pair per group: 4:1
  • MQA: 8 query heads → 1 key/value headOne pair shared by all: 8:1
Download editable draw.io diagram
Exact KV-Cache Memory Footprint (Bytes):
Bytes_KV = 2 (for K & V) × L (layers) × H_kv (KV heads) × d_head × S (seq_len) × B (batch) × b (bytes/elem)

Example (L = 32, H_q = 32, d_head = 128, S = 32,768 tokens, B = 1, BF16 b = 2 bytes):
• MHA (H_kv = 32): 2 × 32 × 32 × 128 × 32,768 × 2 = 17,179,869,184 bytes = 16.00 GiB
• GQA (H_kv = 8): 2 × 32 × 8 × 128 × 32,768 × 2 = 4,294,967,296 bytes = 4.00 GiB
• MQA (H_kv = 1): 2 × 32 × 1 × 128 × 32,768 × 2 = 536,870,912 bytes = 0.50 GiB
In [3]:
Markdown

3. Modern Decoder Block: RoPE, Pre-RMSNorm & SwiGLU

While the original 2017 Transformer used Post-LayerNorm, sinusoidal additive positional embeddings, and ReLU feed-forward networks [1], contemporary open and frontier decoder-only LLMs converge on three standardized architectural upgrades:

A. Rotary Position Embeddings (RoPE — Su et al., 2021)

Instead of adding positional vectors to token embeddings at the input layer, RoPE [4] encodes position directly inside the attention dot product by rotating pairs of dimensions (x_{2i-1}, x_{2i}) in the complex plane by angle m · θ_i, where m is the token index and θ_i = θ_base^{-2(i-1)/d}:

[ x'_{2i-1} ] [ cos(m·θ_i) -sin(m·θ_i) ] [ x_{2i-1} ] [ x'_{2i} ] = [ sin(m·θ_i) cos(m·θ_i) ] [ x_{2i} ]

Relative Position Property (Orthogonality of Rotation Matrices):
<R_m q, R_n k> = q^T R_m^T R_n k = q^T R_{n - m} k

Because the inner product depends strictly on the relative distance (n - m), RoPE preserves relative positional awareness across arbitrary offsets, remains fully compatible with linear/KV-cached attention, and supports long-context extension via base-frequency scaling (ABF / YaRN).

B. Pre-RMSNorm & SwiGLU Feed-Forward Networks

Component Original Transformer (2017) Modern LLM Standard Mathematical Form & Engineering Benefit
Normalization Placement & Operator Post-LayerNorm:
LayerNorm(x + Sublayer(x))
Pre-RMSNorm [7]:
x + Sublayer(RMSNorm(x))
RMSNorm(x) = (x / sqrt(mean(x^2) + ε)) ⊙ γ
Preserves an unimpeded identity residual gradient highway and eliminates mean-centering overhead (~10–40% faster norm kernels).
Position-Wise FFN 2-Matrix ReLU FFN:
max(0, xW_1 + b_1)W_2 + b_2
3-Matrix SwiGLU [8]:
(Swish_1(xW_gate) ⊙ xW_up) W_down
Swish_1(z) = z · σ(z)
Multiplicative gating improves perplexity at equal parameter/FLOP budgets (hidden width typically scaled by (2/3) · 4d to hold parameter count constant).
In [4]:
Markdown

4. IO-Aware Exact Attention: FlashAttention & Online Softmax

Dao et al. (2022) [5] identified that standard attention is memory-bandwidth bound, not compute bound. Materializing the N × N attention score matrix S = QK^T / sqrt(d_k) and probability matrix P = softmax(S) requires reading and writing Θ(N^2) elements to slow GPU High Bandwidth Memory (HBM), even though on-chip SRAM is over an order of magnitude faster (~19 TB/s SRAM vs. ~1.5–3.3 TB/s HBM on A100/H100 GPUs).

FlashAttention [5] and FlashAttention-2 [6] compute exact attention (zero approximation) without ever writing S or P to HBM by combining two techniques:

  1. SRAM Block Tiling: Queries are partitioned into blocks of B_r rows and Keys/Values into blocks of B_c columns sized to fit inside on-chip SRAM (B_c = ceil(M / (4d)), where M is SRAM capacity).
  2. Online Softmax Rescaling: Softmax normally requires three passes over a row (find row max m, compute exponentials and denominator sum ℓ, divide by ℓ). FlashAttention updates running row statistics block-by-block and rescales the partial output accumulator Õ whenever a new block introduces a larger maximum m_new:
For each Key/Value tile j:
S^{(j)} = (Q_i K_j^T) / sqrt(d)
m_new = max( m_old, rowmax(S^{(j)}) )
P_tilde = exp( S^{(j)} - m_new )
ℓ_new = exp(m_old - m_new) · ℓ_old + rowsum(P_tilde)
O_tilde = diag(exp(m_old - m_new)) · O_tilde + P_tilde · V_j

After all tiles finish (FlashAttention-2 deferred normalization):
O_i = diag(ℓ_final)^{-1} · O_tilde
IO Complexity & Backward Pass Recomputation
Standard attention incurs Θ(N d + N^2) HBM accesses. FlashAttention reduces HBM accesses to Θ(N^2 d^2 / M)—for typical d = 64–128 and SRAM M = 192–228 KB, yielding up to 9× fewer HBM reads/writes and 2–4× wall-clock speedup [5], [6]. During backpropagation, instead of storing the N × N matrix P from the forward pass, FlashAttention stores only the output O and row-wise logsumexp vector L = m + log(ℓ) (O(N) memory) and recomputes S and P on-chip in SRAM.
In [5]:
Python

5. Verification Lab: KV-Cache Scaling, RoPE Invariance & Tiled Online Softmax

The self-contained NumPy implementation below verifies three foundational invariants numerically: (1) KV-Cache memory reduction across MHA, GQA, and MQA; (2) exact shift invariance of Rotary Position Embeddings under equal position offsets; and (3) floating-point equivalence between block-tiled online softmax (FlashAttention-2) and full materialized causal attention.

transformer_foundations_lab.py — Numerical Verification of KV-Cache, RoPE & FlashAttention
import numpy as np

# 1. Exact KV-Cache Memory Calculator (GiB)
def kv_cache_gib(layers: int, kv_heads: int, head_dim: int, seq_len: int,
                 batch_size: int = 1, bytes_per_elem: int = 2) -> float:
    total_bytes = 2 * layers * kv_heads * head_dim * seq_len * batch_size * bytes_per_elem
    return total_bytes / (1024 ** 3)

print(f"MHA (32 KV heads): {kv_cache_gib(32, 32, 128, 32768):.2f} GiB")
print(f"GQA (8 KV heads):  {kv_cache_gib(32, 8, 128, 32768):.2f} GiB")
print(f"MQA (1 KV head):   {kv_cache_gib(32, 1, 128, 32768):.2f} GiB")

# 2. RoPE Relative-Position Invariance: <R_m q, R_n k> == <R_{m+delta} q, R_{n+delta} k>
def apply_rope(x: np.ndarray, pos: int, base: float = 10000.0) -> np.ndarray:
    d = x.shape[-1]
    half = d // 2
    freqs = base ** (-np.arange(0, half, dtype=np.float64) / half)
    angles = pos * freqs
    cos_a, sin_a = np.cos(angles), np.sin(angles)
    x1, x2 = x[..., 0::2], x[..., 1::2]
    out = np.empty_like(x, dtype=np.float64)
    out[..., 0::2] = x1 * cos_a - x2 * sin_a
    out[..., 1::2] = x1 * sin_a + x2 * cos_a
    return out

rng = np.random.default_rng(42)
q, k = rng.standard_normal(64), rng.standard_normal(64)
dot_1 = np.dot(apply_rope(q, 17), apply_rope(k, 42))
dot_2 = np.dot(apply_rope(q, 117), apply_rope(k, 142))
print(f"RoPE <R_17 q, R_42 k>   = {dot_1:.8f}")
print(f"RoPE <R_117 q, R_142 k> = {dot_2:.8f}")
print(f"Shift invariance diff   = {abs(dot_1 - dot_2):.2e}")

# 3. Tiled Online Softmax (FlashAttention-2) vs. Materialized Causal Attention
def standard_causal_attention(Q: np.ndarray, K: np.ndarray, V: np.ndarray) -> np.ndarray:
    N, d = Q.shape
    S = (Q @ K.T) / np.sqrt(d)
    S = np.where(np.triu(np.ones((N, N), dtype=bool), k=1), -np.inf, S)
    P = np.exp(S - np.max(S, axis=-1, keepdims=True))
    return (P / np.sum(P, axis=-1, keepdims=True)) @ V

def flash_attention_tiled(Q: np.ndarray, K: np.ndarray, V: np.ndarray,
                          Br: int = 16, Bc: int = 16) -> np.ndarray:
    N, d = Q.shape
    scale = 1.0 / np.sqrt(d)
    O = np.zeros((N, d), dtype=np.float64)

    for i in range(0, N, Br):
        q_end = min(i + Br, N)
        Qi = Q[i:q_end]
        rows = q_end - i
        m_i = np.full((rows, 1), -np.inf, dtype=np.float64)
        l_i = np.zeros((rows, 1), dtype=np.float64)
        O_tilde = np.zeros((rows, d), dtype=np.float64)

        for j in range(0, q_end, Bc):
            k_end = min(j + Bc, N)
            Kj, Vj = K[j:k_end], V[j:k_end]
            Sij = (Qi @ Kj.T) * scale

            q_idx = np.arange(i, q_end)[:, None]
            k_idx = np.arange(j, k_end)[None, :]
            Sij = np.where(k_idx > q_idx, -np.inf, Sij)

            m_new = np.maximum(m_i, np.max(Sij, axis=-1, keepdims=True))
            alpha = np.exp(m_i - m_new)
            P_tilde = np.exp(Sij - m_new)

            l_i = alpha * l_i + np.sum(P_tilde, axis=-1, keepdims=True)
            O_tilde = alpha * O_tilde + (P_tilde @ Vj)
            m_i = m_new

        O[i:q_end] = O_tilde / l_i
    return O

Q, K, V = rng.standard_normal((64, 32)), rng.standard_normal((64, 32)), rng.standard_normal((64, 32))
max_err = np.max(np.abs(standard_causal_attention(Q, K, V) - flash_attention_tiled(Q, K, V)))
print(f"FlashAttention max abs error vs exact: {max_err:.2e}")
Out [5]: MHA (32 KV heads): 16.00 GiB GQA (8 KV heads): 4.00 GiB MQA (1 KV head): 0.50 GiB RoPE <R_17 q, R_42 k> = -4.50127871 RoPE <R_117 q, R_142 k> = -4.50127871 Shift invariance diff = 3.55e-15 FlashAttention max abs error vs exact: 5.55e-16
In [6]:
Quiz

6. Questions and Explanations

Select the best answer for each of the 8 questions below, then click Submit & Grade Quiz to evaluate your understanding with source-cited explanations.

Q1 · Knowledge checkIn Scaled Dot-Product Attention softmax(QK^T / sqrt(d_k))V, why are the dot products divided by sqrt(d_k)?
Explanation: Vaswani et al. (2017) [1] proved that if components of q and k are independent with mean 0 and variance 1, q · k has variance d_k. Scaling by 1 / sqrt(d_k) normalizes variance back to 1, preventing extreme logits from saturating softmax into regions with near-zero gradients.
Q2 · Knowledge checkHow does Grouped-Query Attention (GQA) reduce KV-Cache memory consumption compared to Multi-Head Attention (MHA)?
Explanation: Ainslie et al. (2023) [3] introduced GQA as an interpolation between MHA (H_kv = H_q) and MQA (H_kv = 1) [2]. Sharing one KV head per group of query heads reduces KV-Cache memory and HBM memory-bandwidth overhead proportionally while preserving MHA-level modeling quality.
Q3 · Knowledge checkConsider a 32-layer Transformer with H_q = 32, H_kv = 8 (GQA), d_head = 128, running at batch size B = 1 and sequence length S = 32,768 in BF16 (2 bytes/element). What is the exact KV-Cache size?
Explanation: Using 2 × L × H_kv × d_head × S × B × b = 2 × 32 × 8 × 128 × 32,768 × 1 × 2 = 4,294,967,296 bytes = 4.00 GiB (128 KiB per token). Standard MHA (H_kv = 32) would require 16.00 GiB.
Q4 · Knowledge checkWhat mathematical property allows Rotary Position Embeddings (RoPE) to capture relative token distance inside standard dot-product attention?
Explanation: Su et al. (2021) [4] showed that rotating 2D subspaces of q at position m and k at position n yields an inner product invariant to absolute position shifts and dependent only on relative distance (n - m).
Q5 · Knowledge checkHow does FlashAttention achieve 2–4× faster execution and linear O(N) memory without approximating attention weights?
Explanation: Dao et al. (2022) [5] and Dao (2023) [6] eliminated the Θ(N^2) HBM memory-bandwidth bottleneck by keeping tiled blocks of S and P inside fast on-chip SRAM and maintaining running row-wise max/sum statistics for exact online softmax rescaling.
Q6 · Knowledge checkDuring the FlashAttention backward pass, how are gradients computed without having saved the N × N probability matrix P in HBM during the forward pass?
Explanation: Selective activation recomputation [5] trades a small amount of extra FLOPs in fast SRAM to avoid Θ(N^2) HBM reads/writes, storing only the O(N) logsumexp vector L from the forward pass to reconstruct exact softmax blocks on-chip.
Q7 · Knowledge checkWhat distinguishes RMSNorm (Zhang & Sennrich, 2019) from standard LayerNorm?
Explanation: Zhang & Sennrich (2019) [7] showed that the re-scaling invariance of LayerNorm—not mean-centering—accounts for its stabilization benefits, allowing RMSNorm to drop mean subtraction and bias terms for faster GPU execution.
Q8 · Knowledge checkIn a SwiGLU Feed-Forward Network (Shazeer, 2020), how is the layer output computed?
Explanation: Shazeer (2020) [8] introduced Gated Linear Unit variants including SwiGLU, which replaces the single up-projection and ReLU of standard FFNs with two parallel projections—one passed through Swish_1(z) = z · σ(z) and multiplied element-wise with the second—before the down-projection.
In [7]:
References

References and source links

  1. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems (NeurIPS 2017) / arXiv:1706.03762.
    https://arxiv.org/abs/1706.03762
  2. Shazeer, N. (2019). Fast Transformer Decoding: One Write-Head is All You Need (Multi-Query Attention). arXiv:1911.02150.
    https://arxiv.org/abs/1911.02150
  3. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., & Sanghai, S. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. Proceedings of EMNLP 2023 / arXiv:2305.13245.
    https://arxiv.org/abs/2305.13245
  4. Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864.
    https://arxiv.org/abs/2104.09864
  5. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. Advances in Neural Information Processing Systems (NeurIPS 2022) / arXiv:2205.14135.
    https://arxiv.org/abs/2205.14135
  6. Dao, T. (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. International Conference on Learning Representations (ICLR 2024) / arXiv:2307.08691.
    https://arxiv.org/abs/2307.08691
  7. Zhang, B., & Sennrich, R. (2019). Root Mean Square Layer Normalization (RMSNorm). Advances in Neural Information Processing Systems (NeurIPS 2019) / arXiv:1910.07467.
    https://arxiv.org/abs/1910.07467
  8. Shazeer, N. (2020). GLU Variants Improve Transformer (SwiGLU). arXiv:2002.05202.
    https://arxiv.org/abs/2002.05202