A rigorous engineering breakdown of autoregressive Transformer internals—from variance-stabilized self-attention and Grouped-Query Attention (GQA) memory budgets to Rotary Position Embeddings (RoPE), Pre-RMSNorm/SwiGLU blocks, and SRAM-tiled online softmax in FlashAttention.
Markdown
1. Scaled Dot-Product Attention & Variance Stabilization
Introduced by Vaswani et al. (2017) [1], the Transformer replaces recurrent and convolutional sequence processing with Scaled Dot-Product Attention. Given projected Query (Q ∈ R^{N × d_k}), Key (K ∈ R^{N × d_k}), and Value (V ∈ R^{N × d_v}) matrices for sequence length N, attention computes a content-addressable weighted sum:
Where M_{ij} = 0 if i >= j, and M_{ij} = -∞ if i < j (causal autoregressive mask)
Scaling Attention Logits by sqrt(d_k)
Suppose the scalar components of query vector q ∈ R^{d_k} and key vector k ∈ R^{d_k} are independent random variables with mean 0 and variance 1. The unscaled dot product q · k = ∑_{i=1}^{d_k} q_i k_i has mean 0, but its variance grows linearly with head dimension d_k [1]:
Scaling by 1 / sqrt(d_k) restores unit variance:
Var( (q · k) / sqrt(d_k) ) = (1 / d_k) · Var(q · k) = 1
Without the 1 / sqrt(d_k) factor, large head dimensions (such as d_k = 128, where standard deviation reaches sqrt(128) ≈ 11.31) push pre-softmax logits to extreme magnitudes. Because the Jacobian of softmax satisfies ∂p_i / ∂z_j = p_i (δ_{ij} - p_j), saturated one-hot probabilities (p_i → 1) drive gradients toward zero, stalling optimization.
Diagram
2. Attention Head Topologies (MHA, GQA, MQA) & KV-Cache Bottlenecks
During autoregressive inference, generating token t requires keys and values from all prior tokens 1..t-1. To avoid recomputing historical projections at every step, inference engines store past Key and Value tensors in GPU High Bandwidth Memory (HBM) as the KV-Cache.
In standard Multi-Head Attention (MHA) [1], every query head has a dedicated Key and Value head (H_q = H_{kv}), causing the KV-Cache to dominate GPU memory at long context lengths. Multi-Query Attention (MQA) (Shazeer, 2019) [2] shares a single KV head across all query heads (H_{kv} = 1), maximizing memory bandwidth savings at the cost of slight quality degradation. Grouped-Query Attention (GQA) (Ainslie et al., 2023) [3] interpolates between MHA and MQA by partitioning H_q query heads into G groups (H_{kv} = G), each sharing one Key and one Value head.
Diagram as text and editable source
- MHA: 8 query heads → 8 key/value headsOne pair per query head: 1:1
- GQA: 8 query heads2 groups of 4 → 2 key/value headsOne pair per group: 4:1
- MQA: 8 query heads → 1 key/value headOne pair shared by all: 8:1
Bytes_KV = 2 (for K & V) × L (layers) × H_kv (KV heads) × d_head × S (seq_len) × B (batch) × b (bytes/elem)
Example (L = 32, H_q = 32, d_head = 128, S = 32,768 tokens, B = 1, BF16 b = 2 bytes):
• MHA (H_kv = 32): 2 × 32 × 32 × 128 × 32,768 × 2 = 17,179,869,184 bytes = 16.00 GiB
• GQA (H_kv = 8): 2 × 32 × 8 × 128 × 32,768 × 2 = 4,294,967,296 bytes = 4.00 GiB
• MQA (H_kv = 1): 2 × 32 × 1 × 128 × 32,768 × 2 = 536,870,912 bytes = 0.50 GiB
Markdown
3. Modern Decoder Block: RoPE, Pre-RMSNorm & SwiGLU
While the original 2017 Transformer used Post-LayerNorm, sinusoidal additive positional embeddings, and ReLU feed-forward networks [1], contemporary open and frontier decoder-only LLMs converge on three standardized architectural upgrades:
A. Rotary Position Embeddings (RoPE — Su et al., 2021)
Instead of adding positional vectors to token embeddings at the input layer, RoPE [4] encodes position directly inside the attention dot product by rotating pairs of dimensions (x_{2i-1}, x_{2i}) in the complex plane by angle m · θ_i, where m is the token index and θ_i = θ_base^{-2(i-1)/d}:
Relative Position Property (Orthogonality of Rotation Matrices):
<R_m q, R_n k> = q^T R_m^T R_n k = q^T R_{n - m} k
Because the inner product depends strictly on the relative distance (n - m), RoPE preserves relative positional awareness across arbitrary offsets, remains fully compatible with linear/KV-cached attention, and supports long-context extension via base-frequency scaling (ABF / YaRN).
B. Pre-RMSNorm & SwiGLU Feed-Forward Networks
| Component | Original Transformer (2017) | Modern LLM Standard | Mathematical Form & Engineering Benefit |
|---|---|---|---|
| Normalization Placement & Operator | Post-LayerNorm:LayerNorm(x + Sublayer(x)) |
Pre-RMSNorm [7]:x + Sublayer(RMSNorm(x)) |
RMSNorm(x) = (x / sqrt(mean(x^2) + ε)) ⊙ γPreserves an unimpeded identity residual gradient highway and eliminates mean-centering overhead (~10–40% faster norm kernels). |
| Position-Wise FFN | 2-Matrix ReLU FFN:max(0, xW_1 + b_1)W_2 + b_2 |
3-Matrix SwiGLU [8]:(Swish_1(xW_gate) ⊙ xW_up) W_down |
Swish_1(z) = z · σ(z)Multiplicative gating improves perplexity at equal parameter/FLOP budgets (hidden width typically scaled by (2/3) · 4d to hold parameter count constant). |
Markdown
4. IO-Aware Exact Attention: FlashAttention & Online Softmax
Dao et al. (2022) [5] identified that standard attention is memory-bandwidth bound, not compute bound. Materializing the N × N attention score matrix S = QK^T / sqrt(d_k) and probability matrix P = softmax(S) requires reading and writing Θ(N^2) elements to slow GPU High Bandwidth Memory (HBM), even though on-chip SRAM is over an order of magnitude faster (~19 TB/s SRAM vs. ~1.5–3.3 TB/s HBM on A100/H100 GPUs).
FlashAttention [5] and FlashAttention-2 [6] compute exact attention (zero approximation) without ever writing S or P to HBM by combining two techniques:
- SRAM Block Tiling: Queries are partitioned into blocks of
B_rrows and Keys/Values into blocks ofB_ccolumns sized to fit inside on-chip SRAM (B_c = ceil(M / (4d)), whereMis SRAM capacity). - Online Softmax Rescaling: Softmax normally requires three passes over a row (find row max
m, compute exponentials and denominator sumℓ, divide byℓ). FlashAttention updates running row statistics block-by-block and rescales the partial output accumulatorÕwhenever a new block introduces a larger maximumm_new:
S^{(j)} = (Q_i K_j^T) / sqrt(d)
m_new = max( m_old, rowmax(S^{(j)}) )
P_tilde = exp( S^{(j)} - m_new )
ℓ_new = exp(m_old - m_new) · ℓ_old + rowsum(P_tilde)
O_tilde = diag(exp(m_old - m_new)) · O_tilde + P_tilde · V_j
After all tiles finish (FlashAttention-2 deferred normalization):
O_i = diag(ℓ_final)^{-1} · O_tilde
Θ(N d + N^2) HBM accesses. FlashAttention reduces HBM accesses to Θ(N^2 d^2 / M)—for typical d = 64–128 and SRAM M = 192–228 KB, yielding up to 9× fewer HBM reads/writes and 2–4× wall-clock speedup [5], [6]. During backpropagation, instead of storing the N × N matrix P from the forward pass, FlashAttention stores only the output O and row-wise logsumexp vector L = m + log(ℓ) (O(N) memory) and recomputes S and P on-chip in SRAM.
Python
5. Verification Lab: KV-Cache Scaling, RoPE Invariance & Tiled Online Softmax
The self-contained NumPy implementation below verifies three foundational invariants numerically: (1) KV-Cache memory reduction across MHA, GQA, and MQA; (2) exact shift invariance of Rotary Position Embeddings under equal position offsets; and (3) floating-point equivalence between block-tiled online softmax (FlashAttention-2) and full materialized causal attention.
import numpy as np
# 1. Exact KV-Cache Memory Calculator (GiB)
def kv_cache_gib(layers: int, kv_heads: int, head_dim: int, seq_len: int,
batch_size: int = 1, bytes_per_elem: int = 2) -> float:
total_bytes = 2 * layers * kv_heads * head_dim * seq_len * batch_size * bytes_per_elem
return total_bytes / (1024 ** 3)
print(f"MHA (32 KV heads): {kv_cache_gib(32, 32, 128, 32768):.2f} GiB")
print(f"GQA (8 KV heads): {kv_cache_gib(32, 8, 128, 32768):.2f} GiB")
print(f"MQA (1 KV head): {kv_cache_gib(32, 1, 128, 32768):.2f} GiB")
# 2. RoPE Relative-Position Invariance: <R_m q, R_n k> == <R_{m+delta} q, R_{n+delta} k>
def apply_rope(x: np.ndarray, pos: int, base: float = 10000.0) -> np.ndarray:
d = x.shape[-1]
half = d // 2
freqs = base ** (-np.arange(0, half, dtype=np.float64) / half)
angles = pos * freqs
cos_a, sin_a = np.cos(angles), np.sin(angles)
x1, x2 = x[..., 0::2], x[..., 1::2]
out = np.empty_like(x, dtype=np.float64)
out[..., 0::2] = x1 * cos_a - x2 * sin_a
out[..., 1::2] = x1 * sin_a + x2 * cos_a
return out
rng = np.random.default_rng(42)
q, k = rng.standard_normal(64), rng.standard_normal(64)
dot_1 = np.dot(apply_rope(q, 17), apply_rope(k, 42))
dot_2 = np.dot(apply_rope(q, 117), apply_rope(k, 142))
print(f"RoPE <R_17 q, R_42 k> = {dot_1:.8f}")
print(f"RoPE <R_117 q, R_142 k> = {dot_2:.8f}")
print(f"Shift invariance diff = {abs(dot_1 - dot_2):.2e}")
# 3. Tiled Online Softmax (FlashAttention-2) vs. Materialized Causal Attention
def standard_causal_attention(Q: np.ndarray, K: np.ndarray, V: np.ndarray) -> np.ndarray:
N, d = Q.shape
S = (Q @ K.T) / np.sqrt(d)
S = np.where(np.triu(np.ones((N, N), dtype=bool), k=1), -np.inf, S)
P = np.exp(S - np.max(S, axis=-1, keepdims=True))
return (P / np.sum(P, axis=-1, keepdims=True)) @ V
def flash_attention_tiled(Q: np.ndarray, K: np.ndarray, V: np.ndarray,
Br: int = 16, Bc: int = 16) -> np.ndarray:
N, d = Q.shape
scale = 1.0 / np.sqrt(d)
O = np.zeros((N, d), dtype=np.float64)
for i in range(0, N, Br):
q_end = min(i + Br, N)
Qi = Q[i:q_end]
rows = q_end - i
m_i = np.full((rows, 1), -np.inf, dtype=np.float64)
l_i = np.zeros((rows, 1), dtype=np.float64)
O_tilde = np.zeros((rows, d), dtype=np.float64)
for j in range(0, q_end, Bc):
k_end = min(j + Bc, N)
Kj, Vj = K[j:k_end], V[j:k_end]
Sij = (Qi @ Kj.T) * scale
q_idx = np.arange(i, q_end)[:, None]
k_idx = np.arange(j, k_end)[None, :]
Sij = np.where(k_idx > q_idx, -np.inf, Sij)
m_new = np.maximum(m_i, np.max(Sij, axis=-1, keepdims=True))
alpha = np.exp(m_i - m_new)
P_tilde = np.exp(Sij - m_new)
l_i = alpha * l_i + np.sum(P_tilde, axis=-1, keepdims=True)
O_tilde = alpha * O_tilde + (P_tilde @ Vj)
m_i = m_new
O[i:q_end] = O_tilde / l_i
return O
Q, K, V = rng.standard_normal((64, 32)), rng.standard_normal((64, 32)), rng.standard_normal((64, 32))
max_err = np.max(np.abs(standard_causal_attention(Q, K, V) - flash_attention_tiled(Q, K, V)))
print(f"FlashAttention max abs error vs exact: {max_err:.2e}")
Quiz
6. Questions and Explanations
Select the best answer for each of the 8 questions below, then click Submit & Grade Quiz to evaluate your understanding with source-cited explanations.
Q1 · Knowledge checkIn Scaled Dot-Product Attention softmax(QK^T / sqrt(d_k))V, why are the dot products divided by sqrt(d_k)?
q and k are independent with mean 0 and variance 1, q · k has variance d_k. Scaling by 1 / sqrt(d_k) normalizes variance back to 1, preventing extreme logits from saturating softmax into regions with near-zero gradients.
Q2 · Knowledge checkHow does Grouped-Query Attention (GQA) reduce KV-Cache memory consumption compared to Multi-Head Attention (MHA)?
H_kv = H_q) and MQA (H_kv = 1) [2]. Sharing one KV head per group of query heads reduces KV-Cache memory and HBM memory-bandwidth overhead proportionally while preserving MHA-level modeling quality.
Q3 · Knowledge checkConsider a 32-layer Transformer with H_q = 32, H_kv = 8 (GQA), d_head = 128, running at batch size B = 1 and sequence length S = 32,768 in BF16 (2 bytes/element). What is the exact KV-Cache size?
2 × L × H_kv × d_head × S × B × b = 2 × 32 × 8 × 128 × 32,768 × 1 × 2 = 4,294,967,296 bytes = 4.00 GiB (128 KiB per token). Standard MHA (H_kv = 32) would require 16.00 GiB.
Q4 · Knowledge checkWhat mathematical property allows Rotary Position Embeddings (RoPE) to capture relative token distance inside standard dot-product attention?
q at position m and k at position n yields an inner product invariant to absolute position shifts and dependent only on relative distance (n - m).
Q5 · Knowledge checkHow does FlashAttention achieve 2–4× faster execution and linear O(N) memory without approximating attention weights?
Θ(N^2) HBM memory-bandwidth bottleneck by keeping tiled blocks of S and P inside fast on-chip SRAM and maintaining running row-wise max/sum statistics for exact online softmax rescaling.
Q6 · Knowledge checkDuring the FlashAttention backward pass, how are gradients computed without having saved the N × N probability matrix P in HBM during the forward pass?
Θ(N^2) HBM reads/writes, storing only the O(N) logsumexp vector L from the forward pass to reconstruct exact softmax blocks on-chip.
Q7 · Knowledge checkWhat distinguishes RMSNorm (Zhang & Sennrich, 2019) from standard LayerNorm?
Q8 · Knowledge checkIn a SwiGLU Feed-Forward Network (Shazeer, 2020), how is the layer output computed?
Swish_1(z) = z · σ(z) and multiplied element-wise with the second—before the down-projection.
References
References and source links
-
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems (NeurIPS 2017) / arXiv:1706.03762.
https://arxiv.org/abs/1706.03762 -
Shazeer, N. (2019). Fast Transformer Decoding: One Write-Head is All You Need (Multi-Query Attention). arXiv:1911.02150.
https://arxiv.org/abs/1911.02150 -
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., & Sanghai, S. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. Proceedings of EMNLP 2023 / arXiv:2305.13245.
https://arxiv.org/abs/2305.13245 -
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864.
https://arxiv.org/abs/2104.09864 -
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. Advances in Neural Information Processing Systems (NeurIPS 2022) / arXiv:2205.14135.
https://arxiv.org/abs/2205.14135 -
Dao, T. (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. International Conference on Learning Representations (ICLR 2024) / arXiv:2307.08691.
https://arxiv.org/abs/2307.08691 -
Zhang, B., & Sennrich, R. (2019). Root Mean Square Layer Normalization (RMSNorm). Advances in Neural Information Processing Systems (NeurIPS 2019) / arXiv:1910.07467.
https://arxiv.org/abs/1910.07467 -
Shazeer, N. (2020). GLU Variants Improve Transformer (SwiGLU). arXiv:2002.05202.
https://arxiv.org/abs/2002.05202