Training a modern foundation or domain-specialized AI model is a multi-stage systems engineering discipline. Each phase transforms raw compute and curated data into distinct capabilities: Pre-Training compresses world knowledge and syntax into base weights; Supervised Fine-Tuning (SFT) instills conversational formatting and instruction following; Preference & Reasoning Post-Training (DPO / GRPO) sharpens multi-step reasoning and behavioral alignment; and Evaluation & Red-Teaming gate production deployment.
AI Training Lifecycle
Click any stage in the pipeline diagram or stage cards below to inspect its training objective, mathematical loss/signal formulation, dataset schema, compute footprint, and failure modes:
Diagram as text and editable source
- Data preparationand tokenization → Pre-trainingNext-token prediction: Training data
- Pre-trainingNext-token prediction → Supervised fine-tuningDemonstrations: Optional adaptation
- Supervised fine-tuningDemonstrations → Preference / rewardoptimization: Optional post-training
- Preference / rewardoptimization → Evaluation anddeployment decisions: Assess results
Data Curation & Tokenization
GPU VRAM Estimator & Alignment Matrix
Configure model parameters, weight quantization, training algorithm, and sequence length below to estimate total GPU memory requirements across weights, gradients, optimizer states, and activations:
Post-Training & Alignment Method Matrix
| Method | Learning Signal | Reward Model Needed? | Memory / Compute Cost | Ideal Use Cases |
|---|---|---|---|---|
| SFT Supervised Fine-Tuning |
Token-level cross-entropy imitating gold-standard demonstration completions (prompt tokens masked with -100). |
No (static (prompt, response) dataset). |
Lowest — Single policy forward/backward pass; no online rollout generation. | Teaching chat templates, JSON/tool-calling schemas, domain style, and cold-start formatting before RL. |
| DPO Direct Preference Opt. |
Offline contrastive Bradley-Terry log-ratio loss maximizing margin between chosen vs. rejected pairs relative to a reference policy. |
No (reward is implicitly parameterized by β log(π_θ / π_ref)). |
Low–Moderate — Offline pairs (no rollout sampling), but requires holding frozen π_ref (or PEFT adapter toggling). |
Tone/helpfulness alignment, refusal calibration, and style preference tuning from static human/AI preference pairs. |
| PPO (RLHF) Proximal Policy Opt. |
Online policy-gradient updates guided by a learned scalar Reward Model + per-token KL penalty + GAE advantage from a Critic network. | Yes (separate trained Reward Model + Value/Critic network). | Highest — Must host 4 models (Actor, Critic, Reward, Reference) plus autoregressive rollout generation. | Open-ended subjective human preferences where dense token-level value estimation stabilizes multi-turn dialogue. |
| GRPO Group Relative Policy Opt. |
Online RL sampling a group of G completions per prompt and computing group-normalized relative advantages (r_i - mean) / std. |
No learned RM (uses deterministic programmatic verifiers for math/code). | Moderate–High — Eliminates the Critic network (~50% RL memory cut vs. PPO), dominated by G rollout generation KV cache. |
Verifiable reasoning tasks: competitive mathematics, code generation with unit tests, formal logic, and SQL/tool execution. |
Data Engineering & Distributed Parallelism
Model quality is bounded by corpus cleanliness, while training speed is bounded by inter-GPU communication bandwidth. Below are the core engineering pillars for data pipelines and multi-GPU scaling:
Data Quality & Curation Pipelines
Web scrapes contain massive boilerplate repetition (navigation bars, license headers, syndicated articles). Removing exact substring duplicates and near-duplicates via MinHash Locality-Sensitive Hashing prevents memorization, accelerates convergence, and lowers extraction leakage.
Applies language identification, perplexity/classifier quality scoring, regex/NER scrubbing for Personally Identifiable Information (emails, SSNs, API keys), and NSFW/toxicity classifiers before tokenization.
Scans training documents against held-out evaluation suites (MMLU, GSM8K, HumanEval, MATH) using n-gram and embedding similarity matching so test prompts never leak into training shards.
Recursively training models on unverified AI-generated outputs causes Model Collapse—progressive loss of low-probability tail distributions and variance collapse. Mitigate by grounding synthetic data with deterministic verifiers (compilers, math solvers) and mixing fresh human-authored corpora.
Distributed Scaling & Memory Parallelism
DDP replicates the full model, gradients, and optimizer states on every GPU (fast, but hits OOM above ~7B–13B). FSDP / ZeRO-3 shards parameters, gradients, and AdamW states across all data-parallel ranks, gathering layers on-the-fly via AllGather and ReduceScatter.
Splits individual attention heads and MLP weight matrices column-wise and row-wise across GPUs within a single node. Requires ultra-high-bandwidth NVLink / NVSwitch interconnects due to per-layer AllReduce synchronization.
Partitions sequential transformer layers across multiple nodes over InfiniBand/RoCE. Uses micro-batch scheduling (such as 1F1B — One Forward, One Backward) to minimize idle GPU "pipeline bubbles."
BF16 preserves FP32's 8-bit exponent range (preventing gradient underflow/overflow without loss scalers) while halving tensor memory. Activation / Gradient Checkpointing discards intermediate forward activations and recomputes them during backprop—cutting activation VRAM by ~70% for ~25% extra FLOPs.
Worked Code Examples (QLoRA SFT, DPO & GRPO)
Reference implementations using Hugging Face transformers, peft, bitsandbytes, and trl. Click Copy on any block to copy clean, runnable Python code:
import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig
model_id = "Qwen/Qwen2.5-7B-Instruct"
# 1. Configure 4-bit NormalFloat4 (NF4) quantization with double quantization
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(
model_id,
quantization_config=bnb_config,
device_map="auto",
torch_dtype=torch.bfloat16,
)
# 2. Target both attention and MLP projections for higher adaptation capacity
peft_config = LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
)
# Dataset formatted with conversational "messages" list; SFTTrainer masks prompt tokens
dataset = load_dataset("json", data_files="data/domain_sft_conversations.jsonl", split="train")
sft_config = SFTConfig(
output_dir="./checkpoints/qwen2.5-7b-qlora-sft",
max_seq_length=4096,
per_device_train_batch_size=2,
gradient_accumulation_steps=8, # Effective global batch size = 16
learning_rate=2e-4,
lr_scheduler_type="cosine",
warmup_ratio=0.05,
num_train_epochs=2,
bf16=True,
gradient_checkpointing=True,
optim="paged_adamw_8bit",
logging_steps=10,
)
trainer = SFTTrainer(
model=model,
args=sft_config,
train_dataset=dataset,
processing_class=tokenizer,
peft_config=peft_config,
)
trainer.train()
trainer.save_model("./checkpoints/qwen2.5-7b-qlora-sft/final_adapter")
import torch
from datasets import Dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig
from trl import DPOTrainer, DPOConfig
sft_model_path = "./checkpoints/qwen2.5-7b-sft-merged"
tokenizer = AutoTokenizer.from_pretrained(sft_model_path)
tokenizer.pad_token = tokenizer.eos_token
# Load policy model; when peft_config is passed, DPOTrainer disables adapters
# to compute reference policy log-probs without duplicating base weights in VRAM!
model = AutoModelForCausalLM.from_pretrained(
sft_model_path,
torch_dtype=torch.bfloat16,
device_map="auto",
)
# Preference dataset schema: {"prompt": str, "chosen": str, "rejected": str}
preference_pairs = Dataset.from_list([
{
"prompt": "Explain SQL injection and how to prevent it in Python.",
"chosen": "SQL injection occurs when untrusted input alters query syntax. Always use parameterized queries (e.g., cursor.execute('SELECT * FROM users WHERE id = %s', (user_id,))) instead of string formatting.",
"rejected": "SQL injection is bad. Just strip single quotes from the user input string before concatenating it into your SQL query."
}
])
dpo_config = DPOConfig(
output_dir="./checkpoints/qwen2.5-7b-dpo",
beta=0.1, # KL divergence penalty weight controlling drift from reference
loss_type="sigmoid", # Standard Bradley-Terry DPO objective
per_device_train_batch_size=2,
gradient_accumulation_steps=8,
learning_rate=5e-6, # Lower LR than SFT to prevent policy collapse
max_length=2048,
max_prompt_length=1024,
bf16=True,
gradient_checkpointing=True,
)
peft_config = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj", "o_proj"])
dpo_trainer = DPOTrainer(
model=model,
ref_model=None, # Uses base weights (adapter disabled) as implicit pi_ref
args=dpo_config,
train_dataset=preference_pairs,
processing_class=tokenizer,
peft_config=peft_config,
)
dpo_trainer.train()
import re
import subprocess
import tempfile
def extract_boxed_answer(text: str) -> str | None:
"""Extracts final answer from \\boxed{...} tags in reasoning rollouts."""
match = re.search(r"\\boxed\{([^{}]+)\}", text)
return match.group(1).strip() if match else None
def format_reward_func(completions: list[str], **kwargs) -> list[float]:
"""Rewards strict ... reasoning structure (+0.25) before final answer."""
pattern = re.compile(r"^.*?\s*.+", re.DOTALL)
return [0.25 if pattern.match(c.strip()) else 0.0 for c in completions]
def math_accuracy_reward_func(completions: list[str], ground_truth: list[str], **kwargs) -> list[float]:
"""Deterministic verifier for GRPO: compares extracted answer against exact ground truth."""
rewards = []
for completion, truth in zip(completions, ground_truth):
pred = extract_boxed_answer(completion)
if pred is not None and pred.replace(" ", "") == truth.strip().replace(" ", ""):
rewards.append(1.0)
else:
rewards.append(0.0)
return rewards
def python_unit_test_reward_func(completions: list[str], test_harness: list[str], **kwargs) -> list[float]:
"""Executes generated Python code against unit tests in an isolated subprocess."""
rewards = []
for completion, tests in zip(completions, test_harness):
code_match = re.search(r"```python\n(.*?)```", completion, re.DOTALL)
if not code_match:
rewards.append(0.0)
continue
candidate_script = f"{code_match.group(1)}\n\n{tests}\n"
with tempfile.NamedTemporaryFile(mode="w", suffix=".py", delete=True) as tmp:
tmp.write(candidate_script)
tmp.flush()
try:
proc = subprocess.run(["python3", tmp.name], timeout=3, capture_output=True)
rewards.append(1.0 if proc.returncode == 0 else 0.0)
except subprocess.TimeoutExpired:
rewards.append(0.0)
return rewards
Study Q&A Bank + Custom Study Note Builder
381 as 1 token vs. 3829 as 38+29) degrades arithmetic unless digit-level splitting (1–3 digits max per token) is enforced; (2) Leading Whitespace & Indentation: " apple" and "apple" map to different token IDs, and poor Python whitespace merges bloat code token counts; (3) Multilingual Fertility: under-represented scripts require 3–5× more tokens per word ("high token fertility"), shrinking effective context windows; and (4) Special Token Injection: unescaped control tokens like <|im_end|> in raw user text must be sanitized to prevent prompt boundary spoofing.
C ≈ 6ND, model parameters (N) and training tokens (D) should scale in equal proportion—roughly 20 tokens per parameter (e.g., 1.4T tokens for a 70B model). However, Chinchilla only minimizes training compute. When serving billions of inference queries, smaller models trained far past the Chinchilla frontier—such as 7B–8B models trained on 10T–15T+ tokens ("inference-optimal over-training")—achieve lower total lifecycle cost and lower latency per request.
1e-5 to 2e-4 with short 1–3 epoch schedules; and (4) KL-Divergence Regularization against the initial SFT checkpoint during post-training.
W' = W_0 + (lora_alpha / r) · B · A, where A ∈ R^(r×k) and B ∈ R^(d×r). Increasing rank r (typically 8 to 64) increases expressive capacity and trainable parameters, which helps on complex reasoning or multi-task domains, whereas small r (4–16) acts as a strong regularizer on small datasets. lora_alpha scales the magnitude of the adapter update via alpha / r (commonly set to alpha = 2 * r). Applying LoRA to all linear projections (attention q, k, v, o plus MLP gate, up, down) consistently outperforms attention-only LoRA.
-β · D_KL(π_θ || π_ref) anchors the updated policy π_θ near the fluent reference policy π_ref, preserving linguistic coherence and diversity while shifting probability mass toward preferred or verified completions.
gradient_accumulation_steps = K), the GPU runs K sequential forward and backward passes on small micro-batches, summing gradients in the gradient buffer (normalized by 1/K) without calling optimizer.step() or freeing/allocating peak activations simultaneously. The optimizer updates weights once every K micro-steps, yielding an exact mathematical equivalent to a global batch size of micro_batch × K × num_GPUs.
nn.CrossEntropyLoss(ignore_index=-100), any target token labeled -100 contributes zero loss and zero gradient. Masking system instructions and user prompt tokens ensures the model focuses 100% of its gradient capacity on learning to generate high-quality assistant responses rather than wasting capacity memorizing user queries (which are supplied at inference time anyway). In multi-turn conversations, only assistant turns carry active token labels.
q, it samples a group of G candidate outputs {o_1, ..., o_G}, scores each with a deterministic verifier r_i, and computes each output's normalized advantage relative to its peers: A_i = (r_i - mean(r)) / (std(r) + ε). This slashes RL VRAM overhead by nearly half while excelling on verifiable math and code tasks.
Interactive 5-Question AI Training Self-Check Quiz
During Supervised Fine-Tuning (SFT) of a causal chat model, why are user prompt and system instruction token labels set to -100?
Why does QLoRA (4-bit NF4 base weights + LoRA adapters) reduce GPU VRAM by ~70–80% compared to standard BF16 Full Fine-Tuning?
How does Direct Preference Optimization (DPO) simplify alignment compared to traditional PPO-based RLHF?
In Group Relative Policy Optimization (GRPO) for math and code reasoning, how is the advantage of each candidate rollout computed without a learned Critic network?
After 8 epochs of domain SFT, training loss approaches 0.05, validation loss rises sharply, and the model fails basic general reasoning benchmarks it previously passed. What occurred and how should you fix it?
References and source links
- Ouyang et al. (2022). Training language models to follow instructions with human feedback.https://arxiv.org/abs/2203.02155
- Hugging Face maintainers. Training a causal language model from scratch.https://huggingface.co/learn/llm-course/chapter7/6
- Rafailov et al. (2023). Direct Preference Optimization.https://arxiv.org/abs/2305.18290
Reference list added during the visual refresh on 14 Nov 2026. It is a reading path, not a completed claim-by-claim audit of this imported notebook.