← Learn / Notebooks

Models & toolsTechnicalStudy guide

Training an AI

Follow the training lifecycle, compare alignment methods and explore data pipelines, GPU memory and worked Python examples.

Before you start: Familiarity with Python and basic machine learning concepts.

Imported AI-assisted study notebook · Added 13 Nov 2026. Source links are retained; the material has not passed ResearchNode’s article source-support review.

Training a modern foundation or domain-specialized AI model is a multi-stage systems engineering discipline. Each phase transforms raw compute and curated data into distinct capabilities: Pre-Training compresses world knowledge and syntax into base weights; Supervised Fine-Tuning (SFT) instills conversational formatting and instruction following; Preference & Reasoning Post-Training (DPO / GRPO) sharpens multi-step reasoning and behavioral alignment; and Evaluation & Red-Teaming gate production deployment.

Pre-Training Objective
Causal Next-Token
PEFT Memory Win
QLoRA 4-bit NF4
Reasoning RL
GRPO (Critic-Free)
Distributed Scaling
FSDP / 3D Parallel
Section 01 · Pipeline Architecture

AI Training Lifecycle

Click any stage in the pipeline diagram or stage cards below to inspect its training objective, mathematical loss/signal formulation, dataset schema, compute footprint, and failure modes:

AI training: one possible development route
Illustrative route, not a universal five-stage recipe. Projects may start from a pretrained model, omit or repeat adaptation steps, and evaluate throughout. Arrows show the route pictured here; they do not imply automatic retraining or guaranteed improvement. Source 1 · Source 2
Diagram as text and editable source
  • Data preparationand tokenization → Pre-trainingNext-token prediction: Training data
  • Pre-trainingNext-token prediction → Supervised fine-tuningDemonstrations: Optional adaptation
  • Supervised fine-tuningDemonstrations → Preference / rewardoptimization: Optional post-training
  • Preference / rewardoptimization → Evaluation anddeployment decisions: Assess results
Download editable draw.io diagram
Stage 01 · Corpus Engineering & Vocabulary

Data Curation & Tokenization

CPU / Distributed Ray / Spark ETL
Engineering Objective
Transform raw, noisy web, code, and domain corpora into high-signal, deduplicated, decontaminated token ID sequences packed cleanly to the target context window.
Signal / Loss Formulation
BPE Merge: (x*, y*) = argmax_{(x,y)} count(x, y) | Jaccard(A, B) = |A ∩ B| / |A ∪ B| ≥ τ_dedup
Dataset Schema & Artifacts
Raw JSONL documents → cleaned text shards → uint16/uint32 binary token arrays (.bin / .idx) with explicit BOS/EOS and role delimiters (<|im_start|>, <|im_end|>).
Compute & Throughput Profile
CPU- and I/O-bound distributed processing (hundreds to thousands of CPU cores). Fast NVMe storage and streaming parquet pipelines dominate wall-clock time.
Common Failure Modes & Mitigations
Duplicate boilerplate causing verbatim memorization; benchmark leakage inflating eval scores; poor tokenizer fertility on code/math/non-English scripts; unescaped special control tokens.
Section 02 · Resource Planning & Post-Training Trade-offs

GPU VRAM Estimator & Alignment Matrix

Configure model parameters, weight quantization, training algorithm, and sequence length below to estimate total GPU memory requirements across weights, gradients, optimizer states, and activations:

Estimated Peak Training VRAM
20.7 GB (includes ~15% PyTorch CUDA allocator overhead)
✓ Fits on Single 24GB GPU (RTX 4090 / L4 / A10G)
Model Weights
14.0 GB
BF16 base weights
Gradients
0.21 GB
LoRA r=16 adapter grads (~1.5%)
Optimizer States
0.84 GB
AdamW states on LoRA adapters
Activations / KV
2.9 GB
Checkpointed activations (4096 tok)

Post-Training & Alignment Method Matrix

Method Learning Signal Reward Model Needed? Memory / Compute Cost Ideal Use Cases
SFT
Supervised Fine-Tuning
Token-level cross-entropy imitating gold-standard demonstration completions (prompt tokens masked with -100). No (static (prompt, response) dataset). Lowest — Single policy forward/backward pass; no online rollout generation. Teaching chat templates, JSON/tool-calling schemas, domain style, and cold-start formatting before RL.
DPO
Direct Preference Opt.
Offline contrastive Bradley-Terry log-ratio loss maximizing margin between chosen vs. rejected pairs relative to a reference policy. No (reward is implicitly parameterized by β log(π_θ / π_ref)). Low–Moderate — Offline pairs (no rollout sampling), but requires holding frozen π_ref (or PEFT adapter toggling). Tone/helpfulness alignment, refusal calibration, and style preference tuning from static human/AI preference pairs.
PPO (RLHF)
Proximal Policy Opt.
Online policy-gradient updates guided by a learned scalar Reward Model + per-token KL penalty + GAE advantage from a Critic network. Yes (separate trained Reward Model + Value/Critic network). Highest — Must host 4 models (Actor, Critic, Reward, Reference) plus autoregressive rollout generation. Open-ended subjective human preferences where dense token-level value estimation stabilizes multi-turn dialogue.
GRPO
Group Relative Policy Opt.
Online RL sampling a group of G completions per prompt and computing group-normalized relative advantages (r_i - mean) / std. No learned RM (uses deterministic programmatic verifiers for math/code). Moderate–High — Eliminates the Critic network (~50% RL memory cut vs. PPO), dominated by G rollout generation KV cache. Verifiable reasoning tasks: competitive mathematics, code generation with unit tests, formal logic, and SQL/tool execution.
Section 03 · Data Quality & Cluster Systems

Data Engineering & Distributed Parallelism

Model quality is bounded by corpus cleanliness, while training speed is bounded by inter-GPU communication bandwidth. Below are the core engineering pillars for data pipelines and multi-GPU scaling:

Data Engineering

Data Quality & Curation Pipelines

1. Exact & Fuzzy Deduplication (MinHash LSH) Jaccard Similarity

Web scrapes contain massive boilerplate repetition (navigation bars, license headers, syndicated articles). Removing exact substring duplicates and near-duplicates via MinHash Locality-Sensitive Hashing prevents memorization, accelerates convergence, and lowers extraction leakage.

2. Heuristic, Toxicity & PII Filtering Privacy & Signal

Applies language identification, perplexity/classifier quality scoring, regex/NER scrubbing for Personally Identifiable Information (emails, SSNs, API keys), and NSFW/toxicity classifiers before tokenization.

3. Benchmark Contamination Filtering 13-gram Overlap

Scans training documents against held-out evaluation suites (MMLU, GSM8K, HumanEval, MATH) using n-gram and embedding similarity matching so test prompts never leak into training shards.

4. Synthetic Distillation & Model Collapse Risks Tail Truncation

Recursively training models on unverified AI-generated outputs causes Model Collapse—progressive loss of low-probability tail distributions and variance collapse. Mitigate by grounding synthetic data with deterministic verifiers (compilers, math solvers) and mixing fresh human-authored corpora.

Systems & Hardware

Distributed Scaling & Memory Parallelism

1. DDP vs. PyTorch FSDP / DeepSpeed ZeRO-3 Data Sharding

DDP replicates the full model, gradients, and optimizer states on every GPU (fast, but hits OOM above ~7B–13B). FSDP / ZeRO-3 shards parameters, gradients, and AdamW states across all data-parallel ranks, gathering layers on-the-fly via AllGather and ReduceScatter.

2. Tensor Parallelism (TP) Intra-Layer NVLink

Splits individual attention heads and MLP weight matrices column-wise and row-wise across GPUs within a single node. Requires ultra-high-bandwidth NVLink / NVSwitch interconnects due to per-layer AllReduce synchronization.

3. Pipeline Parallelism (PP) Inter-Node Layers

Partitions sequential transformer layers across multiple nodes over InfiniBand/RoCE. Uses micro-batch scheduling (such as 1F1B — One Forward, One Backward) to minimize idle GPU "pipeline bubbles."

4. Mixed Precision (BF16) & Gradient Checkpointing Compute vs. VRAM

BF16 preserves FP32's 8-bit exponent range (preventing gradient underflow/overflow without loss scalers) while halving tensor memory. Activation / Gradient Checkpointing discards intermediate forward activations and recomputes them during backprop—cutting activation VRAM by ~70% for ~25% extra FLOPs.

Section 04 · Production Python Recipes

Worked Code Examples (QLoRA SFT, DPO & GRPO)

Reference implementations using Hugging Face transformers, peft, bitsandbytes, and trl. Click Copy on any block to copy clean, runnable Python code:

Example 1 · train_qlora_sft.py — 4-bit NF4 QLoRA Supervised Fine-Tuning (SFTTrainer)
import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig

model_id = "Qwen/Qwen2.5-7B-Instruct"

# 1. Configure 4-bit NormalFloat4 (NF4) quantization with double quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

# 2. Target both attention and MLP projections for higher adaptation capacity
peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
)

# Dataset formatted with conversational "messages" list; SFTTrainer masks prompt tokens
dataset = load_dataset("json", data_files="data/domain_sft_conversations.jsonl", split="train")

sft_config = SFTConfig(
    output_dir="./checkpoints/qwen2.5-7b-qlora-sft",
    max_seq_length=4096,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,   # Effective global batch size = 16
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    num_train_epochs=2,
    bf16=True,
    gradient_checkpointing=True,
    optim="paged_adamw_8bit",
    logging_steps=10,
)

trainer = SFTTrainer(
    model=model,
    args=sft_config,
    train_dataset=dataset,
    processing_class=tokenizer,
    peft_config=peft_config,
)

trainer.train()
trainer.save_model("./checkpoints/qwen2.5-7b-qlora-sft/final_adapter")
Example 2 · train_dpo.py — Direct Preference Optimization (DPOTrainer + KL Beta)
import torch
from datasets import Dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig
from trl import DPOTrainer, DPOConfig

sft_model_path = "./checkpoints/qwen2.5-7b-sft-merged"

tokenizer = AutoTokenizer.from_pretrained(sft_model_path)
tokenizer.pad_token = tokenizer.eos_token

# Load policy model; when peft_config is passed, DPOTrainer disables adapters
# to compute reference policy log-probs without duplicating base weights in VRAM!
model = AutoModelForCausalLM.from_pretrained(
    sft_model_path,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# Preference dataset schema: {"prompt": str, "chosen": str, "rejected": str}
preference_pairs = Dataset.from_list([
    {
        "prompt": "Explain SQL injection and how to prevent it in Python.",
        "chosen": "SQL injection occurs when untrusted input alters query syntax. Always use parameterized queries (e.g., cursor.execute('SELECT * FROM users WHERE id = %s', (user_id,))) instead of string formatting.",
        "rejected": "SQL injection is bad. Just strip single quotes from the user input string before concatenating it into your SQL query."
    }
])

dpo_config = DPOConfig(
    output_dir="./checkpoints/qwen2.5-7b-dpo",
    beta=0.1,                        # KL divergence penalty weight controlling drift from reference
    loss_type="sigmoid",             # Standard Bradley-Terry DPO objective
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=5e-6,              # Lower LR than SFT to prevent policy collapse
    max_length=2048,
    max_prompt_length=1024,
    bf16=True,
    gradient_checkpointing=True,
)

peft_config = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj", "o_proj"])

dpo_trainer = DPOTrainer(
    model=model,
    ref_model=None,                  # Uses base weights (adapter disabled) as implicit pi_ref
    args=dpo_config,
    train_dataset=preference_pairs,
    processing_class=tokenizer,
    peft_config=peft_config,
)

dpo_trainer.train()
Example 3 · grpo_verifiable_rewards.py — Programmatic Math & Code Verification for GRPO
import re
import subprocess
import tempfile

def extract_boxed_answer(text: str) -> str | None:
    """Extracts final answer from \\boxed{...} tags in reasoning rollouts."""
    match = re.search(r"\\boxed\{([^{}]+)\}", text)
    return match.group(1).strip() if match else None

def format_reward_func(completions: list[str], **kwargs) -> list[float]:
    """Rewards strict ... reasoning structure (+0.25) before final answer."""
    pattern = re.compile(r"^.*?\s*.+", re.DOTALL)
    return [0.25 if pattern.match(c.strip()) else 0.0 for c in completions]

def math_accuracy_reward_func(completions: list[str], ground_truth: list[str], **kwargs) -> list[float]:
    """Deterministic verifier for GRPO: compares extracted answer against exact ground truth."""
    rewards = []
    for completion, truth in zip(completions, ground_truth):
        pred = extract_boxed_answer(completion)
        if pred is not None and pred.replace(" ", "") == truth.strip().replace(" ", ""):
            rewards.append(1.0)
        else:
            rewards.append(0.0)
    return rewards

def python_unit_test_reward_func(completions: list[str], test_harness: list[str], **kwargs) -> list[float]:
    """Executes generated Python code against unit tests in an isolated subprocess."""
    rewards = []
    for completion, tests in zip(completions, test_harness):
        code_match = re.search(r"```python\n(.*?)```", completion, re.DOTALL)
        if not code_match:
            rewards.append(0.0)
            continue
        candidate_script = f"{code_match.group(1)}\n\n{tests}\n"
        with tempfile.NamedTemporaryFile(mode="w", suffix=".py", delete=True) as tmp:
            tmp.write(candidate_script)
            tmp.flush()
            try:
                proc = subprocess.run(["python3", tmp.name], timeout=3, capture_output=True)
                rewards.append(1.0 if proc.returncode == 0 else 0.0)
            except subprocess.TimeoutExpired:
                rewards.append(0.0)
    return rewards
Section 05 · Deep-Dive Engineering Reference

Study Q&A Bank + Custom Study Note Builder

Explore 10 essential AI training Q&As or add your own custom study cards below (persisted automatically when you export via Download HTML).

Byte-Pair Encoding (BPE) greedily merges frequent byte sequences, creating subtle failure modes: (1) Number Splitting: inconsistently splitting multi-digit numbers (e.g., 381 as 1 token vs. 3829 as 38+29) degrades arithmetic unless digit-level splitting (1–3 digits max per token) is enforced; (2) Leading Whitespace & Indentation: " apple" and "apple" map to different token IDs, and poor Python whitespace merges bloat code token counts; (3) Multilingual Fertility: under-represented scripts require 3–5× more tokens per word ("high token fertility"), shrinking effective context windows; and (4) Special Token Injection: unescaped control tokens like <|im_end|> in raw user text must be sanitized to prevent prompt boundary spoofing.
The Chinchilla scaling law (Hoffmann et al.) demonstrated that for a fixed pre-training FLOP budget C ≈ 6ND, model parameters (N) and training tokens (D) should scale in equal proportion—roughly 20 tokens per parameter (e.g., 1.4T tokens for a 70B model). However, Chinchilla only minimizes training compute. When serving billions of inference queries, smaller models trained far past the Chinchilla frontier—such as 7B–8B models trained on 10T–15T+ tokens ("inference-optimal over-training")—achieve lower total lifecycle cost and lower latency per request.
Catastrophic forgetting occurs when aggressive updates on a narrow downstream dataset overwrite general reasoning, safety boundaries, or instruction-following abilities acquired during pre-training/SFT. Key mitigations include: (1) Data Replay / Mixing: blending 10%–30% diverse general-purpose instruction data into domain SFT batches; (2) Low-Rank Parameter Isolation (LoRA): constraining updates to a low-rank subspace while keeping base weights frozen; (3) Conservative Learning Rates & Cosine Decay: using 1e-5 to 2e-4 with short 1–3 epoch schedules; and (4) KL-Divergence Regularization against the initial SFT checkpoint during post-training.
LoRA decomposes a weight update into two low-rank matrices: W' = W_0 + (lora_alpha / r) · B · A, where A ∈ R^(r×k) and B ∈ R^(d×r). Increasing rank r (typically 8 to 64) increases expressive capacity and trainable parameters, which helps on complex reasoning or multi-task domains, whereas small r (4–16) acts as a strong regularizer on small datasets. lora_alpha scales the magnitude of the adapter update via alpha / r (commonly set to alpha = 2 * r). Applying LoRA to all linear projections (attention q, k, v, o plus MLP gate, up, down) consistently outperforms attention-only LoRA.
Under Goodhart's Law ("when a measure becomes a target, it ceases to be a good measure"), an unconstrained RL policy will exploit flaws in a reward model or verifier—producing degenerate outputs such as excessive verbosity, sycophancy, repetitive formatting, or gibberish that scores high on proxy metrics ("reward hacking"). Adding the reverse-KL penalty -β · D_KL(π_θ || π_ref) anchors the updated policy π_θ near the fluent reference policy π_ref, preserving linguistic coherence and diversity while shifting probability mass toward preferred or verified completions.
Peak activation memory scales linearly with the per-GPU micro-batch size. With gradient accumulation (gradient_accumulation_steps = K), the GPU runs K sequential forward and backward passes on small micro-batches, summing gradients in the gradient buffer (normalized by 1/K) without calling optimizer.step() or freeing/allocating peak activations simultaneously. The optimizer updates weights once every K micro-steps, yielding an exact mathematical equivalent to a global batch size of micro_batch × K × num_GPUs.
Data contamination occurs when benchmark questions or solutions appear inside training data, inflating offline metrics while real-world generalization fails. Defenses include: (1) Pre-training Decontamination: filtering out documents with high 10-to-13-gram overlap or MinHash similarity against public benchmarks; (2) Private Held-Out Canary Suites: maintaining internal, never-published task evaluations refreshed quarterly; and (3) Dynamic / Live Benchmarks: evaluating on newly released problems (e.g., LiveCodeBench, post-cutoff math competitions) and checking min-k% token log-probability anomalies for verbatim memorization.
Post-Training Quantization (PTQ) compresses an already-trained model (e.g., GPTQ, AWQ) using a small calibration set without retraining. Quantization-Aware Training (QAT) inserts fake-quantization rounding nodes during forward passes so weights adapt to low-bit rounding error during training—critical for ultra-low precision (INT4/FP4/ternary). QLoRA freezes base weights in information-theoretically optimal 4-bit NormalFloat (NF4) (matched to normally distributed neural weights) with double quantization (quantizing the quantization constants themselves) while backpropagating through high-precision BF16 LoRA adapters.
In PyTorch's nn.CrossEntropyLoss(ignore_index=-100), any target token labeled -100 contributes zero loss and zero gradient. Masking system instructions and user prompt tokens ensures the model focuses 100% of its gradient capacity on learning to generate high-quality assistant responses rather than wasting capacity memorizing user queries (which are supplied at inference time anyway). In multi-turn conversations, only assistant turns carry active token labels.
Standard PPO requires training a separate, parameter-heavy Critic (Value) model alongside the policy to estimate baseline expected returns at every token. Group Relative Policy Optimization (GRPO) replaces the learned Critic with an empirical Monte Carlo baseline: for each prompt q, it samples a group of G candidate outputs {o_1, ..., o_G}, scores each with a deterministic verifier r_i, and computes each output's normalized advantage relative to its peers: A_i = (r_i - mean(r)) / (std(r) + ε). This slashes RL VRAM overhead by nearly half while excelling on verifiable math and code tasks.

+ Add Custom Study Q&A

Persists in exported HTML file
Section 06 · Knowledge Verification

Interactive 5-Question AI Training Self-Check Quiz

Score: 0 / 5
Question 1 of 5 · SFT Internals · Loss Masking Select an answer

During Supervised Fine-Tuning (SFT) of a causal chat model, why are user prompt and system instruction token labels set to -100?

Engineering Explanation: PyTorch’s CrossEntropyLoss uses ignore_index=-100 by default. Masking prompt tokens ensures 100% of the training loss and gradient signal updates the model on predicting high-quality assistant completions.
Question 2 of 5 · PEFT / QLoRA · Memory Architecture Select an answer

Why does QLoRA (4-bit NF4 base weights + LoRA adapters) reduce GPU VRAM by ~70–80% compared to standard BF16 Full Fine-Tuning?

Engineering Explanation: In Full Fine-Tuning, FP32 AdamW optimizer states (8 bytes/param) + BF16 gradients (2 bytes/param) + BF16 weights (2 bytes/param) consume ~12–16 bytes per parameter. QLoRA shrinks frozen base weights to ~0.55 bytes/param and only holds gradients/optimizer states for the small LoRA adapters.
Question 3 of 5 · Post-Training · DPO vs. PPO Select an answer

How does Direct Preference Optimization (DPO) simplify alignment compared to traditional PPO-based RLHF?

Engineering Explanation: DPO derives a closed-form mapping between optimal RLHF policies and reward functions, turning alignment into a stable supervised binary cross-entropy loss over offline (prompt, chosen, rejected) pairs while still regularizing drift via β.
Question 4 of 5 · Reasoning RL · GRPO Select an answer

In Group Relative Policy Optimization (GRPO) for math and code reasoning, how is the advantage of each candidate rollout computed without a learned Critic network?

Engineering Explanation: GRPO samples a group of G rollouts per prompt, evaluates them with programmatic verifiers (exact math match or unit test pass/fail), and uses the group’s empirical mean and standard deviation as the advantage baseline—eliminating the Critic model.
Question 5 of 5 · Diagnostics · Overfitting vs. Catastrophic Forgetting Select an answer

After 8 epochs of domain SFT, training loss approaches 0.05, validation loss rises sharply, and the model fails basic general reasoning benchmarks it previously passed. What occurred and how should you fix it?

Engineering Explanation: Rising validation loss alongside collapsing general benchmark scores signals both domain overfitting and catastrophic forgetting. Shortening epochs, lowering LR, constraining LoRA rank, and blending general replay data restores generalization.
✓ Done

References and source links

  1. Ouyang et al. (2022). Training language models to follow instructions with human feedback.https://arxiv.org/abs/2203.02155
  2. Hugging Face maintainers. Training a causal language model from scratch.https://huggingface.co/learn/llm-course/chapter7/6
  3. Rafailov et al. (2023). Direct Preference Optimization.https://arxiv.org/abs/2305.18290

Reference list added during the visual refresh on 14 Nov 2026. It is a reading path, not a completed claim-by-claim audit of this imported notebook.