← Learn / Notebooks

Models & toolsTechnicalStudy guide

PEFT and Quantization: LoRA, QLoRA, DoRA, GPTQ and AWQ

Analyze adapter updates, training-state memory, quantized weight storage and calibration-based quantization, then work through merging and a synthetic NF4 teaching lab.

Before you start: You should understand matrix multiplication, gradients, floating-point precision and the purpose of an optimizer.

AI-assisted study notebook · Published 2026-11-16. Separate automated source and visual reviews passed; these checks are not independent expert fact verification. Source audit · Review record.

1. Full fine-tuning memory assumptions

Parameter-efficient fine-tuning changes which parameters receive updates. Quantization changes how values are represented. These decisions interact, but they answer different questions. This guide follows a dense projection from full training through LoRA and DoRA, then examines QLoRA and three inference quantization approaches. The worked example separates algebraic equivalence, representation error and storage accounting.

Start with an explicit accounting model. Assume every trainable parameter has a two-byte model value, a two-byte gradient, a four-byte master value and two four-byte Adam moments. The sum is 16 bytes per parameter. This is a scenario, not a universal property of mixed-precision training: removing a master copy or changing gradient storage changes the sum. Sharding and offloading also change where those bytes reside.

Assumed persistent training states, excluding activations and temporary buffers
StateAssumed precisionBytes per parameter
Model valueBF16 or FP162
GradientBF16 or FP162
Master valueFP324
First Adam momentFP324
Second Adam momentFP324
TotalSpecified scenario16

For 70 billion parameters, the arithmetic gives 1,120 billion bytes: 1,120 decimal GB, or approximately 1,043 GiB. That is an aggregate state estimate, not a measured GPU requirement. It excludes saved activations, temporary workspaces and allocator overhead. An autoregressive serving KV cache is a separate allocation; adding a serving-cache estimate indiscriminately to a training estimate would mix workloads.

Aghajanyan, Gupta and Zettlemoyer studied adaptation in low-dimensional parameterizations. Their results motivate investigating restricted update spaces; they do not prove that every task has a low-rank solution in every layer. Intrinsic dimension of an optimization parameterization and rank of an individual matrix update are different quantities. [2]

Freezing a base matrix removes its trainable gradient and optimizer-state allocations in our accounting. The base values still exist. For a model with N frozen BF16 values and P adapter parameters using the assumed 16-byte training layout, a simplified persistent-state budget is 2N + 16P bytes. This expression deliberately leaves activation memory as a separate term; a small P alone cannot establish whether training fits.

2. LoRA matrices, initialization and merging

Use column-vector inputs. Let W₀ have shape d_out × d_in, A have shape r × d_in and B have shape d_out × r. Standard LoRA freezes W₀ and trains A and B, using h = W₀x + sB(Ax), with s = α/r. Hu et al. initialize A randomly with a Gaussian distribution and B to zero. Their scaling rationale reduces hyperparameter retuning; it is not a universal rank-invariance guarantee. [1]

Forward computation splits input x between frozen W₀ and trainable A followed by B. The base output W₀x and scaled adapter output sB(Ax) meet at addition to produce h. Gradient paths are omitted; only A and B receive parameter updates.

The two outgoing arrows from x carry the same input. The upper branch computes W₀x. The lower branch first reduces the input to r coordinates with A, then maps it to d_out coordinates with B. Its final arrow includes multiplication by s. The addition node combines corresponding output coordinates. All arrows represent forward activations, not optimizer updates. Download editable diagram.

For an upstream gradient g = ∂L/∂h, direct differentiation gives ∂L/∂B = s·g(Ax)ᵀ and ∂L/∂A = s·Bᵀgxᵀ. At B = 0, the initial adapter contribution and A gradient vanish. The B gradient can be nonzero, provided the input projection and upstream gradient support it. Random A does not guarantee a nonzero gradient for every example. If both matrices start at zero, both gradients vanish for this isolated branch.

The same differentiation explains why frozen does not mean disconnected. The input gradient is W₀ᵀg + sAᵀBᵀg. An earlier trainable component can receive a gradient through W₀ even though W₀ itself is not updated. This distinction matters when interpreting memory savings: omitting base-weight gradients is not permission to discard every intermediate needed by backpropagation.

For fixed adapter weights, distributivity gives W_merged = W₀ + sBA and W_merged x = W₀x + sB(Ax). This permits deployment as one dense projection. The paper describes merging before inference. [1] The identity is an algebraic statement. It does not measure milliseconds, include the one-time merge cost, or promise identical floating-point rounding. The lab compares two evaluation orders with a tolerance.

For four square 4096-dimensional projections at rank 16, the full parameter count is 4 × 4096² = 67,108,864. The adapter count is 4 × 16 × (4096 + 4096) = 524,288: exactly 1/128, or 0.78125%, of those selected projection parameters. This denominator excludes every other model component. Increasing the target set changes the total; the example does not establish a model-wide 99% reduction.

3. DoRA magnitude and direction

DoRA separates a weight vector's magnitude from its direction and uses a low-rank update for the directional component. In the paper's column convention, write V = W₀ + sBA and W′ = m ⊙ V/‖V‖c, where each column is normalized and m has one value per column. Initialize m with the column norms of W₀. The learned effective matrix can be materialized for inference. Reported similarities to full fine-tuning are experimental findings, not a guarantee of identical optimization trajectories. [4]

Consider the synthetic column v = (3, 4)ᵀ. Its norm is 5 and its unit direction is (0.6, 0.8)ᵀ. With magnitude m = 5, reconstruction returns (3, 4)ᵀ. Changing only m to 10 gives (6, 8)ᵀ. Changing direction to (0, 1)ᵀ while retaining m = 5 gives (0, 5)ᵀ. These are independently selectable coordinates in the parameterization; optimization need not change them independently in practice.

For the stated orientation, m has d_in entries, adding d_in trainable scalars per projection. A transposed storage convention changes which array axis represents each vector. Check the mathematical vector being normalized before translating a formula into code. Division also requires a nonzero norm: the illustrative DoRA helper rejects a zero column rather than silently inventing a direction. These details make a small example useful for catching shape errors before model integration.

4. QLoRA storage, scale quantization and paging

QLoRA combines a frozen quantized base with trainable adapters. NF4 provides 16 nonuniform reconstruction values informed by a normal-distribution model, including zero. The normality assumption is useful but not universal, and the paper's information-theoretic description should not be read as a guarantee of minimum task error for every tensor. Double quantization compresses block scales; paged optimizers move optimizer-state pages between GPU and host memory. [3]

Storage and arithmetic precision are separate. The authors' integration explanation describes dequantization into BF16 for computation while the base remains frozen. Gradients train the adapters rather than the quantized base values. This is not an assertion that every kernel holds every reconstructed value in registers, or that all implementations use the same compute type. [8]

The scale budget can be derived explicitly. One 32-bit scale per 64 weights costs 32/64 = 0.5 bits per weight. With eight bits per first-level scale and one 32-bit second-level scale per 256 such scales, the overhead becomes 8/64 + 32/(64 × 256) = 0.126953125 bits per weight. Adding the four-bit indices gives 4.126953125, conventionally rounded to 4.127. The block sizes and scale precisions are the paper's configuration. [3]

An equivalent counting exercise uses 16,384 weights: their four-bit indices consume 8,192 bytes; 256 eight-bit first-level scales consume 256 bytes; one second-level FP32 scale consumes four bytes. The sum is 8,452 bytes. This idealized payload omits offsets, padding, tensor metadata and nonquantized components. It is not the size of a Python object or a complete model checkpoint.

The QLoRA paper reports fine-tuning a 65B model on a 48 GB GPU under its experimental setup. That result does not establish that an arbitrary 70B configuration fits. Paging addresses particular memory-pressure events; it cannot guarantee unlimited capacity or eliminate transfer costs. [3]

Merging after quantization introduces another distinction. Let Ŵ be a reconstructed base and D = sBA. The unmerged calculation is Ŵx + Dx. Materializing Ŵ + D preserves that algebra, but storing Q(Ŵ + D) introduces a new quantization operation. In general, dequant(Q(Ŵ + D)) is not Ŵ + D. Compare the final deployment representation, not only the floating-point merge, when checking output changes.

5. LLM.int8(), GPTQ and AWQ

Different mechanisms for reducing inference representation cost
MethodMechanismInterpretation boundary
LLM.int8()Vector-wise INT8 multiplication with an FP16 path for activation outlier dimensions.A mixed-precision computation, not a promise that every operation is INT8. [5]
GPTQWeight quantization using calibration activations and second-order compensation for rounding error.Layer reconstruction on calibration inputs is not a guarantee of downstream quality. [6]
AWQActivation-informed channel scaling before weight-only quantization.The final method protects salient channels through scaling, rather than keeping a mandatory 1% in FP16. [7]

LLM.int8() addresses large activation features through a separate higher-precision contribution. Its paper uses an outlier threshold of 6.0 and reports roughly 0.1% outlier dimensions in its analysis. Those observations concern the studied models; 6.7B parameters is not a universal boundary at which all architectures develop identical outliers. Nor do the experiments establish that all rounding methods necessarily fail below eight bits. [5]

GPTQ targets the layer objective ‖WX − ŴX‖²F, with calibration samples arranged as columns of X. Its damped inverse-Hessian expression is (2XXᵀ + λI)⁻¹. The algorithm quantizes columns and compensates remaining weights, using a Cholesky formulation and block updates for practicality. This is a local reconstruction procedure, not language-model fine-tuning through the complete task loss. [6]

AWQ observes that activation-based selection identifies sensitive channels more effectively than weight magnitude in its experiments. Its scaling search avoids backpropagation and second-order inversion. The approximately 1% finding motivates the method; it is not a universal critical fraction or a deployment prescription. Generalization and acceleration results remain tied to the evaluated workloads and implementations. [7]

A two-coordinate example explains activation awareness without claiming to reproduce either algorithm. Let a weight error be e = (0.1, 0) and input x = (10, 1). The output error e·x is 1. Moving the same weight error to the second coordinate gives e = (0, 0.1) and output error 0.1. Equal weight-error magnitudes can therefore produce different output errors. Calibration describes which inputs make errors consequential.

For any invertible diagonal S, WX = (WS)(S⁻¹X). Quantization breaks this exact equivalence because Q(WS) need not equal Q(W)S. This creates an opportunity to improve the allocation of quantization resolution, but scaling one channel can also alter a shared block range. The identity alone does not select the best scale. Keep method names separate from runtime claims: this notebook provides no hardware benchmark or fixed calibration-set-size recommendation.

6. Synthetic lab: merging, reconstruction and packing

The original standard-library-only lab is illustrative. Its deterministic inputs are synthetic; it makes no model, network, file or shell calls. It computes parameter counts, compares merged and unmerged projections, checks a DoRA reconstruction, and quantizes small blocks using rounded NF4 reconstruction values. It also packs two four-bit indices per byte. No displayed number is presented as measured execution output.

Use W₀ = [[1, 2], [3, 4]], A = [[1, −1]], B = [[0.5], [−0.25]], x = [2, 1] and s = 2. First Ax = [1]; next sB(Ax) = [1, −0.5]; the base result is [4, 10]. The predicted unmerged result is therefore [5, 9.5]. The merged matrix is [[2, 1], [2.5, 4.5]], which produces the same result. These are hand calculations for the specified inputs.

Python example
# Illustrative synthetic excerpt; complete implementation in lab.py.
w0 = [[1.0, 2.0], [3.0, 4.0]]
a = [[1.0, -1.0]]
b = [[0.5], [-0.25]]
x = [2.0, 1.0]
scale = 2.0

base = matvec(w0, x)
adapter = matvec(b, matvec(a, x))
unmerged = [u + scale * v for u, v in zip(base, adapter)]
merged = matvec(merge(w0, a, b, scale), x)
assert all(abs(u - v) < 1e-12 for u, v in zip(unmerged, merged))
assert merged == [5.0, 9.5]

Download complete illustrative lab.py. The full lab chooses the nearest reconstruction value after dividing each block by its absolute maximum. Ties choose the lower index. An all-zero block receives zero scale and zero-value indices. Blocks with an odd number of indices have one unused nibble; the original count is retained for decoding.

The reconstruction constants are rounded from QLoRA Appendix E. [3] This makes the exercise an NF4 teaching approximation, not an exact bitsandbytes implementation. It omits double quantization, BF16 rounding, GPU kernels and production metadata. Its Python lists are not four-bit storage. Only the explicitly packed byte string has two indices per byte, and scales still require separate representation.

The assertions separate useful failure modes: a wrong merge scale changes projection output; an incorrect nibble order breaks index recovery; a lost length adds a spurious value; and mishandled zero blocks produce invalid normalization. Quantization RMSE is computed from the synthetic weights, not copied from the supplied notebook. Do not substitute the original example's RMSE or maximum difference: different inputs and arithmetic produce different values.

For further inspection, change the block size from four to eight while retaining the same values. Compare reconstruction errors and scale counts. Then multiply one input coordinate by ten and compare output error as well as weight RMSE. These exercises ask different questions: payload size measures representation cost, RMSE measures weight reconstruction, and output error measures the effect on selected inputs. None measures language-model accuracy.

7. Explanatory questions

Q1 · InitializationWhy initialize A randomly and B to zero?

This is the initialization in the original LoRA paper. It makes BA zero initially. Our derivative gives ∂L/∂B = s·g(Ax)ᵀ, which can be nonzero, while ∂L/∂A initially vanishes. Setting both matrices to zero blocks both gradients in this branch. [1]

Q2 · MergingWhat does a LoRA merge preserve, and what does it not establish?

Distributivity preserves the real-arithmetic projection for fixed weights: W₀x + sBAx = (W₀ + sBA)x. It removes the explicit adapter branch. It establishes neither measured latency nor bitwise equality across floating-point evaluation orders. Requantizing the merged matrix adds representation error. [1]

Q3 · DoRAWhat additional freedom does DoRA introduce?

A learned magnitude accompanies a normalized direction. In the stated column convention there is one magnitude per column, while a low-rank update parameterizes direction. Our (3, 4) example changes length without changing direction. Similarity to full-training behavior is not equality of optimization trajectories. [4]

Q4 · NF4Why use nonuniform NF4 levels rather than a uniform four-bit grid?

NF4's normal-distribution premise motivates denser reconstruction values near zero. Actual tensors and block normalization can depart from that premise. Neither this motivation nor a small synthetic RMSE proves that NF4 minimizes every model's task error. [3]

Q5 · Double quantizationHow does quantizing scales reduce the idealized overhead?

Under the specified layout, replace 32 bits per 64 weights with eight bits per 64 weights plus 32 bits per 16,384 weights. The resulting overhead is 0.126953125 bits per weight, rather than 0.5. Scales remain present, and complete allocation size includes more than this payload calculation. [3]

Q6 · GPTQWhat does GPTQ compensate after rounding a column?

It adjusts remaining unquantized weights using curvature information from the calibration reconstruction objective. The goal concerns layer outputs on those inputs, not merely independent weight-rounding error. A small reconstruction loss does not prove generalization to arbitrary inputs. [6]

Q7 · AWQWhy examine activations when identifying sensitive weight channels?

Weight errors multiply inputs. Our two-coordinate example produces ten times greater output error when the same perturbation multiplies an input of ten rather than one. AWQ uses activation-informed scaling; its motivating approximately 1% observation is not a universal rule to retain that fraction in FP16. [7]

Q8 · MemoryWhen is 16 bytes per trainable parameter the correct state estimate?

When the explicitly assumed copies exist: two-byte weights, two-byte gradients, a four-byte master value and two four-byte moments. Sum those allocations before multiplying by parameter count. The answer is a state subtotal, not an activation-inclusive or necessarily per-device memory requirement.

8. Scope and source-check disclosure

This study guide concerns dense linear projections and the cited methods' mechanisms. It does not reproduce training experiments, rank selection studies, GPU kernels or deployment benchmarks. Algebraic examples isolate particular effects; they are not estimates of end-to-end quality or speed. The diagram represents one forward projection, and the tables compare mechanisms rather than rank products.

Source-check methodology and limitations

The evidence record distinguishes source-backed method descriptions from original arithmetic, synthetic inputs and code inspection. The bibliography identifies primary research and the public Hugging Face integration article, whose saved document and revision identifiers appear in the source metadata. Generated summaries are not part of the cited evidence. Detailed method claims identify supporting full-text sections rather than relying on abstracts alone.

The cited versions, access date, saved-document identifiers and claim locations are recorded in the accompanying package. Short evidence excerpts identify supporting passages; equations and examples have separate derivation bases. Paper results remain attributed to their authors. Source support is not independent expert verification, and illustrative code is not benchmark evidence.

9. Bibliography

References and source links

  1. Hu et al. (2021), v2. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022.
    https://arxiv.org/pdf/2106.09685
  2. Aghajanyan, Gupta and Zettlemoyer (2021). Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. ACL-IJCNLP.
    https://aclanthology.org/2021.acl-long.568.pdf
  3. Dettmers et al. (2023), v1. QLoRA: Efficient Finetuning of Quantized LLMs.
    https://arxiv.org/pdf/2305.14314
  4. Liu et al. (2024), v3. DoRA: Weight-Decomposed Low-Rank Adaptation.
    https://arxiv.org/html/2402.09353v3
  5. Dettmers et al. (2022), v2. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale.
    https://arxiv.org/html/2208.07339v2
  6. Frantar et al. (2023), v2. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR 2023.
    https://arxiv.org/pdf/2210.17323
  7. Lin et al. (2024), v5. AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration. MLSys 2024.
    https://arxiv.org/html/2306.00978v5
  8. Belkada, Dettmers, Pagnoni, Gugger and Mangrulkar (2023). Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA. Hugging Face.
    https://huggingface.co/blog/4bit-transformers-bitsandbytes