← News

ResearchSources: arXiv · 20 Nov 2026

Complete-SVD paper proposes logit-matching low-rank compression for LLMs

An arXiv abstract describes a post-training compression method that minimizes the distance between original and compressed model logits, using a weighted SVD with an estimated output-side factor.

In plain words

The authors present Complete-SVD, a way to shrink large language models after training by replacing weight matrices with smaller low-rank approximations. Instead of only approximating the weights, the method tries to keep the model's output scores (logits) close to the uncompressed model's, treating all changes in those scores equally. The authors report lower perplexity than prior output-aware methods in their tests.

What was announced

The method builds on output-aware compression, which measures error at the model output rather than only at each compressed layer. Earlier approaches weight logit changes by the observed next token or by the model's predicted probabilities; Complete-SVD instead minimizes the L2 distance between the logits of the original and compressed models, weighting all logit changes equally. Linearizing the network around the original weights and applying a Kronecker-factored approximation reduces this objective to a weighted SVD with a closed-form solution.

The output-side factor of that solution is intractable to compute exactly, so the authors estimate it without bias from random probes, at a cost of K backward passes per sequence, independent of vocabulary size and sequence length. In their experiments with uniform ranks and WikiText-2 calibration, Complete-SVD achieves the lowest C4 and PTB perplexity in every tested setting, including ratio 0.7 under LEMS rank allocation, and improves IFEval instruction-following accuracy over the strongest prior output-aware methods. A variant using KL divergence, KL-SVD, lowers calibration perplexity in most settings and raises downstream accuracy, but raises cross-corpus perplexity.

What it does not show

  • The evidence is the paper's abstract only; the full methodology, experimental setup and peer review status are not available in the supplied passages.
  • All reported perplexity and accuracy results are the authors' own claims and have not been independently confirmed.

Sources

Source-support review completed 21 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 21 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.