Critically damped Polyak stepsizes steer gradient descent toward flatter minima
A new arXiv paper proposes an adaptive stepsize rule with the first uniform-in-time guarantees for sharpness and loss reduction on a class of quadratic objectives.
In plain words
Training a neural network involves choosing how large each learning step should be. The authors propose a rule, called critically damped Polyak (CDP) stepsizes, that adjusts step sizes to local curvature so training settles in flatter regions of the loss landscape, which many practitioners associate with better solutions.
What was announced
The paper, posted as an arXiv abstract, describes CDP as a modification of the classical Polyak stepsize rule, multiplying it by a factor of 4 and a mild damping factor. According to the authors, the factor 4 creates equioscillations around the manifold of minimizers, while the damping drives convergence to that manifold. They say this lets the method operate at the edge of stability, adapting step sizes to local curvature, and yields the first uniform-in-time guarantees covering both sharpness and loss reduction for a broad class of locally quadratic objectives.
The authors further report that once the loss is small, the method's projected iterates behave like gradient descent on the logarithm of the top Hessian eigenvalue, paralleling the flatness bias of normalized gradient descent, while its step lengths shrink near the minimizer manifold so it can converge at a finite learning rate. They state the theory covers non-negative losses with locally quadratic Morse-Bott structure, and that empirically CDP reduces sharpness in settings beyond the theory, including nonlinear teacher-student networks and fine-tuning of pre-trained LLMs, with an effect similar to Sharpness-Aware Minimization.
What it does not show
- The evidence is the paper's abstract only; the guarantees and empirical results are authors' claims not yet independently verified, and the abstract is truncated mid-sentence on the empirical results.
Sources
Source-support review completed 21 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 21 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.