Bias-only training in ReLU networks behaves like least squares, arXiv study finds
A new arXiv paper argues that training only the biases of wide ReLU networks can be described as a sequence of linear least-squares problems, with transitions tied to neuron activation switches.
In plain words
Neural networks are usually trained by adjusting all their parameters. This paper studies what happens when only the bias terms change while the weights stay fixed at random values. The authors find that, in very wide one-layer ReLU networks, this training behaves almost like a standard linear least-squares problem.
What was announced
The authors of the arXiv preprint study one-hidden-layer ReLU networks whose weights are randomly initialized and fixed, leaving only the biases trainable. They analyse training through what they call activation codes: the binary patterns of which neurons are active for a given input. Because gradients with respect to the biases depend on each input only through its activation code, training can be broken into stages. While the codes stay fixed, they show training reduces to a linear least-squares problem, with transitions whenever neurons change their activation states.
The paper reports that the fraction of neurons switching activation states shrinks as the network widens, so in the infinite-width limit the dynamics approach linear least squares on the initial activation codes. Experiments described by the authors support this: activation switches can noticeably affect training in narrower networks, but their relative frequency and impact on the dynamics diminish as width grows. The findings come from a single preprint whose abstract is available, so the analysis has not been independently verified here.
What it does not show
- Only the abstract of the arXiv preprint was available, so the full paper's methods and proofs were not reviewed and the claims have not been independently confirmed.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.