arXiv tutorial explains what universal approximation theorems do and do not prove
A new arXiv paper reviews the mathematics behind neural network expressivity, stressing that approximation guarantees say nothing about training efficiency or generalization.
In plain words
Neural networks are widely used to model complicated patterns, and mathematicians have long proved they can in principle represent almost any function. The paper's key point is that such proofs do not tell engineers how big a network must be, whether training will succeed, or how well it will work on new data.
What was announced
The paper, posted as a cross-listed abstract on arXiv, recounts the classical universal approximation theorems of Cybenko, Funahashi, Hornik and Leshno, which justify why neural networks can model complex nonlinear phenomena. It notes that shallow networks can approximate any continuous function on compact sets, and that Leshno's theorem showed non-polynomial activations are necessary for universality, explaining why ReLU, logistic and hyperbolic tangent activations work while pure polynomials do not.
The author emphasizes the limits of these results: they give no efficiency, training or generalization guarantees, and do not indicate how many neurons a given accuracy requires or whether gradient descent will find suitable parameters. The review also cites Barron's quantitative error bounds for functions with bounded Fourier norm and Montúfar's result that expressivity grows exponentially with depth, measured in linear regions, making deep networks strictly more expressive than shallow ones with equal parameter counts.
What it does not show
- The source is only the paper's abstract, so the full article's arguments, proofs and scope could not be checked.
Sources
Source-support review completed 21 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 21 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.