Silent outputs can hide features rather than forget them, arXiv study finds
A small-model study traces how a neural network can suppress a feature while retaining it, and why a quiet output alone does not prove the feature is gone.
In plain words
Neural networks can stop reporting a feature without erasing it, because features share internal representations. The authors studied a small autoencoder, a model that compresses and reconstructs inputs, where each feature's contribution can be traced directly. They show a hidden signal can be restored by a targeted change, so silence at an output is not proof of forgetting.
What was announced
The paper, posted as an arXiv abstract, examines a small autoencoder so individual feature contributions can be traced directly. With the representation held fixed, restoring a single bias partially recovers detection of a target feature by exposing a signal hidden behind the output's activation threshold. Tests on inputs lacking the target distinguish this genuine recovery from interventions that merely raise output levels, the authors report.
The picture changes when the representation itself is allowed to move. The authors find that training only the other outputs can weaken the target's contribution, even when its own output carries no loss; a conditional theoretical analysis attributes this pressure to interference with retained predictions. Yet more flexible predictors still detect residual target information. The authors conclude that recovery depends on feature geometry and on which parameters are trained, and that a silent output is insufficient evidence of forgetting.
What it does not show
- The evidence is an abstract only; full methods, results and peer review status are not visible in the supplied passage.
- Findings come from a small autoencoder, so the authors' own scope does not establish how the effects transfer to larger models.
Sources
Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 17 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.