Rational Sparse Autoencoder replaces fixed activations with a trainable function
A new arXiv paper proposes the Rational Sparse Autoencoder, which learns its encoder activation instead of hard-coding ReLU, JumpReLU, or TopK, reporting better reconstruction at matched sparsity.
In plain words
Sparse autoencoders are tools that break a language model's internal activity into simpler, more readable parts. The authors replace the fixed mathematical switch inside these tools with a small learnable function, so the tool can tune how it decides which parts to keep. They report clearer reconstructions at the same level of sparsity, using only 20 extra parameters per autoencoder.
What was announced
Sparse autoencoders are standard tools for mechanistic interpretability, but existing families fix the encoder nonlinearity to choices such as ReLU, JumpReLU, or TopK, which hard-codes a sparsity mechanism and can distort the reconstruction-versus-sparsity trade-off. The Rational Sparse Autoencoder (RSAE), introduced by the paper's authors on arXiv, replaces that fixed activation with a trainable rational function. The authors state a theoretical reconstruction advantage at matched expected sparsity, guaranteeing non-inferiority to ReLU and, under stated assumptions, provable improvements over JumpReLU and TopK.
The model is built through a two-stage pipeline: an initialization that copies pre-trained baseline SAE weights and calibrates scale parameters and rational coefficients, followed by fine-tuning under a sparsity-regularized reconstruction objective that holds the expected sparsity at the teacher's value. On residual-stream activations of three open-weight language models, across all three baseline activation families, the authors report improved reconstruction fidelity after fine-tuning at the same sparsity level, consistent across host models and the full range of baseline sparsity tested. The upgrade adds only 20 scalar parameters per autoencoder, and one training run takes roughly 4 to 25 minutes on a single consumer GPU.
What it does not show
- The evidence is the paper's abstract only; the theoretical claims and reported gains have not been independently verified, and the stated assumptions behind the JumpReLU and TopK improvements are not detailed in the excerpt.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.