ICA Lens finds interpretable language-model directions by seeking non-Gaussian projections
The arXiv paper proposes skipping sparse autoencoder training and instead searching activation space directly for directions whose projections are statistically non-Gaussian.
In plain words
Language models process text through internal activation values, and researchers try to find directions in that space that correspond to understandable concepts. The authors observe that such interpretable directions tend to have non-Gaussian projection distributions, meaning their values are not spread in the familiar bell-curve pattern. ICA Lens searches for those directions directly, without training a sparse dictionary first.
What was announced
The paper, whose abstract is available on arXiv, reports that projection distributions of natural-text residual-stream activations along sparse autoencoder decoder directions are markedly more non-Gaussian than along random directions, with most contexts forming a concentrated background and recurring patterns occupying one tail. On that basis the authors introduce ICA Lens, a workflow for discovering, annotating, and evaluating non-Gaussian directions in residual-stream activations without training another sparse dictionary.
Testing across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 9B Base, the authors state that the recovered directions capture coherent lexical, syntactic, and semantic patterns under human and LLM-based evaluation, and that non-Gaussianity correlates positively with human-label confidence and automated interpretability. They also report the coordinates are competitive with SAEs on interpretability, sparse reconstruction, concept prediction, and causal interventions, while fitting in minutes per layer and using 7-43 times less memory. These claims come from the paper's own abstract and have not been independently verified.
What it does not show
- The evidence is the paper's abstract only; full methods, experimental detail, and peer review status are not covered.
- All results, including the memory and speed comparisons, are the authors' own claims and have not been independently confirmed.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.