← News

ResearchSources: arXiv · 16 Nov 2026

Researchers link long-range activation patterns in LLMs to persistent latent features

An arXiv paper proposes measuring autocorrelations across transformer layers and argues that slowly decaying tails reflect latent variables that persist over long context spans, with persistence growing in deeper layers.

In plain words

The authors study how activations inside large language models stay correlated across long stretches of text. They argue that hidden features which persist over many tokens leave slow-decaying correlation patterns, and that deeper layers encode more persistent features. They support this with synthetic data, sparse-autoencoder features, and ablation tests.

What was announced

The paper, posted on arXiv, addresses a long-standing claim that LLM hidden layers encode increasingly abstract representations, which the authors say is hard to test without an operational definition of abstraction. Their alternative builds on the idea that a sequence's correlations reflect the structure of the process generating it. They define a layer's autocorrelation function as the average dot product between activations at two context positions separated by a lag, so that latent variables encoded persistently along the context should produce slowly decaying tails in this measure.

In synthetic data with controlled hierarchical latent structure, the authors report that these tails grow across layers because abstract latent variables govern long stretches of the input, and that latent persistence predicts the qualitative structure of the autocorrelations. For real text, they use sparse-autoencoder features as proxies for latent variables and find that feature persistence correlates strongly with autocorrelation magnitude across datasets and models, with persistence times increasing with depth. They also report that ablating the persistent sparse-autoencoder features lowers the autocorrelations. The findings come from the authors' own measurements and have not been independently confirmed.

What it does not show

  • The source is the paper's abstract, so methods, datasets, and model details cannot be verified beyond what the abstract states.

Sources

Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 17 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.