← News

Models & toolsSources: arXiv · 18 Nov 2026

Study shows transformers fade prompt influence as generation grows

Researchers report that a prompt's pull on a language model's output decays roughly like a power law as tokens accumulate, with trained models fading faster than untrained ones.

In plain words

When a language model writes a long answer, its memory of the original instructions seems to weaken. The authors measure this fade and find it follows a predictable mathematical pattern, and they prove that, under stated conditions, the prompt's direct effect shrinks toward zero as the generated text gets longer.

What was announced

Borrowing an analogy from black-hole physics, where only a few conserved quantities survive a collapse, the authors ask how much an autoregressive transformer keeps of its prompt after producing many tokens. They find that when two prompts share the same continuation, the gap between their hidden states shrinks roughly as a power law in the number of generated tokens. A fixed prompt's share of uniform attention fades at a rate near one, while trained models beyond the early layers forget faster, with exponents around 0.55 to 0.7, than untrained networks of the same design, which sit near 0.2 to 0.5.

For a class of finite-depth pre-LN softmax transformers with bounded attention logits and non-degenerate pre-norm activations, the paper proves the prompt offset is bounded by a term scaling with the prompt length times a logarithmic factor, divided by the token count. The bound fixes only the asymptotic shape and, by the authors' own account, says little at the lengths they measured in trained networks. An exactly solvable linear relay model shows the bound cannot be tightened and matches the untrained curves. Separately, cancelling the dilution with a logit bias on the prompt halted the decay early in training, and training appears to suppress relay by roughly a billion tokens, before attention sinks emerge. A toolkit for measuring the exponent in Hugging Face causal language models is publicly available.

What it does not show

  • The evidence is the paper's abstract only; the full article was not reviewed, so the proof conditions and experiments cannot be independently checked.
  • The authors state the theoretical bound is uninformative at the lengths measured in trained networks, so it does not by itself quantify forgetting in practice.
  • The decay rates are reported as approximations from the authors' own measurements; no independent confirmation is provided.

For security and IT teams

Access
A toolkit that measures the decay exponent for any Hugging Face causal language model is publicly available, per the abstract.

Sources

Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 19 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.