← News

ResearchSources: arXiv · 18 Nov 2026

LLMs that hand-write model weights do better with lookup tables than Transformers, paper reports

An arXiv abstract reports that when language models write weights without gradient training, they build sparse tables that beat Transformers they also wrote, with far fewer parameters.

In plain words

Usually AI models are trained by adjusting weights step by step with gradients. In this study, large language models instead wrote the weights directly by hand. Given that freedom, they built simple lookup tables that answered questions more accurately than the Transformers they also wrote, using far fewer parameters.

What was announced

The arXiv abstract argues that Transformers were chosen because they suit gradient descent and GPU hardware, conditions that do not apply when a language model writes weights directly. The authors gave three frontier LLMs ten labeled examples and a sandbox, mainly on multiple-choice questions, asking them either to fill a fixed architecture or design one. All three chose to compile plain-text facts into a sparse table queried by a fixed procedure, reaching up to 54.4% test accuracy. An LLM-designed model outperformed the most accurate Transformer the same LLM wrote while using about 1,800x fewer parameters.

The abstract attributes the tables' appeal to inspectability: each weight's meaning can be stated, a wrong weight can be found and rewritten, and each wrong answer points to one removable row that affects about one other answer. From this the authors propose an associative-memory-based model up to 17x faster on GPUs than a same-sized Transformer. They note limits: pretrained LLMs remain more accurate, and the claim concerns which structure to build when writing weights, not overall accuracy. Testing covered pretrained weights, other benchmarks, and generative-answer tasks.

What it does not show

  • The evidence is an abstract only; full methods, benchmarks and peer review are not visible in the supplied passages.
  • All reported figures come from the authors' own experiments and are not independently confirmed.
  • The tasks were mainly multiple-choice, so results may not extend to other task types.

For security and IT teams

Scope
The reported comparison covers LLM-written weights for multiple-choice-style tasks; the authors state pretrained LLMs remain more accurate, so the finding concerns architecture choice, not replacing pretrained models.

Sources

Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 19 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.