← News

Models & toolsSources: arXiv · 20 Nov 2026

Canon layers cut pretraining token needs by about 30% in 64-run study

A large-scale pretraining study finds Canon layers consistently help Transformers, while standard benchmarks can hide basic weaknesses in linear models.

In plain words

Researchers trained 64 language models from scratch to test whether architecture choices matter at scale. Adding Canon layers, which mix information between nearby tokens, let Transformers reach the same accuracy with roughly 30% fewer training tokens. The study also found that benchmark scores can hide failures in simple context-based tasks.

What was announced

The authors ran 64 from-scratch pretraining runs covering Transformers and three linear-model families at 1.3B to 8B parameters, using 1-2T Nemotron-CC tokens, at a reported cost of roughly 100k USD per 8B run. At academic scale (1.3B parameters, 100B tokens), architectural differences largely vanished under careful tuning, but became measurable at 1-2T tokens. Canon layers, which add local token mixing, consistently improved Transformers, saving about 30% of pretraining tokens for the same MMLU accuracy at 8B.

The study also introduced GLA5 and GDN2, variants of gated linear attention and Gated DeltaNet with low-rank output gates. With Canon, gated linear attention matched Gated DeltaNet and outperformed Mamba2 at 8B/1T, suggesting that extra machinery in those models may mainly mix neighboring tokens. Yet even at 8B/1T, linear models that matched or beat Transformers on standard benchmarks could not reliably retrieve information from a short 500-token context, and no model reliably beat random guessing on a simple two-hop question. The authors conclude that small synthetic playgrounds below 100M parameters predicted these differences, and recommend including in-context reasoning examples in pretraining data early rather than reserving them for later stages.

What it does not show

  • The evidence is limited to the paper's abstract, so full experimental details and results are not visible here.
  • The findings come from the authors' own runs and have not been independently confirmed.
  • The two-hop and 500-token retrieval weaknesses are reported at 8B/1T scale in the abstract; behavior at other scales is not established in this excerpt.

For security and IT teams

Scope
The reported Canon benefit is specific to 1-2T token pretraining; at 1.3B parameters and 100B tokens, architectural differences largely disappeared under careful tuning.

Sources

Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 20 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.