Tree-grams: hierarchical n-gram models approximate language with tunable context and capacity
Researchers introduce tree-grams, autoregressive n-gram models built from small tensors, reporting performance beyond n-gram baselines and competitive with transformers on their tests.
In plain words
The authors propose a new kind of simple statistical language model, called tree-grams, where two dials control how much context the model sees and how large it is. Because these dials are explicit, the model can serve as a controlled stand-in for studying how language models behave. In their experiments it beat other n-gram baselines and stayed competitive with transformer models.
What was announced
Tree-grams construct the conditional probability of the next token from a hierarchical composition of order-2 and order-3 tensors, according to the paper's abstract. The context length n sets the maximal window the model can condition on, while a rank hyperparameter d controls capacity, producing what the authors describe as an interpretable and systematically scalable family of language approximators. The reported results show tree-grams outperforming existing n-gram baselines while remaining competitive with deep transformer-based language models.
The authors position these models as controlled approximations for research use. They report that hyperparameter transfer under muP becomes more robust when narrower models are tuned on simpler data generated by tree-grams with moderate n and d, and that entropy and the effective range of correlations predict the performance gap after fine-tuning. They also observe emergent sparsity when routing across candidate topologies, allowing pruning of up to 95 percent of topologies without significant performance change. These claims come from the abstract alone; the full paper was not reviewed.
What it does not show
- The evidence is limited to the paper's abstract; experimental details, benchmarks and evaluation settings could not be verified.
- Claims of competitiveness with transformers and the 95 percent pruning figure are author-reported and not independently confirmed.
Sources
Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 19 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.