Researchers propose LLM Complexity, a token-based measure of task difficulty
An arXiv paper argues token length is a poor stand-in for classical time complexity and introduces a metric based on the accuracy-token curve.
In plain words
The authors study how hard problems are for large language models, which are AI systems that generate answers token by token. They argue that counting generated tokens, a natural proxy for inference-time compute, does not track classical time complexity. Their new metric instead measures the area under the accuracy-token curve, and models rank task difficulty consistently under it.
What was announced
The paper, posted to arXiv, treats LLMs as computational systems with an input, inference-time compute, and output stage, and identifies the number of generated tokens as a directly observable proxy for that compute. The authors report that the tempting analogy between token length and classical time complexity fails: they say they disprove the hypothesis that inference-time token usage aligns monotonically with classical time complexity, attributing the divergence to how LLMs allocate computation during inference.
As an alternative, the authors introduce LLM Complexity, a metric that quantifies problem difficulty from the LLM perspective by characterizing the area bounded above by the accuracy-token curve. They find that diverse LLMs induce highly consistent task rankings under this metric, suggesting it captures a stable notion of difficulty across models. Using that property, they build an easy-to-hard curriculum for supervised fine-tuning that they report improves accuracy while reducing token consumption. They also state that the metric generalizes to real-world tasks and yields more stable cross-model correlations than human-annotated difficulty labels. The claims come from the paper's abstract; the underlying experiments are not independently verified here.
What it does not show
- The evidence is the paper's abstract only; experimental details, datasets, and model lists are not visible, and no independent confirmation of the reported results is available.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.