← News

ResearchSources: arXiv · 16 Nov 2026

Researchers propose framework to trace how LLMs process benchmarks internally

A task-centric evaluation framework combines attention, activation and representation signals to characterise internal model processing across benchmarks, moving beyond task-level scores.

In plain words

Large language models are usually judged only by their scores on tests, which shows what they achieve but not how. The authors built a framework that inspects signals inside the model, such as attention and activation patterns, to characterise the processing behind a score. They say this can reveal failure modes that scores hide and help pick better benchmarks and models.

What was announced

Current evaluations of large language models report results at the dataset or task level, which the paper's authors say reveals what a model achieves but offers little insight into the internal processing behind that performance. Their proposed framework is task-centric and combines attention-based and activation-based signals with representation-based measures. It is designed to characterise benchmarks along a continuum from retrieval-dominant to reasoning-dominant, and to surface failure modes that task-level scores conceal.

The team evaluated 16 LLMs across 24 diverse benchmarks and ran targeted internal-state analyses on representative models and tasks. They report systematic differences in processing between knowledge-centric and reasoning-centric tasks. According to the authors, the approach shifts evaluation from asking what score a model achieves toward characterising the processing patterns associated with that score, which they suggest could support more targeted benchmark selection and more informed model choice. The code has been open sourced on GitHub.

What it does not show

  • The source is an abstract only; detailed results, methods and any peer review are not available in the supplied evidence.
  • The reported differences in processing and the claimed benefits for benchmark selection come from the authors themselves and are not independently confirmed.

Sources

Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 17 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.