Study finds LLM code optimizers lean on pretrained knowledge, not feedback
A controlled study of LLM agents in GPU kernel optimization reports that model behavior is driven mainly by pretrained priors, with explicit instructions and some feedback loops offering little benefit.
In plain words
Researchers tested whether AI coding agents improve code by using feedback or by relying on what they already learned during training. Across three controlled experiments, the models largely ignored new information, such as input sizes, and performed worse on unusual cases. Results improved only when the model worked with a language it knew well, like CUDA.
What was announced
The arXiv paper, posted as a version 2 replacement with an abstract-level scope, describes three controlled experiments examining the common propose-evaluate-revise loop used by LLM optimization agents. The authors report that in pure black-box optimization, LLMs behave as greedy optimizers. In zero-shot kernel generation, providing explicit input-size information had no measurable effect: models converged to the same kernel parameters regardless of size or temperature, as though the size instruction were invisible. Performance also sharply degraded when optimizing kernels for uncommon sizes, regardless of the language used.
In the third experiment, on feedback-loop kernel optimization, the authors report that CUDA code improved monotonically under iterative feedback, while TVM IR actively degraded, which they attribute to models operating with a low-density language. The paper concludes that LLMs in code optimization depend heavily on pretrained priors rather than provided feedback or agentic structure. The findings come from the authors' own experiments and have not been independently confirmed; the abstract does not specify which models, workloads or hardware were tested.
What it does not show
- Evidence is limited to the paper's abstract; full experimental details, model choices and hardware are not visible in the supplied passages.
- The findings are the authors' own reported results and are not independently confirmed.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.