BuggyMLE injects bugs into ML repositories to grade agent difficulty
Researchers behind the BuggyMLE paper propose a difficulty metric and a bug-injection framework that turn working machine learning repos into thousands of harder, graded agent tasks.
In plain words
The authors built a way to make machine learning engineering tests for AI agents harder in a controlled, measurable way. They inject bugs into real, working ML code repositories, creating thousands of task variants sorted by difficulty. They also show that training on these tasks can improve a model's coding performance.
What was announced
The paper argues that existing machine learning engineering benchmarks hold agents back by restricting them to single-file edits without full repository or shell access, while common repository-level software engineering benchmarks have become saturated. The authors introduce the Code Complexity Index, a difficulty metric built from ten structural features such as repository scale, dependency depth, and patch size. They report that CCI strongly predicts average solve rates across eight public coding benchmarks, with patch size and complexity among the top predictors.
Guided by that metric, the team created BuggyMLE, a framework in which agents inject controlled mutations into working ML repositories. From 222 source repositories they synthesized over 6,600 bug-injected variants totalling nearly 260,000 lines of injected code. One product, BuggyMLE-Bench-Lite, is described as the first difficulty-graded, repository-level variant of MLE-Bench Lite; across 14 evaluated models, the top solve rate reportedly falls from 95% on the easiest tier to 9% on the hardest. The authors also built a three-level training curriculum over 200 additional repositories and fine-tuned a Qwen3.5-9B model that they say rivals frontier models on the benchmark, while raising its SWE-bench Lite resolve rate from 24% to 33% and its LiveCodeBench v6 pass@1 from 69% to 71%.
What it does not show
- All figures come from the paper's own abstract; no independent evaluation or confirmation is reflected here.
- The source is an abstract only, so methodology details, evaluation setup, and model availability are not covered.
Sources
Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 20 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.