← News

Agents & MCPSources: arXiv · 19 Nov 2026

Rubric-based scoring evaluates coding agent patches without running tests

Researchers adapted repository-grounded rubrics into an execution-free measure of how well a code patch meets task requirements, validating the approach against test outcomes on SWE-bench Verified.

In plain words

Coding agents are usually judged by running their code against tests, but tests are often missing and need a working setup for each repository. The researchers instead have an agent study the repository first and write weighted criteria, then a tool-free judge scores a patch by the weighted share of criteria it meets. This score agreed closely with test results in their validation.

What was announced

The authors, in an arXiv abstract, describe adapting agentic rubrics, previously used to choose among candidate patches, into an execution-free evaluation. An agent explores the repository before the change and writes weighted binary criteria; a judge without tools then scores a diff by the weighted share of criteria it satisfies. This avoids two problems with test-based evaluation: tests may be missing or incomplete, and running them requires a working environment for each repository.

To validate the ordering the score produces, the team used 7,291 labeled patches from fifteen systems on SWE-bench Verified. Within the 400 tasks having both resolved and unresolved patches, the mean per-task AUROC was 0.826, rising to 0.838 on tasks not used to tune the rubric-generation prompt. Ranking systems by their failing patches closely matched ranking by resolve rate, with a Spearman correlation of 0.968. Case analyses attributed disagreements to tests that do not match the task, execution-blocking defects the textual assessment misses, and errors in criteria or verdicts. The authors caution the validation does not establish absolute score magnitudes or requirement satisfaction beyond agreement with tests, positioning rubric scores as a complement where tests or execution environments cannot be assumed.

What it does not show

  • The evidence is the paper's abstract only; full method details and results are not available for verification.
  • The reported agreement is with test outcomes on SWE-bench Verified; the authors state it does not establish absolute score magnitudes or requirement satisfaction beyond that agreement.
  • Disagreement cases showed the rubric approach can miss execution-blocking defects and produce errors in criteria or verdicts.

Sources

Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 20 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.