← News

Agents & MCPSources: arXiv · 16 Nov 2026

VDL benchmark finds coding agents lose accuracy on semantically equivalent code rewrites

An arXiv paper introduces a probabilistic language for generating semantics-preserving repository variants and reports resolve-rate drops of up to 6.7 percentage points across tested agents.

In plain words

Researchers built a tool that rewrites a codebase in ways that change its appearance but not its behavior, then checked whether AI coding agents still fix the same issues. Success rates fell for the tested agents, showing that reliability depends partly on how code is written, not just what it does.

What was announced

The paper, posted on arXiv, argues that the reliability of agents repairing repository-level issues under superficial code variation is poorly understood. Its authors sample semantically equivalent variants of a repository from an explicitly defined distribution and compare each agent's resolve rate on the variants with its rate on the original. Different distributions expose different weaknesses, such as sensitivity to routine refactoring or to misleading code near the fix location. To express these distributions, the authors developed VDL, a probabilistic programming language whose programs randomly apply semantics-preserving transformations to a repository.

In their evaluation, the authors defined a generic distribution not targeted at any particular failure mode and tested two agentic scaffolds, mini-SWE agent and OpenCode, each paired with one of four models: Claude Opus 4.5, Kimi K2.5, MiniMax M2.5, and Qwen 3.6-27B. Running on SWE-bench Verified and SWE-bench Pro instances, resolve rates dropped by up to 6.7 percentage points, with the decline described as significant in 6 of 16 configurations. The authors also report that no robustness ranking of models holds across scaffolds, a pattern they call jagged.

What it does not show

  • The evidence is the paper's abstract only; full methodology, statistical details, and per-configuration results are not available in the supplied passage.
  • The reported drops come from the authors' own evaluation and have not been independently confirmed.
  • The abstract does not specify which configurations drove the significant drops or how the 6.7 percentage-point maximum was reached.

For security and IT teams

Scope
The findings cover two scaffolds (mini-SWE agent and OpenCode) with four specific models on SWE-bench Verified and SWE-bench Pro instances, not all coding agents.

Sources

Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 17 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.