← News

Agents & MCPSources: arXiv · 18 Nov 2026

ANCHOR benchmark shows coding agents often meet functional goals but miss developer constraints

The benchmark's authors report that even the strongest of 19 tested agent configurations fully satisfied every scored constraint on under a fifth of tasks.

In plain words

Coding agents can fix code correctly while still breaking rules developers care about, such as how to work, which tools to use, and what to report. The ANCHOR benchmark, described in a new arXiv paper, tests whether agents follow such constraints. Its authors found the best system still fell short on many tasks.

What was announced

The paper introduces ANCHOR (Assessing Naturalistic Constraint Hierarchies and Ordering in Repository-Grounded Agentic Coding), a benchmark of naturalistic, multilingual repository tasks spanning eight language groups. It pairs a taxonomy of 10 categories and 78 leaf constraint types distilled from real developer interactions with an automated pipeline that composes task-grounded constraints into typed AND/THEN/PRIORITY relations, and evaluates both the code diff and the execution trajectory using rule-based and LLM judges.

Across 19 agent configurations built on 2 scaffolds and 10 foundation models over 358 tasks, the authors report that Claude Opus 4.8 under Cline reached 70.1% atomic compliance and 63.7% relation compliance, while 19.0% of tasks passed every scored checklist item under the strict all-item FCCR criterion. Failures concentrated in workflow ordering, tool-use discipline, and reporting. A five-engineer audit of constraints from 50 instances reported 94% feasibility, 94% repository consistency, and 96% realism. In the primary paired set of 14 configurations, the mean change in task completion after adding constraints was +0.7 percentage points, with individual changes in both directions.

What it does not show

  • The evidence is the paper's abstract only; full methodology, task details, and caveats are not available for verification.
  • All results, including the audit scores and compliance figures, are the authors' own measurements and have not been independently confirmed.
  • The abstract does not name the other 18 agent configurations or their scores, so relative rankings beyond the strongest system are unknown.

For security and IT teams

Scope
The benchmark targets repository-grounded agentic coding and evaluates both code changes and execution trajectories, not just final correctness.

Sources

Source-support review completed 19 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 19 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.