← News

Agents & MCPSources: arXiv · 16 Nov 2026

Paper argues agent test suites need per-cell scoring and published coverage

An arXiv paper reports that scoring generated evaluation suites only by overall pass rate can hide real defects, and proposes cell-level scoring with coverage reported openly.

In plain words

Researchers tested whether automatically generated test suites for AI agents actually catch problems. They found a mathematical ceiling: a defect touching only a few passing scenarios can barely shift the overall score, so it goes unnoticed. Scoring each test category separately, and stating how much ground a suite covers, fixes much of this.

What was announced

The authors frame suite generation as a search over a space of requirements, personas and scenario kinds, then argue that reducing the result to a single mean pass rate discards what the search produced. Injecting seven controlled defects into a tau-squared-bench agent across two domains, they derive a ceiling: a defect whose traced footprint spans b baseline-passing scenarios can contribute at most b/N to a suite of N. For four of five traceable defects that ceiling sat at or below a 0.10 detection threshold, meaning no effect within the footprint could trip it.

Defects fell into two regimes. Trace-aligned ones were recoverable by per-cell scoring: the tool-removal defect was caught in full, and a second appeared as a significant improvement, while the suite-level rate moved the wrong way. Trace-orthogonal defects showed behavioural footprints; forced escalation registered zero traced coverage in retail. Diversity helped only where it aligned with defects: a happy-path-only suite found nothing, while a kind-diverse one caught an orchestration defect in 5 of 20 draws. The paper recommends per-cell scoring and publishing coverage as a first-class property.

What it does not show

  • Findings rest on one agent in tau-squared-bench across two domains with seven injected defects; the abstract does not establish generalisation to other agents or benchmarks.
  • The source is an abstract only, so full method details, baselines and peer review status are not verified here.

For security and IT teams

Scope
The evidence covers generated evaluation suites for a tau-squared-bench agent in two domains, not production agent monitoring.

Sources

Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 17 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.