Paper argues agent test suites need per-cell scoring and published coverage
An arXiv paper reports that scoring generated evaluation suites only by overall pass rate can hide real defects, and proposes cell-level scoring with coverage reported openly.
In plain words
Researchers tested whether automatically generated test suites for AI agents actually catch problems. They found a mathematical ceiling: a defect touching only a few passing scenarios can barely shift the overall score, so it goes unnoticed. Scoring each test category separately, and stating how much ground a suite covers, fixes much of this.
What was announced
The authors frame suite generation as a search over a space of requirements, personas and scenario kinds, then argue that reducing the result to a single mean pass rate discards what the search produced. Injecting seven controlled defects into a tau-squared-bench agent across two domains, they derive a ceiling: a defect whose traced footprint spans b baseline-passing scenarios can contribute at most b/N to a suite of N. For four of five traceable defects that ceiling sat at or below a 0.10 detection threshold, meaning no effect within the footprint could trip it.
Defects fell into two regimes. Trace-aligned ones were recoverable by per-cell scoring: the tool-removal defect was caught in full, and a second appeared as a significant improvement, while the suite-level rate moved the wrong way. Trace-orthogonal defects showed behavioural footprints; forced escalation registered zero traced coverage in retail. Diversity helped only where it aligned with defects: a happy-path-only suite found nothing, while a kind-diverse one caught an orchestration defect in 5 of 20 draws. The paper recommends per-cell scoring and publishing coverage as a first-class property.
What it does not show
- Findings rest on one agent in tau-squared-bench across two domains with seven injected defects; the abstract does not establish generalisation to other agents or benchmarks.
- The source is an abstract only, so full method details, baselines and peer review status are not verified here.
For security and IT teams
- Scope
- The evidence covers generated evaluation suites for a tau-squared-bench agent in two domains, not production agent monitoring.
Sources
Source-support review completed 17 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.
Coverage: arXiv: abstract.
Updates and corrections
No corrections so far.
First published 17 Nov 2026
New to the subject? AI from the Ground Up starts from the beginning.