← News

Agents & MCPSources: arXiv · 19 Nov 2026

AgentAbstain benchmark finds LLM agents often act when they should hold back

A new arXiv paper introduces the first systematic framework for testing whether tool-using LLM agents know when not to act, reporting that even the best of 17 frontier models reached only 59.5% paired accuracy.

In plain words

LLM agents are AI systems that carry out tasks by calling tools, and they can take actions that are hard to undo. The AgentAbstain authors built a benchmark of paired tasks—one where the agent should act, and a near-identical variant where it should abstain—to measure this judgment. Across 17 frontier models, the best agent scored 59.5%, and abstention skill did not track general task-solving ability.

What was announced

The paper, AgentAbstain, argues that existing evaluations mostly reward task success while ignoring whether agents recognize when not to act, a gap the authors say creates risks of unintended and irreversible actions under ambiguity, conflicting constraints, or tool failures. The benchmark contains 263 paired tasks across 42 executable sandbox environments, with each pair consisting of a should-act task and a should-abstain variant created by perturbing the instruction, tool, or environment state. The scenarios follow an agent-native taxonomy of eight abstention situations spanning pre-execution reasoning and runtime discovery.

To keep the benchmark fresh and resist data contamination, the authors propose AbstainGen, an automated pipeline that synthesizes sandbox environments and paired tasks end-to-end, validated by deterministic replay and semantic LLM judges; independent annotators rated 94-98% of sampled tasks as well-designed. Testing 17 frontier LLMs across 4 agent harnesses, the best agent, Gemini 3.1 Pro, achieved only 59.5% paired accuracy, meaning it was correct on both the act and abstain sides of each pair. The authors report that abstention capability is largely independent of general task-solving capability, so scaling task-solving alone will not close the gap. They also identify failure modes such as post-hoc abstention, where agents execute irreversible actions before recognizing abstention triggers. Code and dataset are open-sourced at agentabstain.github.io.

What it does not show

  • The evidence is the paper's abstract only; full methodology, baselines, and caveats cannot be verified from these passages.
  • Results are self-reported by the paper's authors and have not been independently confirmed.
  • The abstract does not define the eight abstention scenarios in detail or list all models and harnesses tested.

For security and IT teams

Scope
The benchmark covers tool-using LLM agents in executable sandbox environments, not general chatbot behavior.

Sources

Source-support review completed 20 Nov 2026. This story was drafted with AI assistance and checked against retained source passages. Automated checking is not independent verification.

Coverage: arXiv: abstract.

Updates and corrections

No corrections so far.

First published 20 Nov 2026

New to the subject? AI from the Ground Up starts from the beginning.