Paper 2606.28733
Agentic Abstention: Do Agents Know When to Stop Instead of Act?
- Published
- Jun 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- 42 stars
01 In brief
Summary
The paper introduces Agentic Abstention, the problem of deciding when an LLM agent should stop acting and abstain rather than continue interacting with an environment.
Unlike single-turn LLM abstention, this is a sequential decision problem where agents can answer, abstain, or act at each turn.
The authors construct a benchmark of over 28,000 tasks across web shopping (WebShop), terminal environments (Terminal-Bench 2.0), and interactive QA (AbstentionBench), evaluating 13 LLM-as-agent systems and 2 scaffolds.
Results show agents often abstain too late or not at all, especially when tasks initially appear feasible but become infeasible after interaction.
Timely abstention is a major challenge, with most models achieving under 40% timely recall.
Model scale, reasoning, and scaffolding affect abstention differently; larger models sometimes perform worse on timely abstention.
The proposed method, CONVOLVE, distills interaction trajectories into reusable stopping rules added to agent context, improving timely abstention without parameter updates.
On WebShop, CONVOLVE raises Llama-3.3-70B's timely recall from 26.7% to 57.4% and overall recall from 83.2% to 100.0%.
02 From the paper
Abstract
LLM agents are expected to act over multiple turns, using search, browsing interfaces, and terminal tools to complete user goals. Yet not every goal is well specified or achievable in the available environment. In such cases, a reliable agent should recognize that further interaction is unlikely to help and abstain from additional tool calls. We define Agentic Abstention, the problem of deciding when an agent should stop acting under uncertainty. Unlike standard LLM abstention, which is usually evaluated as a single-turn answer-or-abstain decision, agentic abstention is a sequential decision problem: an agent can answer, abstain, or gather more information at each turn, and the need to abstain may only become clear after interacting with the environment. We study this problem across web shopping, terminal environments, and question answering, evaluating 13 LLM-as-agent systems and 2 agent scaffolds on more than 28,000 tasks. Our results show that the main challenge is not only whether agents can abstain, but also when they abstain. Some agents never abstain when they should, while others do so only after many unnecessary interactions. This gap is especially large on tasks where the instruction appears feasible until the environment reveals otherwise (e.g., no valid result matches the instruction). We further find that model scale, reasoning, and agent scaffolding affect abstention in different ways, where larger or more capable models sometimes perform worse at timely abstention. Finally, we introduce CONVOLVE, a context engineering method for improving agentic abstention that distills full interaction trajectories into reusable stopping rules. On WebShop, CONVOLVE substantially improves timely abstention without updating model parameters, raising Llama-3.3-70B's timely recall rate from 26.7 to 57.4. Our dataset and code are available at https://lhannnn.github.io/agentic-abstention