Paper 2606.22388
PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems
- Published
- Jun 2026
- Research lab
- Independent
- Citations
- 4
- GitHub
- 38 stars
01 In brief
Summary
PlanBench-XL is a new interactive benchmark for evaluating long-horizon planning of LLM tool-use agents in large-scale tool ecosystems.
It consists of 327 retail tasks over 1,665 tools, requiring agents to iteratively retrieve tools, infer implicit sub-goals, and adapt to dynamic environments.
The benchmark features a retrieval-mediated environment with bi-directional exploration (forward and backward anticipation) and an optional blocking mechanism that simulates real-world unpredictability through explicit failures, implicit failures, and semantically misleading tools.
Experiments on ten leading LLMs show that massive-tool planning remains challenging: GPT-5.4 achieves 51.90% accuracy in block-free settings but drops to 11.36% under the most severe blocking.
Key findings include: agents are especially vulnerable to silent failures and longer recovery paths; broad exploration correlates with success (Pearson r=0.902) but is insufficient without precise exploitation; most failures occur after partial progress due to tool-selection errors rather than retrieval failures; and models exhibit distinct termination policies (GPT surrenders, DeepSeek/Llama commit wrong values, Gemini keeps searching).
The benchmark provides a scalable framework for diagnosing agentic planning failures and highlights the need for robust adaptive planning in large, imperfect tool environments.
02 From the paper
Abstract
LLM agents increasingly operate in large tool ecosystems, where real-world tasks require discovering relevant tools, inferring implicit sub-goals, and adapting to dynamic environments over long horizons. However, existing benchmarks rarely evaluate planning under retrieval-limited tool visibility. To address this gap, we introduce PlanBench-XL, an interactive benchmark of 327 retail tasks over 1,665 tools that tests whether agents can iteratively retrieve usable tools, invoke them to uncover intermediate evidence for subsequent calls toward the final goal. PlanBench-XL further features an optional blocking mechanism that simulates real-world unpredictability through missing, failing, or distracting tool functions, forcing agents to detect disrupted paths and adapt at runtime. Experiments on ten leading LLMs show that massive-tool planning remains challenging: while GPT-5.4 achieves 51.90% accuracy in block-free settings, it collapses to 11.36% under the most severe blocking condition. Further analysis shows that agents are especially vulnerable when failures lack explicit error signals or when recovery requires longer alternative tool-use paths. These results establish PlanBench-XL as a testbed for diagnosing agentic planning failures and highlight the need for robust adaptive planning in long-horizon tasks with large, imperfect tool environments.