Paper 2607.08964
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading
- Published
- Jul 2026
- Research lab
- Independent
- Citations
- 3
- GitHub
- 467 stars
01 In brief
Summary
Long-Horizon-Terminal-Bench is a new benchmark of 46 containerized terminal tasks across nine categories, designed to test AI agents on long-horizon workflows.
Unlike prior benchmarks that use binary pass/fail grading, it decomposes each task into graded subtasks, providing dense partial-credit rewards.
Tasks require an average of 239 episodes, 9.8M tokens, and 88.9 minutes per run.
Across 17 frontier models, the strongest (Grok 4.5) achieves only 28.3% pass@1 at a 0.95 reward threshold, with a mean pass rate of 6.4%.
Dense rewards reveal that 62.8% of runs make partial progress, and 79% of unresolved runs end in timeouts, indicating that the main bottleneck is sustained long-horizon completion rather than local reasoning.
The benchmark also exposes false finishes, where agents stop prematurely with high reward, highlighting weak self-verification.
Cost analysis shows Grok 4.5 is cost-efficient at ~$11 per task, while other models vary from $3.6 to $26.
The authors release the benchmark to support future progress on robust long-horizon terminal agents.
02 From the paper
Abstract
AI agents have become capable of autonomously completing short, well-specified tasks. However, existing terminal benchmarks largely focus on simple problems that finish within minutes and are evaluated only by their final outcome. This setup overlooks intermediate progress and partial solutions, yielding sparse reward signals and an incomplete picture of agent capability. We introduce Long-Horizon-Terminal-Bench, a terminal benchmark of 46 long-horizon tasks spanning nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, but is further decomposed into fine-grained graded subtasks. This design enables dense intermediate rewards and partial credit, allowing evaluation to capture not only whether an agent reaches the final goal, but also how far it progresses on open-ended workflows. Tasks in Long-Horizon-Terminal-Bench typically require hundreds of episodes and minutes to hours of execution, stressing long-horizon planning, long-context management, and iterative debugging rather than one-shot problem solving. We evaluate 15 frontier models and find that agents consume on average 9.9M tokens per task, with roughly 231 episodes and 85.3 minutes of execution time per run, making Long-Horizon-Terminal-Bench more demanding than prior terminal-based benchmarks. Even the strongest tested model achieves 15.2% pass@1 at a partial-reward threshold of 0.95 and 10.9% at a perfect-reward threshold of 1.0, while the mean pass rate across models is 4.3% and 1.7% under the two thresholds, respectively. These results reveal headroom for improvement. We further analyze failure modes and error patterns, and release Long-Horizon-Terminal-Bench to support future progress on long-horizon terminal agents.