Paper 2603.13594
EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings
- Published
- Mar 2026
- Research lab
- Independent
- Citations
- 6
- GitHub
- 115 stars
01 In brief
Summary
ENTERPRISEOPS-GYM is a benchmark for evaluating LLM agents in realistic enterprise settings, featuring a containerized sandbox with 164 database tables and 512 tools across eight domains (CSM, HR, ITSM, Email, Calendar, Teams, Drive, Hybrid).
It includes 1,150 expert-curated tasks, 30 of which are infeasible to test refusal behavior, with SQL-based verifiers checking goal completion, integrity, policy compliance, and side effects.
Evaluation of 14 frontier models shows the best (Claude Opus 4.5) achieves only 37.4% success, with open-source models lagging.
Performance degrades with task horizon, and models often fail to refuse infeasible tasks (best 53.9%).
Adding distractor tools has negligible impact, while providing human-authored plans improves performance by 14–35 percentage points, indicating strategic planning is the primary bottleneck.
Multi-agent orchestration does not close this gap.
The authors conclude current agents are not ready for autonomous enterprise deployment and release the benchmark to advance research in stateful agentic planning and policy compliance.
02 From the paper
Abstract
Large language models are shifting from passive information providers to active agents intended for complex workflows. However, their deployment as reliable AI workers in enterprise is stalled by benchmarks that fail to capture the intricacies of professional environments, specifically, the need for long-horizon planning amidst persistent state changes and strict access protocols. In this work, we introduce EnterpriseOps-Gym, a benchmark designed to evaluate agentic planning in realistic enterprise settings. Specifically, EnterpriseOps-Gym features a containerized sandbox with 164 database tables and 512 functional tools to mimic real-world search friction. Within this environment, agents are evaluated on 1,150 expert-curated tasks across eight mission-critical verticals (including Customer Service, HR, and IT). Our evaluation of 14 frontier models reveals critical limitations in state-of-the-art models: the top-performing Claude Opus 4.5 achieves only 37.4% success. Further analysis shows that providing oracle human plans improves performance by 14-35 percentage points, pinpointing strategic reasoning as the primary bottleneck. Additionally, agents frequently fail to refuse infeasible tasks (best model achieves 53.9%), leading to unintended and potentially harmful side effects. Our findings underscore that current agents are not yet ready for autonomous enterprise deployment. More broadly, EnterpriseOps-Gym provides a concrete testbed to advance the robustness of agentic planning in professional workflows.