Paper 2606.23654
EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions
- Published
- Jun 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- 46 stars
01 In brief
Summary
EnterpriseClawBench is an enterprise agent benchmark built from 5,291 real workplace sessions at an AI startup, yielding 852 reproducible tasks (120 in a manually audited Lite subset).
The pipeline applies mechanical gates (length, fixture, redaction, network) and rewrites prompts into single-turn tasks with role/skill taxonomies, hard rules, and semantic rubrics.
The benchmark data is not released due to proprietary content.
On Lite, the best score is 0.663 (Codex/GPT-5.5), showing tasks are unsaturated.
Key findings: harness–model coupling matters (Claude-family models drop under Hermes due to runtime mismatches); cost–score trade-off is log-like; marketing and finance/ops tasks are harder; GPT-5.5 is the most robust generalist; visual-judged artifacts show score inflation.
Skill transfer experiments on frontend page generation show GPT-5.5 is the best skill creator (+0.068 avg delta), while Haiku 4.5 is the weakest (-0.094).
Judge reliability is strong for text (Spearman 0.790) but weak for visual artifacts (Spearman -0.259).
The paper argues for multidimensional evaluation reporting harness–model combos, artifact delivery, cost, runtime, and skill transfer.
02 From the paper
Abstract
Enterprise agents increasingly operate inside workspaces: they read heterogeneous files, invoke tools, and deliver business artifacts. We introduce EnterpriseClawBench, an enterprise agent benchmark constructed from proprietary, real-world agent sessions. Starting from a large archive of workplace sessions, the EnterpriseClawBench produces 852 reproducible tasks, each paired with recovered fixtures, rewritten prompts, role classes, skill subclasses, hard rules, and semantic rubrics. Because the sessions contain internal enterprise content, we do not release the benchmark data; instead, our reusable contribution is the construction and evaluation protocol. On EnterpriseClawBench, the best configuration reaches only 0.663 (Codex with GPT-5.5). These results show that enterprise agent evaluation must report harness--model combinations, artifact delivery, visual quality, cost, runtime, and skill-transfer behavior, rather than collapsing performance into a single score. Code: https://github.com/FrontisAI/EnterpriseClawBench