Paper 2604.06132
Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents
- Published
- Apr 2026
- Research lab
- Independent
- Citations
- 38
- GitHub
- 741 stars
01 In brief
Summary
Claw-Eval is an end-to-end evaluation suite for LLM-based autonomous agents, addressing gaps in trajectory-opaque grading, underspecified safety/robustness, and narrow task coverage.
It comprises 300 human-verified tasks across 9 categories (General, Multimodal, Multi-turn Dialogue) with 2,159 rubric items.
Each run is audited via three evidence channels: execution traces, audit logs, and environment snapshots.
Scoring evaluates Completion, Safety, and Robustness, with metrics Average Score, Pass@k, and Pass^k over three trials.
Experiments on 14 frontier models reveal: (1) trajectory-opaque evaluation misses 44% of safety violations and 13% of robustness failures; (2) capability does not imply consistency, as Pass@3 remains stable under error injection while Pass^3 drops by up to 24 percentage points; (3) agent capability is multi-dimensional, with rankings varying across task groups and domains.
The benchmark is open-source and reproducible, aiming to guide development of reliably deployable agents.
Key findings include that question quality (r=0.87) predicts multi-turn success far better than round count (r=0.07), and no single model dominates all multimodal domains.
Claw-Eval highlights the need for trajectory-level evidence, multi-dimensional scoring, and broad coverage in agent evaluation.
- The suite includes 300 tasks, 2,159 rubric items, and evaluates 14 models.
- Hybrid grading detects safety violations and robustness issues that vanilla LLM judges miss.
- Error injection degrades consistency (Pass^3) more than peak capability (Pass@3).
- Multi-turn performance correlates with question precision (r=0.87) not round count (r=0.07).
- Multimodal capability is domain-specific; no model leads across all domains.
02 From the paper
Abstract
Large language models are increasingly deployed as autonomous agents for multi-step workflows in real-world software environments. However, existing agent benchmarks are limited by trajectory-opaque grading, underspecified safety and robustness evaluation, and narrow coverage of modalities and interaction paradigms. We introduce Claw-Eval, an end-to-end evaluation suite addressing these gaps with 300 human-verified tasks spanning 9 categories across three groups: general service orchestration, multimodal perception and interaction, and multi-turn professional dialogue. To enable trajectory-aware grading, each run is recorded through three independent evidence channels: execution traces, audit logs, and environment snapshots, yielding 2,159 fine-grained rubric items. The scoring protocol evaluates Completion, Safety, and Robustness, with Average Score, Pass@k, and Pass^k across three trials to distinguish genuine capability from lucky outcomes. Experiments on 14 frontier models show that: (1) Trajectory-opaque evaluation is systematically unreliable, missing 44% of safety violations and 13% of robustness failures detected by our framework. (2) Capability does not imply consistency, with Pass@3 remaining stable under error injection while Pass^3 dropping by up to 24 percentage points. (3) Agent capability is strongly multi-dimensional, with model rankings varying across task groups and metrics, indicating that our heterogeneous evaluation coverage is essential. Claw-Eval highlights directions for developing agents that are not only capable but reliably deployable.