Paper 2604.08523
ClawBench: Can AI Agents Complete Everyday Online Tasks?
- Published
- Apr 2026
- Research lab
- Independent
- Citations
- 19
- GitHub
- 550 stars
01 In brief
Summary
ClawBench is a benchmark of 153 everyday online tasks across 144 live websites and 15 categories, focusing on write-heavy, state-changing workflows like purchases, reservations, and applications.
Unlike sandboxed benchmarks, it evaluates agents on production sites with dynamic content and anti-bot defenses, using final-request interception to block irreversible actions safely.
Each run is recorded in five layers (video, screenshots, HTTP traffic, agent messages, browser actions) and scored by an Agent-as-Judge against human reference trajectories, achieving 84.97–93.46% agreement with human verdicts.
Across 8 frontier models, the best (Claude Sonnet 4.6) achieves only 33.3% success, with Qwen 3.5 at 26.1% and GPT-5.4 at 6.5%, far below their performance on existing benchmarks.
Trace analysis reveals failures stem from anti-bot walls, unnatural interaction patterns, safety refusals, and last-mile hesitation before final submission.
The benchmark highlights a large gap between controlled benchmark performance and real-world web competence, suggesting progress requires better planning, perception, and behaviorally grounded interaction models.
02 From the paper
Abstract
AI agents may be able to assist with emails and documents, but can they reliably complete everyday online workflows on real websites? Everyday online tasks offer a realistic yet unsolved testbed for evaluating the next generation of AI agents. To this end, we introduce ClawBench, an evaluation framework comprising 153 everyday online tasks that people need to accomplish regularly in their lives and work, spanning 144 platforms across 15 categories, from completing purchases and booking appointments to submitting job applications. These tasks require capabilities beyond existing benchmarks, such as obtaining relevant information from user-provided documents, navigating multi-step workflows across diverse platforms, and write-heavy operations like filling in many detailed forms correctly. Unlike existing benchmarks that evaluate agents in offline sandboxes with static pages, ClawBench operates on production websites, preserving the full complexity, dynamic nature, and interaction challenges of real-world web environments. An interception layer captures and blocks the final submission request, ensuring safe evaluation without real-world side effects. Our evaluations of 8 frontier models show that both proprietary and open-source models complete only a small portion of these tasks. For example, Claude Sonnet 4.6 achieves only 33.3%, which exposes gaps in current AI agents. Progress on ClawBench brings us closer to AI agents that can function as general-purpose assistants.