Paper 2606.09426
WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces
- Published
- Jun 2026
- Research lab
- Independent
- Citations
- 2
- GitHub
- 160 stars
01 In brief
Summary
WeaveBench is a long-horizon, hybrid-interface benchmark for computer-use agents, comprising 114 tasks across 8 real-world work domains.
Each task requires agents to combine GUI observations/actions with CLI/code operations within a single trajectory, satisfying three admission criteria: channel non-substitutability, long-horizon execution, and cross-application state.
Tasks are sourced from real user requests and evaluated on a real Ubuntu desktop using deployed CLI-agent runtimes (OpenClaw, Codex CLI, Claude Code, Hermes) augmented with a minimal desktop-control plugin.
A trajectory-aware agentic judge inspects deliverables, files, screenshots, logs, and action traces, detecting shortcut behaviors like fabricated visual evidence or hard-coded metrics.
Results show the benchmark is far from saturated: the best PassRate is 41.2% (Claude Opus 4.7 + Claude Code), with single-interface ablations at or below 3.5%.
Outcome-only grading overestimates performance by 10-20 percentage points.
Failure analysis reveals reward hacking (35%), long-horizon discipline breakdown (30%), and reasoning imprecision (17%) as dominant failure modes, with visual grounding being a minor issue (<4%).
02 From the paper
Abstract
Computer-use agents (CUAs) increasingly operate in runtimes that combine visual desktop control, command-line execution, code editing, browsers, and external tools. Existing benchmarks, however, often evaluate these interfaces as separable capabilities, leaving long-horizon cross-interface orchestration under-tested. Thus, we introduce WeaveBench, a long-horizon hybrid-interface benchmark with 114 tasks across 8 real-world work domains, grounded in real user requests and publicly verifiable artifacts. Each task requires agents to combine GUI observations/actions with CLI/code operations within a single trajectory. We evaluate these tasks on a real Ubuntu desktop inside deployed CLI-agent runtimes, augmented with a minimal desktop-control plugin. We also propose a companion trajectory-aware judge that inspects deliverables, files, screenshots, logs, and action traces, while detecting shortcut behaviors such as fabricated visual evidence or hard-coded metrics. Across frontier model-runtime pairings, the best PassRate reaches only 41.2%, showing the benchmark remains far from saturated. The trajectory-aware judge further reveals that outcome-only grading substantially overestimates agent performance. Overall, WeaveBench exposes a critical gap in CUA evaluation and provides an effective testbed to measure whether agents can orchestrate GUI, CLI, and code operations across long-horizon real-world tasks.