The Year in AI Papers
The year/Independent research

Paper 2606.12344

Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks

Mengyu Zheng, Kai Han, Boxun Li, Haiyang Xu, Yuchuan Tian, Wei He, Hang Zhou, Jianyuan Guo, Hailin Hu, Lin Ma, Chao Xu, Guohao Dai, et al.

Published
Jun 2026
Research lab
Independent
Citations
2
GitHub
96 stars

01 In brief

Summary

Claw-SWE-Bench is a multilingual SWE-bench-style benchmark and adapter protocol for evaluating general-purpose agent harnesses (claws) on coding tasks.

It comprises 350 GitHub issue-resolution instances across 8 languages and 43 repositories, drawn from SWE-bench-Multilingual and SWE-bench-Verified-Mini after future-commit cleanup.

The benchmark fixes prompt, runtime budget, workspace contract, patch extraction, and evaluator, while allowing harnesses to be swapped via a shared adapter interface.

A minimal bare adapter yields only 19.1% Pass@1, whereas the full adapter reaches 73.4% with the same GLM 5.1 backbone, showing adapter design is essential.

Across a nine-model sweep, model choice changes Pass@1 by 29.4 percentage points; across a five-claw sweep, harness choice changes it by up to 27.4 points.

Accuracy and cost do not align: similar Pass@1 can differ by orders of magnitude in API cost.

Claw-SWE-Bench Lite, an 80-instance subset, approximates full-set Pass@1 within about 0.4 percentage points while reducing cost to about 22.9% of the full run.

The benchmark treats harness and cost accounting as first-class evaluation axes, providing a reproducible comparison framework.

02 From the paper

Abstract

General-purpose agents such as OpenClaw are increasingly used as autonomous tool users, but their coding ability is difficult to measure under SWE-bench: a generic agent does not by itself satisfy the clean Docker workspace, patch, and prediction contract required for scoring. We introduce Claw-SWE-Bench, a multilingual SWE-bench-style benchmark and adapter protocol that makes heterogeneous agent harnesses, or claws, comparable under fair settings including a fixed prompt, runtime budget, workspace contract, patch extraction procedure, and evaluator. The full benchmark contains 350 GitHub issue-resolution instances across 8 languages and 43 repositories, drawn from SWE-bench-Multilingual and SWE-bench-Verified-Mini after future-commit cleanup. We also release Claw-SWE-Bench Lite for faster validation, which is an 80-instance subset selected by a cost-aware, rank-aware procedure over 17 calibration columns. On the full benchmark, OpenClaw with a minimal direct-diff adapter scores only $19.1\%$ Pass@1, whereas the full adapter reaches $73.4\%$ with the same GLM 5.1 backbone, showing that adapter design is essential for enabling OpenClaw-style harnesses to perform coding tasks effectively. Across an OpenClaw $\times$ nine-model sweep and a five-claw $\times$ two-model sweep, model choice changes Pass@1 by $29.4$ pp and harness choice by $27.4$ pp under fixed models; systems with similar accuracy can differ substantially in total API cost. Claw-SWE-Bench therefore treats harness and cost accounting as first-class axes of SWE-style coding-agent evaluation, providing both a full benchmark and a low-cost reference set for reproducible comparison. The data is available at https://github.com/opensquilla/claw-swe-bench and https://huggingface.co/datasets/TokenRhythm/Claw-SWE-Bench.