The Year in AI Papers
The year/Independent research

Paper 2603.03194

BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?

Guoxin Chen, Fanzhe Meng, Jiale Zhao, Minghao Li, Daixuan Cheng, Huatong Song, Jie Chen, Yuzhi Lin, Hui Chen, Xin Zhao, Ruihua Song, Chang Liu, et al.

Published
Mar 2026
Research lab
Independent
Citations
11
GitHub
47 stars

01 In brief

Summary

BeyondSWE is a 500-instance benchmark from 246 real GitHub repositories that evaluates code agents beyond single-repository bug fixing, covering four settings: cross-repository issue resolution (CrossRepo), domain-specific issue resolution (DomainFix), dependency-driven migration (DepMigrate), and document-to-repository generation (Doc2Repo).

These tasks require broader knowledge or resolution scope than existing SWE-bench-style benchmarks, with target solutions affecting an average of 10.9 files and 1039.6 lines.

Evaluation shows the benchmark is far from saturated: the best OpenHands-based agent (DeepSeek-V4-Pro Max) scores 46.12, while the strongest Codex harness with GPT-5.4 (xhigh) reaches 56.65 under a search-aware prompt.

Using SearchSWE, a controlled diagnostic baseline adding web search and fetch tools, search access improves most models but gains are limited and uneven; the Codex configuration improves from 48.48 to 56.65 with explicit search prompting.

Case studies identify three failure modes: evidence retrieved at wrong granularity, external knowledge not grounded in local dependency versions, and keyword-matched but semantically unrelated results contaminating context.

Cost and token analysis shows higher budgets do not reliably improve performance, indicating token productivity rather than volume is the bottleneck.

The authors conclude that deep search for coding remains an open problem, requiring agents that can combine external evidence with repository-local reasoning and execution-based verification.

02 From the paper

Abstract

Current code-agent benchmarks primarily evaluate localized issue resolution within a single target repository, leaving under-tested many software engineering tasks that require external knowledge or broader repository-level changes. We introduce BeyondSWE, a 500-instance benchmark drawn from 246 real-world GitHub repositories to evaluate code agents beyond single-repository bug fixing. BeyondSWE covers four representative settings: cross-repository issue resolution, domain-specific issue resolution, dependency-driven migration, and document-to-repository generation, spanning both broader knowledge scope and broader resolution scope. Our evaluation shows that BeyondSWE remains far from saturated: the best OpenHands-based agent reaches 46.12 average score, while the strongest Codex harness with GPT-5.4 (xhigh) reaches 56.65 under a search-aware prompt. To study whether external information access closes this gap, we use SearchSWE as a controlled diagnostic baseline for search-augmented coding. Search access improves most models and substantially helps some tasks, but the gains remain limited and uneven, showing that current agents still struggle to convert retrieved information into precise, version-compatible, and locally actionable code changes. These results suggest that deep search for coding remains an open problem: progress requires agents that can reliably combine external evidence with repository-local reasoning and execution-based verification.