Paper 2606.07297
SWE-Explore: Benchmarking How Coding Agents Explore Repositories
- Published
- Jun 2026
- Research lab
- Independent
- Citations
- 10
- GitHub
- 39 stars
01 In brief
Summary
SWE-Explore is a benchmark that isolates and evaluates the repository exploration capability of coding agents, independent of patch generation.
It formalizes exploration as a ranked, line-level context selection task: given an issue and repository, an explorer returns a ranked list of code regions under a fixed line budget.
The benchmark includes 848 issues across 10 programming languages and 203 open-source repositories, derived from SWE-bench Verified, SWE-bench-Pro, and SWE-bench Multilingual.
Ground truth is trajectory-grounded: line-level core regions are extracted from successful agent trajectories (at least two per instance) by intersecting read actions, refined via LLM and human audit.
Evaluation metrics include coverage (precision, recall, F1), ranking (nDCG@B, first useful hit), and efficiency (context efficiency, noise rate).
A controlled downstream protocol feeds explorer output as the only context to a fixed patcher, showing metrics like Context Efficiency (r=0.950) and Rec@100 (ρ=0.845) strongly predict repair success.
Experiments show agentic explorers outperform classical retrieval (BM25, TF-IDF, RAG), but all are recall-limited at line level despite high file-level hits.
Missing core context hurts more than redundant context, and LLM choice shifts performance but not the bottleneck.
SWE-Explore provides a focused target for improving repository exploration in coding agents.
02 From the paper
Abstract
Repository-level coding benchmarks such as SWE-bench have driven a rapid surge in the capabilities of coding agents. Yet they usually treat coding tasks as a holistic, binary prediction problem (e.g., resolved or unresolved), neglecting fine-grained agent capabilities such as repository understanding, context retrieval, code localization, and bug diagnosis. In this paper, we introduce SWE-Explore, a benchmark that isolates the evaluation of repository exploration, a critical capability of coding agents. Given a repository and an issue, SWE-Explore asks an explorer to return a ranked list of relevant code regions under a fixed line budget. SWE-Explore covers 848 issues across 10 programming languages and 203 open-source repositories. For each instance, we derive line-level ground truth from independent agent trajectories that successfully solved the same issue, distilling the specific code regions their solution paths actually consulted. We evaluate exploration along coverage, ranking, and context-efficiency dimensions, showing that these metrics strongly track downstream repair behavior. Across a broad set of retrieval methods, general coding agents, and specialized localizers, we find that agentic explorers form a clear tier above classical retrieval. While file-level localization is already strong for modern methods, line-level coverage and efficient ranking remain the key axes differentiating state-of-the-art explorers.