Paper 2601.11077
ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development
- Published
- Jan 2026
- Research lab
- Independent
- Citations
- 3
- GitHub
- 33 stars
01 In brief
Summary
ABC-Bench is a benchmark for evaluating LLM-based agents on full-lifecycle backend development tasks, covering repository exploration, code editing, environment configuration, deployment, and end-to-end API testing.
It includes 224 tasks from 127 MIT-licensed GitHub repositories, spanning 8 languages and 19 frameworks.
The ABC-Pipeline automates task construction via a three-phase process: repository exploration, environment synthesis, and task instantiation with masking.
Evaluation uses a sandboxed setup where agents must build and run a Dockerized service, verified via external API tests.
Results show that even the best model, Claude Sonnet 4.5, achieves only 63.2% pass@1, with environment configuration identified as the primary bottleneck.
Smaller models struggle with basic syntax and path errors, while larger models face logic errors.
Agent framework choice significantly impacts performance, and agentic post-training improves results.
The benchmark highlights a gap between current model capabilities and real-world backend engineering demands.
02 From the paper
Abstract
The evolution of Large Language Models (LLMs) into autonomous agents has expanded the scope of AI coding from localized code generation to complex, repository-level, and execution-driven problem solving. However, current benchmarks predominantly evaluate code logic in static contexts, neglecting the dynamic, full-process requirements of real-world engineering, particularly in backend development which demands rigorous environment configuration and service deployment. To address this gap, we introduce ABC-Bench, a benchmark explicitly designed to evaluate agentic backend coding within a realistic, executable workflow. Using a scalable automated pipeline, we curated 224 practical tasks spanning 8 languages and 19 frameworks from open-source repositories. Distinct from previous evaluations, ABC-Bench require the agents to manage the entire development lifecycle from repository exploration to instantiating containerized services and pass the external end-to-end API tests. Our extensive evaluation reveals that even state-of-the-art models struggle to deliver reliable performance on these holistic tasks, highlighting a substantial disparity between current model capabilities and the demands of practical backend engineering. Our code is available at https://github.com/OpenMOSS/ABC-Bench.