Paper 2512.04324
DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle
- Published
- Dec 2025
- Research lab
- Independent
- Citations
- 11
- GitHub
- 434 stars
01 In brief
Summary
DAComp is a benchmark of 210 tasks evaluating LLM-based agents on enterprise data intelligence, covering repository-level data engineering (DE) and open-ended data analysis (DA).
DE tasks include architecture, implementation, and evolution, requiring multi-stage SQL pipelines on schemas averaging 412 columns and over 2,000 lines of code.
DA tasks involve open-ended business questions requiring planning, iterative coding, and actionable recommendations.
Evaluation uses execution-based metrics (Component Score, Cascading Failure Score, Success Rate) for deterministic tasks and an LLM-judge with hierarchical rubrics for open-ended tasks, validated for reliability.
Results show state-of-the-art agents struggle: DE success rates under 20% and DA scores below 40%, with dependency management and holistic orchestration as key bottlenecks.
A Chinese version, DAComp-zh, is also released.
The benchmark aims to drive development of more capable autonomous data agents for real-world enterprise settings, with data and code available at da-comp.github.io.
The paper was published on December 5, 2025, and includes extensive error analysis and validation of the evaluation framework, showing high human-model agreement and ranking stability across judges and hyperparameters.
The authors are from Institute of Automation CAS, ByteDance Seed, and other institutions, with corresponding authors Kang Liu and Ke Wang.
The benchmark fills a gap by covering both engineering realism and analytical openness, unlike prior benchmarks that focus on isolated tasks or deterministic answers.
The paper concludes that progress requires shifting from code accuracy to planning, open-ended reasoning, and systematic synthesis capabilities.
The benchmark includes 30 DE-Arch, 30 DE-Impl, 50 DE-Evol, and 100 DA tasks, with detailed statistics and baselines…
02 From the paper
Abstract
Real-world enterprise data intelligence workflows encompass data engineering that turns raw sources into analytical-ready tables and data analysis that convert those tables into decision-oriented insights. We introduce DAComp, a benchmark of 210 tasks that mirrors these complex workflows. Data engineering (DE) tasks require repository-level engineering on industrial schemas, including designing and building multi-stage SQL pipelines from scratch and evolving existing systems under evolving requirements. Data analysis (DA) tasks pose open-ended business problems that demand strategic planning, exploratory analysis through iterative coding, interpretation of intermediate results, and the synthesis of actionable recommendations. Engineering tasks are scored through execution-based, multi-metric evaluation. Open-ended tasks are assessed by a reliable, experimentally validated LLM-judge, which is guided by hierarchical, meticulously crafted rubrics. Our experiments reveal that even state-of-the-art agents falter on DAComp. Performance on DE tasks is particularly low, with success rates under 20%, exposing a critical bottleneck in holistic pipeline orchestration, not merely code generation. Scores on DA tasks also average below 40%, highlighting profound deficiencies in open-ended reasoning and demonstrating that engineering and analysis are distinct capabilities. By clearly diagnosing these limitations, DAComp provides a rigorous and realistic testbed to drive the development of truly capable autonomous data agents for enterprise settings. Our data and code are available at https://da-comp.github.io