Paper 2604.24819
Programming with Data: Test-Driven Data Engineering for Self-Improving LLMs from Raw Corpora
- Published
- Apr 2026
- Research lab
- Independent
- Citations
- 1
- GitHub
- 162 stars
01 In brief
Summary
The paper introduces Programming with Data (ProDa), a paradigm that maps the LLM data-engineering lifecycle onto software development by using a shared three-level knowledge structure (L1 concepts, L2 relations, L3 reasoning chains) extracted from raw corpora.
This structure serves as the common foundation for both training data synthesis and benchmark construction, enabling failures to be traced to specific knowledge gaps and repaired through targeted data patches.
The ProDa framework operationalizes this via Builder, Tester, and Debugger components, guided by the CORE principle.
Applied to 16 disciplines, ProDa-16 benchmark shows high correlation with established benchmarks (mean Spearman ρ=0.847).
After one debugging iteration, models improve consistently across scales (e.g., Qwen-3-32B reaches 79.52% vs 77.35% at V1), with general capabilities preserved.
ProDa outperforms baseline synthesis methods (Alpaca, EasyDataset, DataFlow) in sample efficiency and accuracy.
The work releases ProDaLib (227k concepts, 16k evaluation items, 160k training samples) and ProDa Studio IDE.
02 From the paper
Abstract
Reliably transferring specialized human knowledge from text into large language models remains a fundamental challenge in artificial intelligence. Fine-tuning on domain corpora has enabled substantial capability gains, but the process operates without feedback: when a model fails on a domain task, there is no method to diagnose what is deficient in the training data, and the only recourse is to add more data indiscriminately. Here we show that when a structured knowledge representation extracted from the source corpus serves as the shared foundation for both training data and evaluation, the complete data-engineering lifecycle maps onto the software development lifecycle in a precise and operative way: training data becomes source code specifying what the model should learn, model training becomes compilation, benchmarking becomes unit testing, and failure-driven data repair becomes debugging. Under this correspondence, model failures decompose into concept-level gaps and reasoning-chain breaks that can be traced back to specific deficiencies in the data and repaired through targeted patches, with each repair cycle producing consistent improvements across model scales and architectures without degrading general capabilities. We formalize this principle as Programming with Data and instantiate it across sixteen disciplines spanning the natural sciences, engineering, biomedicine, and the social sciences, releasing a structured knowledge base, benchmark suite, and training corpus as open resources. By demonstrating that the relationship between training data and model behaviour is structurally traceable and systematically repairable, this work establishes a principled foundation for the reliable engineering of human expertise into language models.