Paper 2601.16344
DSGym: A Holistic Framework for Evaluating and Training Data Science Agents
- Published
- Jan 2026
- Research lab
- Together AI
- Citations
- 12
- GitHub
- 58 stars
01 In brief
Summary
DSGym is a standardized framework for evaluating and training data science agents in isolated, stateful execution environments.
It addresses limitations in existing benchmarks, which often allow tasks to be solved without accessing data files (shortcut solvability), lack cross-benchmark standardization, and have narrow domain coverage.
DSGym provides a modular architecture with unified task objects, a default CodeAct-like agent interface, and a manager-worker container system that enforces read-only data mounts and writable workspaces.
The framework includes DSGym-Tasks, a curated suite that refines existing benchmarks (DAEval, QRData, DABStep, MLEBench-Lite) through quality verification and shortcut filtering, and expands coverage with DSBio (90 expert-derived bioinformatics tasks) and DSPredict (92 Kaggle competitions split into easy and hard).
Evaluations of frontier models reveal persistent gaps in domain-specific scientific workflows, with over 80% of failures on DSBio due to domain-grounding errors, and a simplicity bias in modeling tasks where agents often stop after producing runnable but under-optimized solutions.
DSGym also supports training via execution-verified synthetic data generation; a 4B model fine-tuned on 2,000 synthetic examples outperformed GPT-4o on standardized analysis benchmarks.
The framework is designed as a live, extensible testbed for reproducible evaluation and training of data science agents.
02 From the paper
Abstract
Data science agents promise to accelerate discovery and insight-generation by turning data into executable analyses and findings. Yet existing data science benchmarks fall short due to fragmented evaluation interfaces that make cross-benchmark comparison difficult, narrow task coverage and a lack of rigorous data grounding. In particular, we show that a substantial portion of tasks in current benchmarks can be solved without using the actual data. To address these limitations, we introduce DSGym, a standardized framework for evaluating and training data science agents in self-contained execution environments. Unlike static benchmarks, DSGym provides a modular architecture that makes it easy to add tasks, agent scaffolds, and tools, positioning it as a live, extensible testbed. We curate DSGym-Tasks, a holistic task suite that standardizes and refines existing benchmarks via quality and shortcut solvability filtering. We further expand coverage with (1) DSBio: expert-derived bioinformatics tasks grounded in literature and (2) DSPredict: challenging prediction tasks spanning domains such as computer vision, molecular prediction, and single-cell perturbation. Beyond evaluation, DSGym enables agent training via execution-verified data synthesis pipeline. As a case study, we build a 2,000-example training set and trained a 4B model in DSGym that outperforms GPT-4o on standardized analysis benchmarks. Overall, DSGym enables rigorous end-to-end measurement of whether agents can plan, implement, and validate data analyses in realistic scientific context.