Paper 2603.26164
DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models
- Published
- Mar 2026
- Research lab
- Independent
- Citations
- 3
- GitHub
- 1.9K stars
01 In brief
Summary
DataFlex is a unified data-centric dynamic training framework built on LLaMA-Factory, designed to treat data as a first-class optimization variable.
It integrates three paradigms—data selection, data mixture optimization, and data reweighting—through modular trainer abstractions (Select, Mix, Weight Trainers) and pluggable components (selectors, mixers, weighters).
The framework supports both online and offline methods, standardizes model-dependent operations like embedding extraction, inference, and gradient computation, and scales with DeepSpeed ZeRO-3.
Experiments show dynamic methods consistently outperform static full-data training on MMLU for data selection (e.g., LESS achieves 0.452 vs.
0.394 baseline on Mistral-7B; Reweight achieves 0.453 vs.
0.319 on Llama-3.2-3B).
For data mixture, DoReMi and ODM improve MMLU accuracy and perplexity over default proportions when pretraining Qwen2.5-1.5B on SlimPajama at 6B and 30B token scales.
DataFlex also achieves runtime improvements over original implementations, with up to 57.13% time reduction in multi-GPU settings for LESS and consistent 1-3.5% speedups for TSDS.
The framework provides a drop-in replacement for standard training workflows, requiring only a short 'dataflex' section in configuration files, and is available as open-source code with documentation and datasets.
Overall, DataFlex offers an effective, efficient, and reproducible infrastructure for studying and deploying data-centric dynamic training methods for LLMs.
- DataFlex unifies data selection, mixture, and reweighting in one framework, supporting both online and offline algorithms.
- It provides modular trainer abstractions and pluggable components, enabling easy integration and fair comparison.
- Experiments show dynamic methods outperform static baselines in MMLU accuracy and perplexity across multiple models and scales.
- DataFlex achieves runtime improvements…
02 From the paper
Abstract
Data-centric training has emerged as a promising direction for improving large language models (LLMs) by optimizing not only model parameters but also the selection, composition, and weighting of training data during optimization. However, existing approaches to data selection, data mixture optimization, and data reweighting are often developed in isolated codebases with inconsistent interfaces, hindering reproducibility, fair comparison, and practical integration. In this paper, we present DataFlex, a unified data-centric dynamic training framework built upon LLaMA-Factory. DataFlex supports three major paradigms of dynamic data optimization: sample selection, domain mixture adjustment, and sample reweighting, while remaining fully compatible with the original training workflow. It provides extensible trainer abstractions and modular components, enabling a drop-in replacement for standard LLM training, and unifies key model-dependent operations such as embedding extraction, inference, and gradient computation, with support for large-scale settings including DeepSpeed ZeRO-3. We conduct comprehensive experiments across multiple data-centric methods. Dynamic data selection consistently outperforms static full-data training on MMLU across both Mistral-7B and Llama-3.2-3B. For data mixture, DoReMi and ODM improve both MMLU accuracy and corpus-level perplexity over default proportions when pretraining Qwen2.5-1.5B on SlimPajama at 6B and 30B token scales. DataFlex also achieves consistent runtime improvements over original implementations. These results demonstrate that DataFlex provides an effective, efficient, and reproducible infrastructure for data-centric dynamic training of LLMs.