OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration

OPUS is a dynamic data selection framework for LLM pre-training that scores candidate samples by projecting their optimizer-induced effective updates onto a target direction from a stable, in-distribution proxy.

It addresses the limitations of static filters and raw-gradient-based dynamic methods by aligning selection with the actual update geometry of optimizers like AdamW and Muon.

To ensure scalability, OPUS uses the ghost technique and CountSketch projections, incurring only 4.7% additional compute overhead, and employs Boltzmann sampling to maintain diversity.

Experiments show that OPUS outperforms industrial baselines and even full 200B-token training in GPT-2 Large/XL pre-training on FineWeb and FineWeb-Edu with 30B tokens, achieving an average 2.2% accuracy improvement over random selection and an 8× reduction in computation.

In continued pre-training of Qwen3-8B-Base on SciencePedia, OPUS achieves superior performance using only 0.5B tokens compared to full training with 3B tokens, demonstrating significant data efficiency gains.

Ablations confirm the importance of optimizer-aware scoring, Boltzmann sampling, and the benchmark-matched proxy construction.

The method generalizes to out-of-distribution benchmarks and improves domain-wise perplexity, indicating broad training signal quality rather than overfitting to the proxy set.

Future work includes extending optimizer-aligned selection to richer training regimes like data mixtures.

The paper is a preprint under review, authored by researchers from SJTU, Alibaba, UW-Madison, UIUC, and Mila.

The code and data are not publicly released yet.

The method is theoretically grounded and empirically validated across diverse settings, making it a promising approach for efficient and principled data selection in the data-wall regime.

The paper includes detailed derivations of…