Paper 2511.03276
Diffusion Language Models are Super Data Learners
- Published
- Nov 2025
- Research lab
- Independent
- Citations
- 50
- GitHub
- 228 stars
01 In brief
Summary
The paper demonstrates that under data-constrained pre-training, diffusion language models (DLMs) consistently surpass autoregressive (AR) models of equal size, a phenomenon termed the 'Intelligence Crossover.' This crossover shifts later with more or higher-quality data, earlier with larger models, and persists across dense and sparse architectures.
The gains are attributed to any-order modeling, super-dense compute from iterative bidirectional denoising, and built-in Monte Carlo augmentation.
At scale, a 1.7B DLM trained on 10B unique Python tokens with a 1.5T-token compute budget outperforms an AR coder under matched settings.
A 1B DLM achieves >56% on HellaSwag and >33% on MMLU using only 1B tokens.
The paper also shows that rising validation cross-entropy does not necessarily imply degraded downstream performance.
DLMs require significantly more FLOPs, but when unique data is scarce, they extract more value per token, making them compelling for data-bound regimes.
02 From the paper
Abstract
Under strictly controlled pre-training settings, we observe a Crossover: when unique data is limited, diffusion language models (DLMs) consistently surpass autoregressive (AR) models by training for more epochs. The crossover shifts later with more or higher-quality data, earlier with larger models, and persists across dense and sparse architectures. We attribute the gains to three compounding factors: (1) any-order modeling, (2) super-dense compute from iterative bidirectional denoising, and (3) built-in Monte Carlo augmentation; input or parameter noise improves AR under data constraint but cannot close the gap. At scale, a 1.7B DLM trained with a ~1.5T-token compute budget on 10B unique Python tokens overtakes an AR coder trained with strictly matched settings. In addition, a 1B-parameter DLM achieves > 56% accuracy on HellaSwag and > 33% on MMLU using only 1B tokens, without any special tricks, just by repeating standard pre-training data. We also show that rising validation cross-entropy does not imply degraded downstream performance in this regime.