Paper 2601.15892
Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Model
- Published
- Jan 2026
- Research lab
- Independent
- Citations
- 13
- GitHub
- 84 stars
01 In brief
Summary
Stable-DiffCoder is a diffusion-based language model for code that reuses the Seed-Coder architecture, data, and training pipeline but replaces autoregressive (AR) training with a block diffusion continual pretraining (CPT) stage.
The authors introduce a tailored warmup and a block-wise clipped noise schedule to stabilize training and improve knowledge compression.
Under identical data and architecture, Stable-DiffCoder outperforms its AR counterpart (Seed-Coder) on most code benchmarks, including HumanEval, MBPP, MultiPL-E, CRUXEval, MHPP, BigCodeBench, and CanItEdit, while being slightly behind on LiveCodeBench and Aider.
It achieves state-of-the-art results among ~8B diffusion and AR code models.
The work demonstrates that diffusion-based training can serve as effective data augmentation, improving code generation, reasoning, and editing, especially for low-resource languages.
Limitations include reduced performance on non-code tasks and long-context editing scenarios.
02 From the paper
Abstract
Diffusion-based language models (DLLMs) offer non-sequential, block-wise generation and richer data reuse compared to autoregressive (AR) models, but existing code DLLMs still lag behind strong AR baselines under comparable budgets. We revisit this setting in a controlled study and introduce Stable-DiffCoder, a block diffusion code model that reuses the Seed-Coder architecture, data, and training pipeline. To enable efficient knowledge learning and stable training, we incorporate a block diffusion continual pretraining (CPT) stage enhanced by a tailored warmup and block-wise clipped noise schedule. Under the same data and architecture, Stable-DiffCoder overall outperforms its AR counterpart on a broad suite of code benchmarks. Moreover, relying only on the CPT and supervised fine-tuning stages, Stable-DiffCoder achieves stronger performance than a wide range of \~8B ARs and DLLMs, demonstrating that diffusion-based training can improve code modeling quality beyond AR training alone. Moreover, diffusion-based any-order modeling improves structured code modeling for editing and reasoning, and through data augmentation, benefits low-resource coding languages.