Paper 2512.15745
LLaDA2.0: Scaling Up Diffusion Language Models to 100B
- Published
- Dec 2025
- Research lab
- Independent
- Citations
- 123
- GitHub
- 501 stars
01 In brief
Summary
LLaDA2.0 introduces a family of discrete diffusion language models (dLLMs) scaled up to 100B parameters by converting pre-trained auto-regressive (AR) models, avoiding costly training from scratch.
The conversion uses a three-phase Warmup-Stable-Decay (WSD) continual pre-training strategy: progressively increasing block size in block diffusion (warmup), large-scale full-sequence diffusion (stable), and reverting to compact block diffusion (decay).
A document-level attention mask prevents cross-document interference.
Post-training includes SFT with complementary masking and mask ratio bandwidth, a confidence-aware parallel loss for efficient decoding, and DPO adapted via ELBO.
Two instruction-tuned MoE variants are released: LLaDA2.0-mini (16B) and LLaDA2.0-flash (100B).
Evaluations on 47 benchmarks show LLaDA2.0-mini averages 64.34, close to Ling-mini-2.0 (65.77), while LLaDA2.0-flash averages 73.18, matching Qwen3-30B-A3B-Instruct-2507 (73.60) and excelling in coding (HumanEval 94.51, MBPP 88.29) and agent tasks (BFCL v3 75.43).
Inference speed reaches 535 tokens/second with CAP, up to 2.1x faster than AR baselines.
Both models are open-sourced.
02 From the paper
Abstract
This paper presents LLaDA2.0 -- a tuple of discrete diffusion large language models (dLLM) scaling up to 100B total parameters through systematic conversion from auto-regressive (AR) models -- establishing a new paradigm for frontier-scale deployment. Instead of costly training from scratch, LLaDA2.0 upholds knowledge inheritance, progressive adaption and efficiency-aware design principle, and seamless converts a pre-trained AR model into dLLM with a novel 3-phase block-level WSD based training scheme: progressive increasing block-size in block diffusion (warm-up), large-scale full-sequence diffusion (stable) and reverting back to compact-size block diffusion (decay). Along with post-training alignment with SFT and DPO, we obtain LLaDA2.0-mini (16B) and LLaDA2.0-flash (100B), two instruction-tuned Mixture-of-Experts (MoE) variants optimized for practical deployment. By preserving the advantages of parallel decoding, these models deliver superior performance and efficiency at the frontier scale. Both models were open-sourced.