Paper 2606.18195
Learning from the Self-future: On-policy Self-distillation for dLLMs
- Published
- Jun 2026
- Research lab
- Independent
- Citations
- 1
- GitHub
- 16 stars
01 In brief
Summary
This paper introduces d-OPSD, the first on-policy self-distillation (OPSD) framework tailored for diffusion large language models (dLLMs).
Existing OPSD methods are autoregressive-centric, using left-to-right prefix conditioning and token-level supervision, which conflicts with dLLMs' arbitrary-order generation.
d-OPSD makes two key contributions: it constructs the self-teacher by using self-generated answers as suffix conditioning, allowing the student to learn from its own 'future' experience, and it shifts supervision from token-level to step-level, aligning with the iterative denoising process.
Experiments on four reasoning benchmarks (GSM8K, MATH500, Countdown, Sudoku) show d-OPSD consistently outperforms RLVR and SFT baselines, achieving superior reasoning performance and sample efficiency, requiring only about 10% of the optimization steps of RLVR.
The paper also includes ablations on divergence objectives, retaining ratios, top-k subset selection, and sampling strategies, and discusses failure modes such as policy collapse.
02 From the paper
Abstract
On-policy self-distillation (OPSD) has proven effective for post-training large language models (LLMs), yet its application to diffusion LLMs (dLLMs) remains unexplored. Existing OPSD methods are inherently autoregressive-centric. They inject privileged information via left-to-right prefix conditioning with token-level divergence supervision, a design that fundamentally conflicts with the arbitraryorder generation of dLLMs. We introduce d-OPSD, the first OPSD framework tailored for dLLMs. Our approach makes two core contributions. First, we reframe self-teacher construction by using self-generated answers as suffix conditioning, enabling the student model to learn from "self future-experience" rather than privileged prefixes. Second, we shift supervision from token-level to step-level, aligning training with the iterative denoising process of dLLMs. Experiments across four reasoning benchmarks show that d-OPSD consistently outperforms RLVR and SFT baselines with superior sample efficiency, requiring only around 10% of the optimization steps by RLVR and opening a promising pathway for dLLM posttraining. The code is available at https://github.com/xingzhejun/d-OPSD.