Paper 2604.03128
Self-Distilled RLVR
- Published
- Apr 2026
- Research lab
- Independent
- Citations
- 56
- GitHub
- Not linked
01 In brief
Summary
The paper analyzes on-policy self-distillation (OPSD) for LLM post-training, where a model serves as both teacher and student, with the teacher using privileged information.
It proves that OPSD's information asymmetry creates an irreducible mutual information gap, causing privileged information leakage and performance degradation after early gains.
To address this, the authors propose RLSD (RLVR with Self-Distillation), which uses the environment reward to determine update direction and the teacher's evidence ratio to modulate update magnitude per token.
This decoupling anchors gradients to reliable environmental feedback while retaining dense token-level credit assignment.
Experiments on Qwen3-VL-8B-Instruct across five multimodal reasoning benchmarks show RLSD outperforms GRPO, OPSD, SDPO, and GRPO+OPSD, achieving the highest average accuracy (56.18%) and faster convergence, while avoiding leakage and training instability.
RLSD requires only the final answer as privileged information and adds minimal computational overhead, serving as a drop-in replacement for GRPO's uniform advantage.
02 From the paper
Abstract
On-policy distillation (OPD) has become a popular training paradigm in the LLM community. This paradigm selects a larger model as the teacher to provide dense, fine-grained signals for each sampled trajectory, in contrast to reinforcement learning with verifiable rewards (RLVR), which only obtains sparse signals from verifiable outcomes in the environment. Recently, the community has explored on-policy self-distillation (OPSD), where the same model serves as both teacher and student, with the teacher receiving additional privileged information such as reference answers to enable self-evolution. This paper demonstrates that learning signals solely derived from the privileged teacher result in severe information leakage and unstable long-term training. Accordingly, we identify the optimal niche for self-distillation and propose \textbf{RLSD} (\textbf{RL}VR with \textbf{S}elf-\textbf{D}istillation). Specifically, we leverage self-distillation to obtain token-level policy differences for determining fine-grained update magnitudes, while continuing to use RLVR to derive reliable update directions from environmental feedback (e.g., response correctness). This enables RLSD to simultaneously harness the strengths of both RLVR and OPSD, achieving a higher convergence ceiling and superior training stability.