Paper 2605.11739
Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation
- Published
- May 2026
- Research lab
- Independent
- Citations
- 4
- GitHub
- 73 stars
01 In brief
Summary
This paper investigates why on-policy distillation (OPD) is more efficient than reinforcement learning (RL) for post-training large language models.
The authors identify two properties of OPD's parameter updates: Functional Redundancy Avoidance, where OPD suppresses updates in low-utility modules (e.g., embeddings, peripheral layers) and concentrates on reasoning-critical middle-layer MLPs; and Early Low-Rank Lock-in, where OPD's updates exhibit stronger low-rank concentration and align with the final update subspace early in training.
These properties enable OPD to achieve comparable reasoning performance with smaller, more compact updates.
Based on these insights, the authors propose EffOPD, a plug-and-play acceleration method that extrapolates along the early stable update direction, using a lightweight validation set to adaptively select the extrapolation step.
Experiments across models from 1.5B to 32B parameters show EffOPD achieves an average 3× training speedup while maintaining final performance, and is orthogonal to existing acceleration techniques.
The paper provides a parameter-dynamics perspective on OPD's efficiency and offers practical insights for designing more efficient post-training methods.
02 From the paper
Abstract
On-policy distillation (OPD) has emerged as an efficient post-training paradigm for large language models. However, existing studies largely attribute this advantage to denser and more stable supervision, while the parameter-level mechanisms underlying OPD's efficiency remain poorly understood. In this work, we argue that OPD's efficiency stems from a form of ``foresight'': it establishes a stable update trajectory toward the final model early in training. This foresight manifests in two aspects. First, at the \textbf{Module-Allocation Level}, OPD identifies regions with low marginal utility and concentrates updates on modules that are more critical to reasoning. Second, at the \textbf{Update-Direction Level}, OPD exhibits stronger low-rank concentration, with its dominant subspaces aligning closely with the final update subspace early in training. Building on these findings, we propose \textbf{EffOPD}, a plug-and-play acceleration method that speeds up OPD by adaptively selecting an extrapolation step size and moving along the current update direction. EffOPD requires no additional trainable modules or complex hyperparameter tuning, and achieves an average training acceleration of $3\times$ while maintaining comparable final performance. Overall, our findings provide a parameter-dynamics perspective for understanding the efficiency of OPD and offer practical insights for designing more efficient post-training methods for large language models.