Paper 2606.07082
On the Geometry of On-Policy Distillation
- Published
- Jun 2026
- Research lab
- Independent
- Citations
- 2
- GitHub
- Not linked
01 In brief
Summary
This paper analyzes the parameter-space geometry of on-policy distillation (OPD) for large language models, comparing it with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR).
Using diagnostics like update sparsity, subspace rotation, spectral drift, and update localization, the authors find that OPD occupies a 'relaxed off-principal' regime: it is more selective and geometry-preserving than SFT but less constrained than RLVR.
Trajectory analysis reveals 'subspace locking': OPD's cumulative updates rapidly enter a narrow, low-dimensional channel that is stable and functionally sufficient, as constraining training to this early subspace preserves OPD performance but degrades SFT.
Control experiments show that token sparsification and off-policy rollouts preserve the rank dynamics, while mixing the OPD objective with RLVR changes them, indicating that objective composition controls the locking.
The authors conclude that OPD is not an interpolation between SFT and RLVR but induces its own update geometry, suggesting that OPD algorithms should be designed as geometry control rather than merely denser token supervision.
02 From the paper
Abstract
On-policy distillation (OPD) is increasingly used to improve large language model reasoning, but its training dynamics remain poorly understood. We characterize the trajectory of OPD updates in parameter space and compare it with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). A suite of parameter-space diagnostics consistently places OPD in a relaxed off-principal regime: compared with SFT, its updates affect fewer weights and avoid principal directions more strongly, while compared with RLVR, they remain less tightly constrained. Beyond this static localization, OPD exhibits subspace locking: its cumulative updates rapidly enter a narrow low-dimensional channel. Constraining training to the update subspace formed early in training preserves OPD performance but substantially degrades SFT, indicating that the locked subspace is functionally sufficient for OPD. Control experiments further show that sparsifying the update tokens and shifting rollout generation off-policy preserve the rank dynamics, whereas mixing the OPD objective with RLVR changes them. Overall, these results suggest that OPD is not merely an intermediate point between SFT and RLVR, but induces its own update geometry in parameter space.