Paper 2604.13016
Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
- Published
- Apr 2026
- Research lab
- Independent
- Citations
- 145
- GitHub
- 892 stars
01 In brief
Summary
This paper systematically investigates on-policy distillation (OPD) for large language models, identifying conditions for success, the token-level mechanism, and practical remedies.
The authors find that OPD succeeds only when the student and teacher share compatible thinking patterns and the teacher offers genuinely new knowledge beyond the student's training data.
Reverse distillation experiments show that same-family teachers are distributionally indistinguishable, and that OPD primarily learns thinking patterns rather than improving benchmark scores.
At the token level, successful OPD is driven by progressive alignment on high-probability overlap tokens, which carry 97-99% of probability mass; optimizing only these tokens suffices.
To rescue failing OPD, the authors propose off-policy cold start via SFT on teacher rollouts and teacher-aligned prompt selection, though the latter risks entropy collapse.
They also show that reward quality degrades with trajectory depth, and that globally informative rewards do not guarantee local exploitability, suggesting limitations for long-horizon tasks.
Sampled-token OPD is sufficient, avoiding the degenerate Top-1 setting.
02 From the paper
Abstract
On-policy distillation (OPD) has become a core technique in the post-training of large language models, yet its training dynamics remain poorly understood. This paper provides a systematic investigation of OPD dynamics and mechanisms. We first identify that two conditions govern whether OPD succeeds or fails: (i) the student and teacher should share compatible thinking patterns; and (ii) even with consistent thinking patterns and higher scores, the teacher must offer genuinely new capabilities beyond what the student has seen during training. We validate these findings through weak-to-strong reverse distillation, showing that same-family 1.5B and 7B teachers are distributionally indistinguishable from the student's perspective. Probing into the token-level mechanism, we show that successful OPD is characterized by progressive alignment on high-probability tokens at student-visited states, a small shared token set that concentrates most of the probability mass (97%-99%). We further propose two practical strategies to recover failing OPD: off-policy cold start and teacher-aligned prompt selection. Finally, we show that OPD's apparent free lunch of dense token-level reward comes at a cost, raising the question of whether OPD can scale to long-horizon distillation.