Paper 2602.12125
Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation
- Published
- Feb 2026
- Research lab
- Independent
- Citations
- 87
- GitHub
- 278 stars
01 In brief
Summary
The paper introduces Generalized On-Policy Distillation (G-OPD), a framework that extends standard on-policy distillation (OPD) by adding a flexible reference model and a reward scaling factor λ.
The authors show that OPD is a special case of dense KL-constrained RL with equal reward and KL weights.
In G-OPD, λ controls the relative weight of the reward term; setting λ>1 (termed ExOPD) enables reward extrapolation, allowing the student to surpass the teacher's performance.
Experiments on math reasoning and code generation with Qwen3 models show that ExOPD consistently outperforms standard OPD and off-policy distillation (SFT) in both single-teacher and multi-teacher settings.
In multi-teacher distillation, ExOPD produces a unified student that outperforms all domain teachers.
In strong-to-weak distillation, using the teacher's pre-RL base model as the reference (reward correction) further improves performance, though it requires access to that model and adds computational cost.
The paper provides theoretical analysis and empirical validation, highlighting ExOPD's potential for efficient and effective distillation.
02 From the paper
Abstract
On-policy distillation (OPD), which aligns the student with the teacher's logit distribution on student-generated trajectories, has demonstrated strong empirical gains in improving student performance and often outperforms off-policy distillation and reinforcement learning (RL) paradigms. In this work, we first theoretically show that OPD is a special case of dense KL-constrained RL where the reward function and the KL regularization are always weighted equally and the reference model can by any model. Then, we propose the Generalized On-Policy Distillation (G-OPD) framework, which extends the standard OPD objective by introducing a flexible reference model and a reward scaling factor that controls the relative weight of the reward term against the KL regularization. Through comprehensive experiments on math reasoning and code generation tasks, we derive two novel insights: (1) Setting the reward scaling factor to be greater than 1 (i.e., reward extrapolation), which we term ExOPD, consistently improves over standard OPD across a range of teacher-student size pairings. In particular, in the setting where we merge the knowledge from different domain experts, obtained by applying domain-specific RL to the same student model, back into the original student, ExOPD enables the student to even surpass the teacher's performance boundary and outperform the domain teachers. (2) Building on ExOPD, we further find that in the strong-to-weak distillation setting (i.e., distilling a smaller student from a larger teacher), performing reward correction by choosing the reference model as the teacher's base model before RL yields a more accurate reward signal and further improves distillation performance. However, this choice assumes access to the teacher's pre-RL variant and incurs more computational overhead. We hope our work offers new insights for future research on OPD.