Trust-Region Behavior Blending for On-Policy Distillation

The paper introduces Trust-Region behavior Blending (TRB), a warmup method for on-policy distillation (OPD) that addresses the issue of poor early student rollouts.

TRB replaces the early rollout policy with a teacher-guided behavior policy that is constrained to stay within a student-centered KL trust region, while keeping the per-prefix reverse-KL OPD loss unchanged.

The KL budget is annealed to zero over a warmup horizon, so training returns to pure student rollouts.

The behavior policy is defined as the distribution closest to the teacher within a KL budget around the student, with a closed-form solution involving a parameter β that is found via binary search.

Experiments on two math-reasoning settings (Qwen3-1.7B-Base ← Qwen3-8B and Qwen3-0.6B-Base ← Qwen3-4B) show TRB achieves the strongest average pass@1 compared to vanilla OPD, Veto, SKD, temperature warmup, SFT warmup, and fixed-ε blending.

TRB also shows lower teacher entropy during warmup and higher success rates on early prefixes.

The method is limited to two settings and increases training-time cost during warmup due to online teacher decoding.