The Year in AI Papers
The year/Independent research

Paper 2606.30626

DOPD: Dual On-policy Distillation

Xinlei Yu, Gen Li, Qingyi Si, Guibin Zhang, Yuqi Xu, Congcong Wang, Shuai Dong, Kaiwen Tuo, Xiangyu Zeng, Kaituo Feng, Qunzhong Wang, Yang Shi, et al.

Published
Jun 2026
Research lab
Independent
Citations
1
GitHub
Not linked

01 In brief

Summary

The paper introduces DOPD (Dual On-policy Distillation), an advantage-aware dual distillation paradigm for large language models (LLMs) and vision-language models (VLMs).

It addresses a failure mode called 'privilege illusion,' where adding privileged information to teacher or student policies creates an apparent performance gap that stems from information asymmetry rather than transferable capability.

DOPD dynamically routes token-level supervision between a privileged teacher and a privileged student policy based on the privilege advantage gap (absolute log-probability difference) and relative predicted probabilities.

Tokens are categorized into four regimes, each receiving tailored distillation: strong full-vocabulary teacher distillation for high-advantage tokens with high teacher confidence, light teacher distillation for low-advantage high-confidence tokens, weak self-regularization for low-advantage low-confidence tokens, and light self-distillation for high-advantage high-student-confidence tokens.

Experiments on Qwen3 and Qwen3-VL models show DOPD outperforms Vanilla OPD and other baselines across eight benchmarks each for LLM and VLM settings, improving average performance by 7.5 and 6.0 points respectively.

It also demonstrates better robustness, scalability across five model pairs, continual learning, out-of-distribution generalization, and training stability.

Ablations confirm the importance of the advantage-aware routing and adaptive divergence objectives.

02 From the paper

Abstract

On-policy distillation (OPD) offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals. To furnish high-quality supervision sources and thereby elevate the performance frontier of distillation, an intuitive direction is to infuse privileged information to either teacher or student itself. However, this additional input induces a potential failure mode we dub privilege illusion: a pattern that conflates the transferable capability gap that students are meant to close, and the information asymmetry gap that can only be mimicked but never replicated. This issue is further amplified by the inherent non-uniformity of token-level supervision, where only a small subset of tokens carries pivotal capability-bearing signals. To this end, we propose DOPD, an advantage-aware dual distillation paradigm that dynamically routes token-level supervision between privileged teacher and privileged student policies based on their advantage gap and relative probabilities. Each token receives supervision of different strength, objective, and strategy from either teacher or student itself, which transfers credible capability while simultaneously receiving auxiliary signals, to alleviate privilege illusion. Extensive experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that DOPD consistently outperforms Vanilla OPD and other counterparts. Further results on stability, robustness, continual learning, and out-of-distribution tasks validate its superiority.