Paper 2604.27083
Co-Evolving Policy Distillation
- Published
- Apr 2026
- Research lab
- Independent
- Citations
- 3
- GitHub
- Not linked
01 In brief
Summary
The paper proposes Co-Evolving Policy Distillation (CoPD) to consolidate multiple expert capabilities into a single model.
It analyzes two standard paradigms: mixed RLVR, which suffers from capability divergence cost, and the static pipeline of training experts then performing on-policy distillation (OPD), which fails to fully absorb teacher capabilities due to large behavioral gaps.
A pilot study shows OPD gain increases with teacher-student top-k token overlap, but standard RLVR training drives overlap down.
CoPD alternates branch-specific RLVR with mutual OPD during training, keeping branches behaviorally close while maintaining complementary knowledge.
Experiments on Qwen3-VL-4B show CoPD outperforms mixed RLVR and static OPD/MOPD baselines in two-branch (text, image) and three-branch (text, image, video) settings, even surpassing domain-specific experts.
Ablations confirm the necessity of bidirectional distillation and continuous co-evolution.
The authors suggest model parallel training as a potential scaling paradigm.
02 From the paper
Abstract
RLVR and OPD have become standard paradigms for post-training. We provide a unified analysis of these two paradigms in consolidating multiple expert capabilities into a single model, identifying capability loss in different ways: mixed RLVR suffers from inter-capability divergence cost, while the pipeline of first training experts and then performing OPD, though avoiding divergence, fails to fully absorb teacher capabilities due to large behavioral pattern gaps between teacher and student. We propose Co-Evolving Policy Distillation (CoPD), which encourages parallel training of experts and introduces OPD during each expert's ongoing RLVR training rather than after complete expert training, with experts serving as mutual teachers (making OPD bidirectional) to co-evolve. This enables more consistent behavioral patterns among experts while maintaining sufficient complementary knowledge throughout. Experiments validate that CoPD achieves all-in-one integration of text, image, and video reasoning capabilities, significantly outperforming strong baselines such as mixed RLVR and MOPD, and even surpassing domain-specific experts. The model parallel training pattern offered by CoPD may inspire a novel training scaling paradigm.