Paper 2608.01735
DAPD: Dual-Anchored Policy Distillation
- Published
- Aug 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- 31 stars
01 In brief
Summary
The paper identifies information asymmetry as the root cause of privilege illusion in on-policy self-distillation (OPSD) for language models.
In OPSD, a teacher conditioned on privileged information (e.g., a reference completion) supervises a student that lacks this information at inference, causing the student to behave as if the privileged information were available, degrading performance.
The authors propose Dual-Anchored Policy Distillation (DAPD) to address this.
DAPD consists of Dual-Path Anchoring (DPA), which introduces a self-conditioned distribution as a bridge to align reference and rollout behavior under matched information conditions, and Dual-Source Anchoring (DSA), which applies these paths in both reference-to-rollout and rollout-to-reference directions to balance reliable reference guidance with student-reachable rollout guidance.
Experiments on Qwen3 models (1.7B to 32B) across reasoning, coding, and instruction-following benchmarks show DAPD consistently outperforms OPSD, with gains of +2.00 points on average at 4B and +2.78 points at 32B.
Ablations confirm the necessity of both paths and sources, and a reference-free variant using dual rollouts shows further potential.
02 From the paper
Abstract
On-policy (self) distillation (OPSD) is increasingly adopted for language-model post-training. It strengthens the teacher with privileged information but can induce a privilege illusion: the student learns privilege-dependent behavior it cannot reproduce from its inference-time context, yet behaves as if the training-time privileged information remained available, ultimately degrading performance. In this paper, we identify information asymmetry between the privileged teacher and the student at inference as the root cause of this failure in OPSD. To resolve this asymmetry, we propose Dual-Anchored Policy Distillation (DAPD), a unified framework with two levels of anchoring. Dual-Path Anchoring (DPA) introduces a self-conditioned bridge and aligns reference and rollout behavior along two matched-information paths, preventing privilege-dependent behavior from being transferred to the inference-time student. Dual-Source Anchoring (DSA) applies these paths in both reference-to-rollout and rollout-to-reference directions, reducing reliance on privileged reference guidance while preserving correctness supervision. Extensive experiments show that DAPD significantly alleviates privilege illusion, outperforming OPSD on Qwen3-4B by +2.00 points on average across tasks. Notably, its gains persist across scales, reaching +2.69 at 4B and +2.78 at 32B.