Paper 2605.11609
Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information
- Published
- May 2026
- Research lab
- Independent
- Citations
- 6
- GitHub
- 33 stars
01 In brief
Summary
The paper identifies a structural bias in on-policy self-distillation for math reasoning, where the per-token signal (conditional pointwise mutual information between the next token and privileged context) rewards shortcut tokens (e.g., 'Given', 'succeeds') and penalizes deliberation tokens (e.g., 'Wait', 'Let').
To fix this, the authors propose Anti-Self-Distillation (AntiSD), which ascends Jensen-Shannon divergence between student and teacher instead of descending KL divergence, reversing the per-token sign.
An entropy-triggered gate disables the term when teacher entropy collapses.
Across five models (4B-30B) on math benchmarks, AntiSD reaches GRPO baseline accuracy in 2-10x fewer steps and improves final accuracy by up to 11.5 points, while default self-distillation underperforms GRPO.
Ablations show sign reversal is key, the gate is essential for Qwen models, and continual AntiSD from a GRPO checkpoint nearly matches from-base performance.
The method also shows small gains on code reasoning tasks.
The PMI characterization is a local per-step analysis, and extensions to multi-turn settings and broader benchmarks are suggested as future work.
The paper concludes that AntiSD offers a scalable path to self-improvement by bootstrapping reasoning through the training signal itself.
The method is a drop-in replacement for default self-distillation with no additional cost.
The authors also note that the gate threshold is auto-calibrated and shared across models, requiring no per-model tuning.
The main limitation is that the analysis is local, not a global optimum statement, and evaluation focuses on math reasoning.
The broader impacts include potential dual-use of stronger reasoning models, but no new attack surface is introduced.
The paper includes…
02 From the paper
Abstract
On-policy self-distillation, where a student is pulled toward a copy of itself conditioned on privileged context (e.g., a verified solution or feedback), offers a promising direction for advancing reasoning capability without a stronger external teacher. Yet in math reasoning the gains are inconsistent, even when the same approach succeeds elsewhere. A pointwise mutual information analysis traces the failure to the privileged context itself: it inflates the teacher's confidence on tokens already implied by the solution (structural connectives, verifiable claims) and deflates it on deliberation tokens ("Wait", "Let", "Maybe") that drive multi-step search. We propose Anti-Self-Distillation (AntiSD), which ascends a divergence between student and teacher rather than descending it: this reverses the per-token sign and yields a naturally bounded advantage in one step. An entropy-triggered gate disables the term once the teacher entropy collapses, completing a drop-in replacement for default self-distillation. Across five models from 4B to 30B parameters on math reasoning benchmarks, AntiSD reaches the GRPO baseline's accuracy in 2 to 10x fewer training steps and improves final accuracy by up to 11.5 points. AntiSD opens a path to scalable self-improvement, where a language model bootstraps its own reasoning through its training signal.