Paper 2605.15155
Self-Distilled Agentic Reinforcement Learning
- Published
- May 2026
- Research lab
- Independent
- Citations
- 13
- GitHub
- 328 stars
01 In brief
Summary
The paper introduces SDAR (Self-Distilled Agentic Reinforcement Learning), a method for post-training multi-turn LLM agents.
It addresses issues with On-Policy Self-Distillation (OPSD), which provides dense token-level guidance but suffers from multi-turn instability and asymmetric trust in privileged guidance (e.g., retrieved skills).
SDAR keeps GRPO as the primary optimization backbone and adds OPSD as a gated auxiliary objective.
A sigmoid gate, based on the teacher-student log-probability gap, modulates distillation per token: positive-gap tokens (teacher-endorsed) are strengthened, while negative-gap tokens (teacher rejections) are softly attenuated.
The gate is detached to ensure stable gradients.
Experiments on Qwen2.5-3B, Qwen2.5-7B, and Qwen3-1.7B across ALFWorld, WebShop, and Search-QA show SDAR improves over GRPO by +9.4% on ALFWorld, +7.0% on Search-QA, and +10.2% on WebShop-Acc (7B), avoids the instability of naive GRPO+OPSD, and outperforms hybrid baselines like Skill-SD and RLSD.
Ablations show gap gating is best, with optimal sharpness β=5 and distillation weight λ=0.01.
SDAR is robust to retrieval quality, even random retrieval outperforming GRPO.
The method internalizes skills, requiring no external skills at inference.
02 From the paper
Abstract
Reinforcement learning (RL) has emerged as a central paradigm for post-training LLM agents, yet its trajectory-level reward signal provides only coarse supervision for long-horizon interaction. On-Policy Self-Distillation (OPSD) complements RL by introducing dense token-level guidance from a teacher branch augmented with privileged context. However, transferring OPSD to multi-turn agents proves problematic: compounding multi-turn instability destabilizes supervision, while skill-conditioned privileged guidance requires asymmetric treatment for negative teacher rejections may arise from imperfect skills retrieval or utilization. We introduce SDAR (Self-Distilled Agentic Reinforcement Learning), which treats OPSD as a gated auxiliary objective while keeping RL as the primary optimization backbone. SDAR maps detached token-level signals into a sigmoid gate, strengthening distillation on teacher-endorsed positive-gap tokens and softly attenuating negative teacher rejections. Across the Qwen2.5 and Qwen3 families on ALFWorld, WebShop, and Search-QA, SDAR substantially improves over GRPO (+9.4% on ALFWorld, +7.0% on Search-QA, +10.2% on WebShop-Acc), avoids the instability of naive GRPO+OPSD, and consistently outperforms hybrid RL--OPSD baselines across model scales.