Paper 2607.14777
SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning
- Published
- Jul 2026
- Research lab
- Independent
- Citations
- 3
- GitHub
- 218 stars
01 In brief
Summary
SEED (SElf-Evolving On-Policy Distillation) is a framework for agentic reinforcement learning that converts completed on-policy trajectories into natural-language hindsight skills and distills their behavioral effect back into the policy model.
It addresses the supervision gap between sparse trajectory-level rewards and token-level policy learning in long-horizon tasks.
SEED has two stages: first, hindsight-skill supervised fine-tuning equips the policy to analyze trajectories and generate skills; second, self-evolving on-policy distillation uses the latest policy checkpoint as both actor and analyzer, re-scoring sampled actions under ordinary and skill-augmented contexts to produce a dense token-level distillation signal, jointly optimized with outcome-based RL (GRPO).
The skills are used only during training, requiring no external memory or prompts at inference.
Experiments on ALFWorld, WebShop, and Search-based QA across three backbones (Qwen2.5-3B, Qwen2.5-7B, Qwen3-1.7B) show SEED consistently outperforms baselines like GRPO, Skill-GRPO, OPSD, and SDAR, improving performance, sample efficiency, and cross-domain generalization.
Ablations confirm the importance of each component: hindsight-skill SFT, self-evolving OPD, and on-policy skills.
Theoretical analysis formalizes the on-policy, dense, and self-evolving properties of the supervision.
02 From the paper
Abstract
Large language models are increasingly trained as interactive agents for long-horizon tasks involving multi-turn interaction, tool use, and environment feedback. Outcome-based reinforcement learning (RL) provides a practical optimization paradigm, but its sparse trajectory-level rewards offer limited guidance on intermediate decisions, leaving a supervision gap between episode-level outcomes and token-level policy learning. We propose SEED (SElf-Evolving On-Policy Distillation), a self-evolving framework that converts completed on-policy trajectories into training-time hindsight skills and distills their behavioral effect back into the policy model. SEED first fine-tunes the policy to analyze completed trajectories and generate natural-language skills that capture reusable workflows, decisive observations, or failure-avoidance rules. During RL, the current policy both collects trajectories and serves as the analyzer that extracts hindsight skills from them. Policy updates therefore improve subsequent decision making and skill analysis together, allowing hindsight supervision to evolve with the policy. SEED then re-scores the sampled actions under ordinary and skill-augmented contexts, converting the skill-induced probability shift into a dense token-level on-policy distillation signal. This signal is jointly optimized with outcome-based RL, keeping the auxiliary supervision aligned with the current trajectory distribution. Extensive experiments on text-based and vision-based agentic tasks show that SEED consistently improves performance and sample efficiency, exhibiting robust generalization to unseen scenarios. Our code is available at https://github.com/jinyangwu/SEED.