The Year in AI Papers
The year/Independent research

Paper 2602.08234

SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning

Peng Xia, Jianwen Chen, Hanyang Wang, Jiaqi Liu, Kaide Zeng, Yu Wang, Siwei Han, Yiyang Zhou, Xujiang Zhao, Haifeng Chen, Zeyu Zheng, Cihang Xie, et al.

Published
Feb 2026
Research lab
Independent
Citations
150
GitHub
928 stars

01 In brief

Summary

SKILLRL is a framework for LLM agents that improves policy learning by distilling raw interaction trajectories into a hierarchical skill library (SKILLBANK) and co-evolving it with the agent's policy during reinforcement learning.

It uses a teacher model to extract general and task-specific skills from both successful and failed episodes, achieving 10-20x token compression.

A cold-start SFT stage teaches the base model to use skills, followed by GRPO-based RL with recursive skill evolution: after each validation epoch, failed trajectories are analyzed to add or refine skills for underperforming task categories.

Experiments on ALFWorld, WebShop, and seven search-augmented QA tasks show SKILLRL outperforms strong baselines by over 15.3%, achieving 89.9% success on ALFWorld and 72.7% on WebShop, and surpassing larger closed-source models like GPT-4o and Gemini-2.5-Pro.

Ablations confirm the importance of hierarchical structure, skill abstraction over raw trajectories, cold-start SFT, and dynamic evolution.

The skill library grows from 55 to 100 skills during training, and SKILLRL reduces context length by about 10.3% compared to raw memory approaches while improving convergence speed and asymptotic performance.

02 From the paper

Abstract

Large Language Model (LLM) agents have shown stunning results in complex tasks, yet they often operate in isolation, failing to learn from past experiences. Existing memory-based methods primarily store raw trajectories, which are often redundant and noise-heavy. This prevents agents from extracting high-level, reusable behavioral patterns that are essential for generalization. In this paper, we propose SkillRL, a framework that bridges the gap between raw experience and policy improvement through automatic skill discovery and recursive evolution. Our approach introduces an experience-based distillation mechanism to build a hierarchical skill library SkillBank, an adaptive retrieval strategy for general and task-specific heuristics, and a recursive evolution mechanism that allows the skill library to co-evolve with the agent's policy during reinforcement learning. These innovations significantly reduce the token footprint while enhancing reasoning utility. Experimental results on ALFWorld, WebShop and seven search-augmented tasks demonstrate that SkillRL achieves state-of-the-art performance, outperforming strong baselines over 15.3% and maintaining robustness as task complexity increases. Code is available at this https://github.com/aiming-lab/SkillRL.