Paper 2602.13949
Experiential Reinforcement Learning
- Published
- Feb 2026
- Research lab
- Independent
- Citations
- 12
- GitHub
- 76 stars
01 In brief
Summary
The paper introduces Experiential Reinforcement Learning (ERL), a training paradigm for language models that embeds an explicit experience–reflection–consolidation loop into reinforcement learning.
In ERL, the model first generates an initial attempt, receives environmental feedback, and then produces a self-reflection that guides a refined second attempt.
The second attempt's success is reinforced via policy gradients and internalized into the base policy through selective distillation, allowing the model to reproduce improved behavior without reflection at inference time.
A cross-episode memory stores successful reflections to stabilize training.
ERL is evaluated against standard RLVR (using GRPO) on FrozenLake, Sokoban, and HotpotQA with two models (Qwen3-4B-Instruct and Olmo3-7B-Instruct).
Results show consistent improvements: up to +81% in Sokoban, +27% in FrozenLake, and +11% in HotpotQA.
Ablations show that removing reflection causes the largest performance drops, while removing memory slows convergence, though in one setting (Olmo3-7B on Sokoban) the no-memory variant slightly outperforms full ERL.
The authors argue ERL improves learning efficiency and final performance by converting feedback into structured behavioral corrections.
02 From the paper
Abstract
Reinforcement learning has become the central approach for language models (LMs) to learn from environmental reward or feedback. In practice, the environmental feedback is usually sparse and delayed. Learning from such signals is challenging, as LMs must implicitly infer how observed failures should translate into behavioral changes for future iterations. We introduce Experiential Reinforcement Learning (ERL), a training paradigm that embeds an explicit experience-reflection-consolidation loop into the reinforcement learning process. Given a task, the model generates an initial attempt, receives environmental feedback, and produces a reflection that guides a refined second attempt, whose success is reinforced and internalized into the base policy. This process converts feedback into structured behavioral revision, improving exploration and stabilizing optimization while preserving gains at deployment without additional inference cost. Across sparse-reward control environments and agentic reasoning benchmarks, ERL consistently improves learning efficiency and final performance over strong reinforcement learning baselines, achieving gains of up to +81% in complex multi-step environments and up to +11% in tool-using reasoning tasks. These results suggest that integrating explicit self-reflection into policy training provides a practical mechanism for transforming feedback into durable behavioral improvement.