Paper 2510.02245
ExGRPO: Learning to Reason from Experience
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 41
- GitHub
- Not linked
01 In brief
Summary
The paper introduces ExGRPO, a framework for improving reinforcement learning from verifiable rewards (RLVR) for large reasoning models by reusing past rollout experiences.
The authors first analyze what makes a reasoning experience valuable, identifying that medium-difficulty questions (with rollout correctness between 25% and 75%) and low-entropy trajectories are most beneficial.
ExGRPO manages a replay buffer partitioned by correctness, samples questions using a Gaussian distribution centered at 0.5, and selects the lowest-entropy trajectory per question.
It uses a mixed-policy objective combining on-policy GRPO with off-policy replay, corrected by importance weighting and policy shaping, plus a delayed start to ensure experience quality.
Experiments on five backbone models (1.5B-8B parameters) show ExGRPO consistently outperforms on-policy RLVR, with average gains of +3.5 points on in-distribution and +7.6 points on out-of-distribution benchmarks.
Notably, ExGRPO stabilizes training on weaker models like Llama-3.1 8B and on continual learning with LUFFY, where on-policy methods collapse.
Ablations confirm the importance of question and trajectory selection, policy shaping, and a balanced experience ratio (ρ=50%).
The method also extends to continuous rewards, showing improvements on chat and instruction-following benchmarks.
02 From the paper
Abstract
Reinforcement learning from verifiable rewards (RLVR) is an emerging paradigm for improving the reasoning ability of large language models. However, standard on-policy training discards rollout experiences after a single update, leading to computational inefficiency and instability. While prior work on RL has highlighted the benefits of reusing past experience, the role of experience characteristics in shaping learning dynamics of large reasoning models remains underexplored. In this paper, we are the first to investigate what makes a reasoning experience valuable and identify rollout correctness and entropy as effective indicators of experience value. Based on these insights, we propose ExGRPO (Experiential Group Relative Policy Optimization), a framework that organizes and prioritizes valuable experiences, and employs a mixed-policy objective to balance exploration with experience exploitation. Experiments on five backbone models (1.5B-8B parameters) show that ExGRPO consistently improves reasoning performance on mathematical/general benchmarks, with an average gain of +3.5/7.6 points over on-policy RLVR. Moreover, ExGRPO stabilizes training on both stronger and weaker models where on-policy methods fail. These results highlight principled experience management as a key ingredient for efficient and scalable RLVR.