Paper 2509.22576
EPO: Entropy-regularized Policy Optimization for LLM Agents Reinforcement Learning
- Published
- Sep 2025
- Research lab
- Independent
- Citations
- 10
- GitHub
- 40 stars
01 In brief
Summary
The paper identifies the exploration-exploitation cascade failure in multi-turn LLM agent reinforcement learning, where shared policy parameters across turns and sparse terminal rewards cause severe entropy oscillations that destabilize training.
To address this, the authors propose Entropy-regularized Policy Optimization (EPO), which combines trajectory-level entropy regularization, an entropy smoothing regularizer that penalizes deviations from historical entropy averages, and adaptive phase-based weighting.
EPO is validated on ScienceWorld and ALFWorld benchmarks, achieving up to 152% improvement in success rate on ScienceWorld and 19.8% on ALFWorld compared to baselines, while transforming unstable training into smooth convergence.
Theoretical analysis shows EPO reduces cumulative entropy deviation from O(T^2) to O(T) and provides a performance bound with zero stability cost under moderate entropy conditions.
The method is compatible with on-policy algorithms like PPO and GRPO, and code is available on GitHub.
The work demonstrates that multi-turn settings require fundamentally different entropy control than traditional RL, offering a general framework for stable training of LLM agents in long-horizon sparse-reward tasks.
The authors also discuss limitations, including the lack of memory mechanisms and potential extensions to vision-language models, and note that LLMs were used only for ancillary support, not for core research contributions.
The paper includes detailed appendices with proofs, experimental setups, and system prompts, and emphasizes the importance of sustained exploration over conventional exploration-to-exploitation scheduling in multi-turn tasks.
The findings highlight that EPO's decoupled entropy regularization with temporal smoothing is superior to advantage reshaping for maintaining long-horizon policy stability, and the adaptive weighting scheme accelerates convergence without sacrificing…
02 From the paper
Abstract
Training LLM agents in multi-turn environments with sparse rewards, where completing a single task requires 30+ turns of interaction within an episode, presents a fundamental challenge for reinforcement learning. We identify a critical failure mode unique to this setting: the exploration-exploitation cascade failure. This cascade begins with early-stage policy premature convergence, where sparse feedback causes agents to commit to flawed, low-entropy strategies. Subsequently, agents enter late-stage policy collapse, where conventional entropy regularization becomes counterproductive, promoting chaotic exploration that destabilizes training. We propose Entropy-regularized Policy Optimization (EPO), a general framework that breaks this failure cycle through three synergistic mechanisms: (1) adopting entropy regularization in multi-turn settings to enhance exploration, (2) an entropy smoothing regularizer that bounds policy entropy within historical averages to prevent abrupt fluctuations, and (3) adaptive phase-based weighting that balances exploration and exploitation across training. Our analysis justifies that EPO guarantees monotonically decreasing entropy variance while maintaining convergence. EPO achieves up to 152% performance improvement on ScienceWorld and up to 19.8% on ALFWorld. Our work demonstrates that multi-turn sparse-reward settings require fundamentally different entropy control than traditional RL, with broad implications for LLM agent training.