Paper 2509.08755
AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning
- Published
- Sep 2025
- Research lab
- Independent
- Citations
- 62
- GitHub
- 834 stars
01 In brief
Summary
AgentGym-RL is a unified, modular reinforcement learning framework for training LLM agents in multi-turn, long-horizon decision-making tasks, without requiring supervised fine-tuning.
It supports diverse environments (web navigation, deep search, digital games, embodied tasks, scientific tasks) and mainstream RL algorithms (PPO, GRPO, REINFORCE++, RLOO).
The framework features a decoupled architecture with environment, agent, and training modules, and includes engineering optimizations for scalability and reliability.
ScalingInter-RL, a proposed training method, progressively increases the interaction horizon during training, balancing exploration and exploitation to improve stability and performance.
Experiments show that a 7B-parameter model trained with AgentGym-RL and ScalingInter-RL achieves an average success rate of 58.6% across 27 tasks, matching or surpassing larger proprietary models like OpenAI o3 and Gemini-2.5-Pro.
Key findings include: RL generally improves agentic intelligence, ScalingInter-RL consistently boosts performance (e.g., +10% on WebArena), large interaction budgets cause training collapse, and post-training compute is more impactful than model size.
The framework is open-sourced to support community research.
02 From the paper
Abstract
Developing autonomous LLM agents capable of making a series of intelligent decisions to solve complex, real-world tasks is a fast-evolving frontier. Like human cognitive development, agents are expected to acquire knowledge and skills through exploration and interaction with the environment. Despite advances, the community still lacks a unified, interactive reinforcement learning (RL) framework that can effectively train such agents from scratch -- without relying on supervised fine-tuning (SFT) -- across diverse and realistic environments. To bridge this gap, we introduce AgentGym-RL, a new framework to train LLM agents for multi-turn interactive decision-making through RL. The framework features a modular and decoupled architecture, ensuring high flexibility and extensibility. It encompasses a wide variety of real-world scenarios, and supports mainstream RL algorithms. Furthermore, we propose ScalingInter-RL, a training approach designed for exploration-exploitation balance and stable RL optimization. In early stages, it emphasizes exploitation by restricting the number of interactions, and gradually shifts towards exploration with larger horizons to encourage diverse problem-solving strategies. In this way, the agent develops more diverse behaviors and is less prone to collapse under long horizons. We perform extensive experiments to validate the stability and effectiveness of both the AgentGym-RL framework and the ScalingInter-RL approach. Our agents match or surpass commercial models on 27 tasks across diverse environments. We offer key insights and will open-source the complete AgentGym-RL framework -- including code and datasets -- to empower the research community in developing the next generation of intelligent agents.