The Year in AI Papers
The year/Independent research

Paper 2510.01051

GEM: A Gym for Agentic LLMs

Zichen Liu, Anya Sims, Keyu Duan, Changyu Chen, Simon Yu, Xiangxin Zhou, Haotian Xu, Shaopan Xiong, Bo Liu, Chenmien Tan, Chuen Yang Beh, Weixun Wang, et al.

Published
Oct 2025
Research lab
Independent
Citations
11
GitHub
504 stars

01 In brief

Summary

GEM (General Experience Maker) is an open-source environment simulator for agentic LLMs, analogous to OpenAI-Gym for traditional RL.

It provides a standardized environment-agent interface with reset() and step(), asynchronous vectorized execution, and modular wrappers.

GEM includes diverse tasks (Math, Code, Game, QA, ReasoningGym, Terminal) and tools (Python, Search, MCP).

The paper introduces REINFORCE with Return Batch Normalization (ReBN), which normalizes returns over the batch, enabling multi-turn RL with dense per-turn rewards and arbitrary discount factors, unlike GRPO.

Benchmarking across 24 environments shows ReBN consistently outperforms vanilla REINFORCE and is competitive with or better than PPO and GRPO.

Experiments demonstrate that discount factor γ<1 encourages efficient strategies (e.g., binary search in GuessTheNumber), and tool integration (Python for math, Search for QA) significantly improves performance.

GEM integrates with five RL frameworks (Oat, Verl, OpenRLHF, ROLL, RL2) and serves as an evaluation toolkit, demonstrated on MCPMark and Terminal-Bench with GPT-5, Gemini-2.5-Pro, and Claude-Sonnet-4.

The framework aims to accelerate agentic LLM research by decoupling environments from training frameworks.

02 From the paper

Abstract

The training paradigm for large language models (LLMs) is moving from static datasets to experience-based learning, where agents acquire skills via interacting with complex environments. To facilitate this transition we introduce GEM (General Experience Maker), an open-source environment simulator designed for the age of LLMs. Analogous to OpenAI-Gym for traditional reinforcement learning (RL), GEM provides a standardized framework for the environment-agent interface, including asynchronous vectorized execution for high throughput, and flexible wrappers for easy extensibility. GEM also features a diverse suite of environments, robust integrated tools, and single-file example scripts demonstrating using GEM with five popular RL training frameworks. Along with this, we also provide a set of baselines across 24 environments using REINFORCE with Return Batch Normalization (ReBN), which -- unlike GRPO -- is compatible with the full RL setting of dense per-turn rewards and offers better credit assignment. We further conduct apple-to-apple benchmarking of PPO, GRPO and REINFORCE in both single- and multi-turn settings using GEM to shed light on the algorithmic designs. Lastly, GEM also functions as a convenient evaluation toolkit besides a training environment. We hope this framework can help accelerate future agentic LLM research.