The Year in AI Papers
The year/Independent research

Paper 2511.03773

Scaling Agent Learning via Experience Synthesis

Zhaorun Chen, Zhuokai Zhao, Kai Zhang, Bo Liu, Qi Qi, Yifan Wu, Tarun Kalluri, Sara Cao, Yuanhao Xiong, Haibo Tong, Huaxiu Yao, Hengduo Li, et al.

Published
Nov 2025
Research lab
Independent
Citations
24
GitHub
Not linked

01 In brief

Summary

DreamGym is a unified framework for scaling reinforcement learning (RL) for LLM-based agents by synthesizing diverse experiences, addressing the high costs and limitations of real-environment rollouts.

It uses a reasoning-based experience model that operates in an abstract textual state space, generating consistent state transitions and reward signals via chain-of-thought reasoning.

An experience replay buffer, seeded with offline data and enriched online, grounds predictions, while a curriculum task generator creates progressively challenging tasks based on reward entropy.

Experiments across WebShop, ALFWorld, and WebArena with Llama and Qwen backbones show DreamGym matches or exceeds traditional RL (GRPO, PPO) using only synthetic interactions, and outperforms baselines by over 30% on non-RL-ready WebArena.

The sim-to-real variant (DreamGym-S2R) achieves over 40% improvement with less than 10% of real data, providing a scalable warm-start.

Ablations confirm the importance of history, reasoning, and task generation for consistency, informativeness, and performance.

02 From the paper

Abstract

While reinforcement learning (RL) can empower autonomous agents by enabling self-improvement through interaction, its practical adoption remains challenging due to costly rollouts, limited task diversity, unreliable reward signals, and infrastructure complexity, all of which obstruct the collection of scalable experience data. To address these challenges, we introduce DreamGym, the first unified framework designed to synthesize diverse experiences with scalability in mind to enable effective online RL training for autonomous agents. Rather than relying on expensive real-environment rollouts, DreamGym distills environment dynamics into a reasoning-based experience model that derives consistent state transitions and feedback signals through step-by-step reasoning, enabling scalable agent rollout collection for RL. To improve the stability and quality of transitions, DreamGym leverages an experience replay buffer initialized with offline real-world data and continuously enriched with fresh interactions to actively support agent training. To improve knowledge acquisition, DreamGym adaptively generates new tasks that challenge the current agent policy, enabling more effective online curriculum learning. Experiments across diverse environments and agent backbones demonstrate that DreamGym substantially improves RL training, both in fully synthetic settings and in sim-to-real transfer scenarios. On non-RL-ready tasks like WebArena, DreamGym outperforms all baselines by over 30%. And in RL-ready but costly settings, it matches GRPO and PPO performance using only synthetic interactions. When transferring a policy trained purely on synthetic experiences to real-environment RL, DreamGym yields significant additional performance gains while requiring far fewer real-world interactions, providing a scalable warm-start strategy for general-purpose RL.