The Year in AI Papers
The year/Independent research

Paper 2510.08558

Agent Learning via Early Experience

Kai Zhang, Xiangchao Chen, Bo Liu, Tianci Xue, Zeyi Liao, Zhihan Liu, Xiyao Wang, Yuting Ning, Zhaorun Chen, Xiaohan Fu, Jian Xie, Yuxuan Sun, et al.

Published
Oct 2025
Research lab
Independent
Citations
57
GitHub
136 stars

01 In brief

Summary

The paper introduces the 'early experience' paradigm for training language agents, which uses the agent's own actions and resulting future states as supervision without external reward signals.

This addresses limitations of both supervised fine-tuning (SFT) on expert data, which is costly and generalizes poorly, and reinforcement learning (RL), which is difficult in environments lacking verifiable rewards.

Two strategies are proposed: implicit world modeling (IWM), where the policy predicts future states from its actions to internalize environment dynamics, and self-reflection (SR), where the agent compares its actions to expert demonstrations and generates rationales for better decisions.

Evaluations across eight diverse environments (e.g., ALFWorld, WebShop, TravelPlanner) and multiple model families (Llama-3.2-3B, Qwen-2.5-7B, Llama-3.1-8B) show consistent improvements over imitation learning in effectiveness, out-of-domain generalization, and data efficiency.

Furthermore, initializing RL (GRPO) with early-experience checkpoints yields higher post-RL performance than imitation-learning warm starts, demonstrating early experience as a practical bridge to the 'era of experience'.

02 From the paper

Abstract

A long-term goal of language agents is to learn and improve through their own experience, ultimately outperforming humans in complex, real-world tasks. However, training agents from experience data with reinforcement learning remains difficult in many environments, which either lack verifiable rewards (e.g., websites) or require inefficient long-horizon rollouts (e.g., multi-turn tool use). As a result, most current agents rely on supervised fine-tuning on expert data, which is challenging to scale and generalizes poorly. This limitation stems from the nature of expert demonstrations: they capture only a narrow range of scenarios, and expose the agent to limited environment diversity. We address this limitation with a middle-ground paradigm we call early experience: interaction data generated by the agent's own actions, where the resulting future states serve as supervision without reward signals. Within this paradigm, we study two strategies of using such data: (1) implicit world modeling, which uses collected states to ground the policy in environment dynamics; and (2) self-reflection, where the agent learns from its suboptimal actions to improve reasoning and decision-making. Evaluation across eight diverse environments and multiple model families shows that our approaches consistently improve effectiveness and out-of-domain generalization, highlighting the value of early experience. Moreover, in environments with verifiable rewards, our results provide promising signals that early experience offers a strong foundation for subsequent reinforcement learning, making it a practical bridge between imitation learning and fully experience-driven agents.