Paper 2603.16856
Online Experiential Learning for Language Models
- Published
- Mar 2026
- Research lab
- Independent
- Citations
- 20
- GitHub
- Not linked
01 In brief
Summary
The paper introduces Online Experiential Learning (OEL), a framework for large language models to continuously improve from their own deployment experience without human annotations, reward models, or access to user-side environments.
OEL operates in two iterated stages: first, transferable experiential knowledge is extracted and accumulated from user-side interaction trajectories; second, this knowledge is consolidated into model parameters via on-policy context distillation, using reverse KL divergence to match a knowledge-conditioned teacher.
The process is reward-free and forms an online learning loop where improved models collect higher-quality trajectories.
Evaluations on text-based games (Frozen Lake, Sokoban) across multiple model scales and thinking/non-thinking variants show consistent improvements in pass rate and token efficiency over successive iterations, while preserving out-of-distribution performance.
Analysis shows extracted experiential knowledge outperforms raw trajectories, and on-policy consistency between knowledge source and policy model is critical.
OEL also mitigates catastrophic forgetting compared to off-policy distillation.
02 From the paper
Abstract
The prevailing paradigm for improving large language models relies on offline training with human annotations or simulated environments, leaving the rich experience accumulated during real-world deployment entirely unexploited. We propose Online Experiential Learning (OEL), a framework that enables language models to continuously improve from their own deployment experience. OEL operates in two stages: first, transferable experiential knowledge is extracted and accumulated from interaction trajectories collected on the user side; second, this knowledge is consolidated into model parameters via on-policy context distillation, requiring no access to the user-side environment. The two stages are iterated to form an online learning loop, where the improved model collects higher-quality trajectories that yield richer experiential knowledge for subsequent rounds. We evaluate OEL on text-based game environments across multiple model scales and both thinking and non-thinking variants. OEL achieves consistent improvements over successive iterations, enhancing both task accuracy and token efficiency while preserving out-of-distribution performance. Our analysis further shows that extracted experiential knowledge is significantly more effective than raw trajectories, and that on-policy consistency between the knowledge source and the policy model is critical for effective learning.