Paper 2508.03680
Agent Lightning: Train ANY AI Agents with Reinforcement Learning
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 58
- GitHub
- 17K stars
01 In brief
Summary
Agent Lightning is a framework that enables reinforcement learning (RL) training of large language models (LLMs) for any AI agent, achieving complete decoupling between agent execution and training with almost zero code modifications.
It formulates agent execution as a Markov decision process (MDP), defining a unified data interface that abstracts agent orchestration logic.
The hierarchical RL algorithm, LightningRL, decomposes agent trajectories into transitions, assigning credit via a module and integrating with existing single-turn RL methods.
The system uses a Training-Agent Disaggregation architecture with a Lightning Server (managing training) and Lightning Client (agent runtime), leveraging observability frameworks like OpenTelemetry for data capture and an Automatic Intermediate Rewarding (AIR) mechanism.
Experiments on text-to-SQL (LangChain), retrieval-augmented generation (OpenAI Agents SDK), and math tool-use (AutoGen) tasks show stable performance improvements.
The framework supports selective optimization of agents in multi-agent systems and avoids issues of long context concatenation and masking, offering a scalable and flexible solution for real-world agent training and deployment.
Future work includes more optimization methods, improved RL algorithms, and system infrastructure advancements.
02 From the paper
Abstract
We present Agent Lightning, a flexible and extensible framework that enables Reinforcement Learning (RL)-based training of Large Language Models (LLMs) for any AI agent. Unlike existing methods that tightly couple RL training with agent or rely on sequence concatenation with masking, Agent Lightning achieves complete decoupling between agent execution and training, allowing seamless integration with existing agents developed via diverse ways (e.g., using frameworks like LangChain, OpenAI Agents SDK, AutoGen, and building from scratch) with almost ZERO code modifications. By formulating agent execution as Markov decision process, we define an unified data interface and propose a hierarchical RL algorithm, LightningRL, which contains a credit assignment module, allowing us to decompose trajectories generated by ANY agents into training transition. This enables RL to handle complex interaction logic, such as multi-agent scenarios and dynamic workflows. For the system design, we introduce a Training-Agent Disaggregation architecture, and brings agent observability frameworks into agent runtime, providing a standardized agent finetuning interface. Experiments across text-to-SQL, retrieval-augmented generation, and math tool-use tasks demonstrate stable, continuous improvements, showcasing the framework's potential for real-world agent training and deployment.