The Year in AI Papers
The year/Qwen

Paper 2512.01374

Stabilizing Reinforcement Learning with LLMs: Formulation and Practices

Chujie Zheng, Kai Dang, Bowen Yu, Mingze Li, Huiqiang Jiang, Junrong Lin, Yuqiong Liu, Hao Lin, Chencan Wu, Feng Hu, An Yang, Jingren Zhou, et al.

Published
Dec 2025
Research lab
Qwen
Citations
35
GitHub
Not linked

01 In brief

Summary

This paper proposes a formulation for reinforcement learning (RL) with large language models (LLMs), showing that optimizing sequence-level rewards via token-level objectives is a first-order approximation that holds only when training–inference discrepancy and policy staleness are minimized.

The authors introduce MiniRL, a minimalist REINFORCE-based algorithm with importance sampling (IS) correction and clipping, and validate their theory through extensive experiments with a 30B Mixture-of-Experts (MoE) model.

Key findings: (1) For on-policy training, MiniRL with IS correction achieves the highest stability; (2) For off-policy training, combining clipping and Routing Replay (R2 or R3) is essential to prevent collapse, with R2 preferred for small off-policiness and R3 for larger off-policiness; (3) Once training is stabilized, different cold-start initializations converge to comparable final performance.

The paper also explains how expert routing in MoE models amplifies discrepancies, and how Routing Replay mitigates this by fixing routed experts during optimization, albeit with potential bias.

The authors conclude that stable training is decisive for scaling RL, and their recipes facilitate future research.

02 From the paper

Abstract

This paper proposes a novel formulation for reinforcement learning (RL) with large language models, explaining why and under what conditions the true sequence-level reward can be optimized via a surrogate token-level objective in policy gradient methods such as REINFORCE. Specifically, through a first-order approximation, we show that this surrogate becomes increasingly valid only when both the training-inference discrepancy and policy staleness are minimized. This insight provides a principled explanation for the crucial role of several widely adopted techniques in stabilizing RL training, including importance sampling correction, clipping, and particularly Routing Replay for Mixture-of-Experts (MoE) models. Through extensive experiments with a 30B MoE model totaling hundreds of thousands of GPU hours, we show that for on-policy training, the basic policy gradient algorithm with importance sampling correction achieves the highest training stability. When off-policy updates are introduced to accelerate convergence, combining clipping and Routing Replay becomes essential to mitigate the instability caused by policy staleness. Notably, once training is stabilized, prolonged optimization consistently yields comparable final performance regardless of cold-start initialization. We hope that the shared insights and the developed recipes for stable RL training will facilitate future research.