The Year in AI Papers
The year/Independent research

Paper 2510.18927

BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping

Zhiheng Xi, Xin Guo, Yang Nan, Enyu Zhou, Junrui Shen, Wenxiang Chen, Jiaqi Liu, Jixuan Huang, Zhihao Zhang, Honglin Guo, Xun Deng, Zhikai Lei, et al.

Published
Oct 2025
Research lab
Independent
Citations
48
GitHub
94 stars

01 In brief

Summary

This paper introduces BAPO (Balanced Policy Optimization with Adaptive Clipping), a method to stabilize off-policy reinforcement learning for large language models (LLMs).

Off-policy RL, where stale data from past policies is used, suffers from unstable optimization, entropy collapse, and gradient explosions.

The authors identify two key issues: (1) an imbalance where negative-advantage samples dominate the policy gradient, and (2) the Entropy-Clip Rule, showing that fixed clipping in PPO-like objectives blocks entropy-increasing updates.

BAPO dynamically adjusts clipping bounds to re-balance positive and negative contributions, preserve entropy, and stabilize training.

Experiments on AIME 2024 and AIME 2025 benchmarks show that a 7B BAPO model achieves 70.8 and 62.5, surpassing open-source counterparts like SkyWork-OR1-7B, while a 32B model achieves 87.1 and 80.0, outperforming proprietary systems like o3-mini and Gemini-2.5-Flash-Thinking.

BAPO also shows robustness in partial rollout settings and on Llama-based models.

02 From the paper

Abstract

Reinforcement learning (RL) has recently become the core paradigm for aligning and strengthening large language models (LLMs). Yet, applying RL in off-policy settings--where stale data from past policies are used for training--improves sample efficiency, but remains challenging: policy entropy declines sharply, optimization often becomes unstable and may even collapse. Through theoretical and empirical analysis, we identify two key insights: (i) an imbalance in optimization, where negative-advantage samples dominate the policy gradient, suppressing useful behaviors and risking gradient explosions; and (ii) the derived Entropy-Clip Rule, which reveals that the fixed clipping mechanism in PPO-like objectives systematically blocks entropy-increasing updates, thereby driving the policy toward over-exploitation at the expense of exploration. Building on these insights, we propose BAlanced Policy Optimization with Adaptive Clipping (BAPO), a simple yet effective method that dynamically adjusts clipping bounds to adaptively re-balance positive and negative contributions, preserve entropy, and stabilize RL optimization. Across diverse off-policy scenarios--including sample replay and partial rollout--BAPO achieves fast, stable, and data-efficient training. On AIME 2024 and AIME 2025 benchmarks, our 7B BAPO model surpasses open-source counterparts such as SkyWork-OR1-7B, while our 32B BAPO model not only achieves state-of-the-art results among models of the same scale but also outperforms leading proprietary systems like o3-mini and Gemini-2.5-Flash-Thinking.