The Year in AI Papers
The year/Independent research

Paper 2510.14545

Agentic Entropy-Balanced Policy Optimization

Guanting Dong, Licheng Bao, Zhongyuan Wang, Kangzhi Zhao, Xiaoxi Li, Jiajie Jin, Jinghan Yang, Hangyu Mao, Fuzheng Zhang, Kun Gai, Guorui Zhou, Yutao Zhu, et al.

Published
Oct 2025
Research lab
Independent
Citations
26
GitHub
1.1K stars

01 In brief

Summary

The paper introduces Agentic Entropy-Balanced Policy Optimization (AEPO), an agentic reinforcement learning algorithm for training multi-turn web agents.

It addresses two entropy-driven challenges: 'High-Entropy Rollout Collapse' (over-branching on few trajectories due to consecutive high-entropy tool-call steps) and 'High-Entropy Token Gradient Clipping' (vanilla RL clipping gradients of high-entropy exploratory tokens).

AEPO comprises two components: (1) a dynamic entropy-balanced rollout mechanism that uses entropy pre-monitoring to adaptively allocate global and branch sampling budgets, and applies a branch penalty on consecutive high-entropy steps; (2) Entropy-Balanced Policy Optimization that inserts a stop-gradient operation into the high-entropy clipping term to preserve and rescale gradients on high-entropy tokens, and uses entropy-aware advantage estimation to prioritize learning on high-uncertainty tokens.

Experiments across 14 datasets show AEPO consistently outperforms 7 mainstream RL algorithms.

With only 1K RL samples, Qwen3-14B with AEPO achieves 47.6% on GAIA, 11.2% on Humanity's Last Exam, and 43.0% on WebWalkerQA for Pass@1; and 65.0%, 26.0%, and 70.0% for Pass@5 respectively.

AEPO improves rollout sampling diversity while maintaining stable policy entropy, enabling scalable web agent training.

02 From the paper

Abstract

Recently, Agentic Reinforcement Learning (Agentic RL) has made significant progress in incentivizing the multi-turn, long-horizon tool-use capabilities of web agents. While mainstream agentic RL algorithms autonomously explore high-uncertainty tool-call steps under the guidance of entropy, excessive reliance on entropy signals can impose further constraints, leading to the training collapse. In this paper, we delve into the challenges caused by entropy and propose the Agentic Entropy-Balanced Policy Optimization (AEPO), an agentic RL algorithm designed to balance entropy in both the rollout and policy update phases. AEPO comprises two core components: (1) a dynamic entropy-balanced rollout mechanism that adaptively allocate global and branch sampling budget through entropy pre-monitoring, while imposing a branch penalty on consecutive high-entropy tool-call steps to prevent over-branching issues; and (2) Entropy-Balanced Policy Optimization that inserts a stop-gradient operation into the high-entropy clipping term to preserve and properly rescale gradients on high-entropy tokens, while incorporating entropy-aware advantage estimation to prioritize learning on high-uncertainty tokens. Results across 14 challenging datasets show that AEPO consistently outperforms 7 mainstream RL algorithms. With just 1K RL samples, Qwen3-14B with AEPO achieves impressive results: 47.6% on GAIA, 11.2% on Humanity's Last Exam, and 43.0% on WebWalker for Pass@1; 65.0% on GAIA, 26.0% on Humanity's Last Exam, and 70.0% on WebWalker for Pass@5. Further analysis reveals that AEPO improves rollout sampling diversity while maintaining stable policy entropy, facilitating scalable web agent training.