The Year in AI Papers
The year/Independent research

Paper 2601.08521

Your Group-Relative Advantage Is Biased

Fengkai Yang, Zherui Chen, Xiaohan Wang, Xiaodong Lu, Jiajun Chai, Guojun Yin, Wei Lin, Shuai Ma, Fuzhen Zhuang, Deqing Wang, Yaodong Yang, Jianxin Li, et al.

Published
Jan 2026
Research lab
Independent
Citations
27
GitHub
Not linked

01 In brief

Summary

This paper identifies a fundamental bias in group-relative advantage estimation used by RLVR algorithms like GRPO.

The authors prove that this estimator systematically underestimates advantages for hard prompts and overestimates them for easy prompts, with bias increasing as prompt difficulty deviates from 0.5 and with smaller group sizes.

To address this, they propose History-Aware Adaptive Difficulty Weighting (HA-DW), which uses an evolving difficulty anchor based on cross-batch reward trends to reweight advantage estimates.

Experiments on five mathematical reasoning benchmarks with Qwen3-4B-Base, Qwen3-8B-Base, and LLaMA-3.2-3B-Instruct show that HA-DW consistently improves performance when integrated into GRPO, GSPO, and DAPO, and outperforms GRPO with larger rollouts.

Theoretical analysis confirms that HA-DW reduces estimation bias under certain conditions on the scaling parameter.

The work highlights the importance of correcting biased advantage estimation for robust RLVR training, with limitations noted for group-relative methods only.

The paper includes detailed proofs, ablations, and case studies in the appendix, and is authored by researchers from Beihang University, UC Berkeley, Peking University, and Meituan, with the corresponding author Yikun Ban (yikunb@buaa.edu.cn).

The arXiv ID is 2601.08521v2, dated 22 Jan 2026.

The main contributions are the discovery of the bias, the HA-DW algorithm, and demonstrated performance gains.

The paper also extends the analysis to continuous bounded reward distributions, showing the bias is not an artifact of binary rewards.

The HA-DW method is plug-and-play and improves reasoning performance, with training dynamics showing higher accuracy and longer reasoning chains.

The paper includes a case study where GRPO+HA-DW correctly solves a problem that…

02 From the paper

Abstract

Reinforcement Learning from Verifier Rewards (RLVR) has emerged as a widely used approach for post-training large language models on reasoning tasks, with group-based methods such as GRPO and its variants gaining broad adoption. These methods rely on group-relative advantage estimation to avoid learned critics, yet its theoretical properties remain poorly understood. In this work, we uncover a fundamental issue of group-based RL: the group-relative advantage estimator is inherently biased relative to the true (expected) advantage. We provide the first theoretical analysis showing that it systematically underestimates advantages for hard prompts and overestimates them for easy prompts, leading to imbalanced exploration and exploitation. To address this issue, we propose History-Aware Adaptive Difficulty Weighting (HA-DW), an adaptive reweighting scheme that adjusts advantage estimates based on an evolving difficulty anchor and training dynamics. Both theoretical analysis and experiments on five mathematical reasoning benchmarks demonstrate that HA-DW consistently improves performance when integrated into GRPO and its variants. Our results suggest that correcting biased advantage estimation is critical for robust and efficient RLVR training.