Paper 2601.05242
GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
01 In brief
Summary
The paper identifies a flaw in applying Group Relative Policy Optimization (GRPO) to multi-reward reinforcement learning: normalizing the summed reward causes distinct reward combinations to collapse into identical advantage values, reducing training signal resolution and causing suboptimal convergence or early failure.
To address this, the authors propose Group reward-Decoupled Normalization Policy Optimization (GDPO), which normalizes each reward independently before summing and then applies batch-wise advantage normalization for stability.
GDPO preserves cross-reward distinctions, leading to more accurate advantage estimates and improved training stability.
Experiments across tool calling, math reasoning, and coding reasoning show GDPO consistently outperforms GRPO in both correctness metrics (accuracy, bug ratio) and constraint adherence (format, length).
For example, GDPO improves AIME accuracy by up to 6.3% for DeepSeek-R1-1.5B and 2.3% for Qwen3-4B-Instruct while reducing length violations.
The paper also discusses reward weighting and conditioning to handle priority variations, finding that conditioning easier rewards on harder ones is more effective than weight adjustment alone.
Overall, GDPO is presented as a more stable and effective alternative to GRPO for multi-reward RL optimization, with implementations available in HF-TRL, verl, and Nemo-RL.
02 From the paper
Abstract
As language models become increasingly capable, users expect them to provide not only accurate responses but also behaviors aligned with diverse human preferences across a variety of scenarios. To achieve this, Reinforcement learning (RL) pipelines have begun incorporating multiple rewards, each capturing a distinct preference, to guide models toward these desired behaviors. However, recent work has defaulted to apply Group Relative Policy Optimization (GRPO) under multi-reward setting without examining its suitability. In this paper, we demonstrate that directly applying GRPO to normalize distinct rollout reward combinations causes them to collapse into identical advantage values, reducing the resolution of the training signal and resulting in suboptimal convergence and, in some cases, early training failure. We then introduce Group reward-Decoupled Normalization Policy Optimization (GDPO), a new policy optimization method to resolve these issues by decoupling the normalization of individual rewards, more faithfully preserving their relative differences and enabling more accurate multi-reward optimization, along with substantially improved training stability. We compare GDPO with GRPO across three tasks: tool calling, math reasoning, and coding reasoning, evaluating both correctness metrics (accuracy, bug ratio) and constraint adherence metrics (format, length). Across all settings, GDPO consistently outperforms GRPO, demonstrating its effectiveness and generalizability for multi-reward reinforcement learning optimization.