Paper 2605.25604
DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learning
- Published
- May 2026
- Research lab
- Independent
- Citations
- 1
- GitHub
- Not linked
01 In brief
Summary
The paper proposes Dynamic Variance-adaptive Advantage Optimization (DVAO) to improve multi-reward reinforcement learning for LLMs, addressing flaws in standard scalarization methods.
Reward Combination (RC) causes training instability due to large advantage magnitudes, while Advantage Combination (AC) uses static weights and ignores cross-objective correlations.
DVAO dynamically adjusts combination weights based on empirical reward variance within a rollout group, up-weighting high-variance objectives and suppressing noisy ones.
The authors prove DVAO bounds advantage magnitudes and introduces a self-adaptive cross-objective regularization mechanism.
Experiments on mathematical reasoning (Qwen3-4B/8B) and tool-use (Qwen2.5-3B/7B) benchmarks show DVAO outperforms baselines (GRPO, RC, AC, GDPO), achieving superior Pareto frontiers and training stability.
DVAO consistently achieves the highest average accuracy and near-perfect length/format compliance, with lower reward variance and faster convergence.
The method is hyperparameter-free and data-driven, providing a robust solution for multi-objective alignment tasks.
Future work includes scaling to more reward functions and extreme memory constraints.
02 From the paper
Abstract
Reinforcement Learning has become a standard paradigm for aligning Large Language Models with human intent and task requirements. While Group Relative Policy Optimization offers an efficient, value-model-free alternative to Proximal Policy Optimization, adapting it to real-world multi-reward settings remains challenging. Standard scalarization practices, such as Reward Combination and Advantage Combination, suffer from significant drawbacks: Reward Combination frequently generates advantages with excessively large squared magnitudes that lead to training instability, while Advantage Combination relies on static hyperparameters and ignores cross-objective correlations. To address these limitations, we propose Dynamic Variance-adaptive Advantage Optimization (DVAO), which dynamically adjusts combination weights based on the empirical reward variance of each objective within a rollout group, effectively up-weighting objectives with a stronger learning signal while suppressing noisy ones. We mathematically prove that DVAO maintains bounded advantage magnitudes for stable training and introduces a self-adaptive cross-objective regularization mechanism. Extensive experiments on mathematical reasoning and tool-use benchmarks using Qwen3 and Qwen2.5 models demonstrate that DVAO significantly outperforms baseline methods, achieving a superior multi-objective Pareto frontier and robust training stability.