Paper 2509.22611
Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning
- Published
- Sep 2025
- Research lab
- Independent
- Citations
- 10
- GitHub
- 29 stars
01 In brief
Summary
The paper introduces Quantile Advantage Estimation (QAE) to stabilize Reinforcement Learning with Verifiable Rewards (RLVR) for LLM reasoning.
It identifies that mean-baseline methods like GRPO and DAPO cause entropy collapse or explosion, harming performance.
QAE replaces the mean reward baseline with a group-wise K-quantile, creating a two-regime gate: for hard queries (success rate ≤ 1−K), it reinforces rare successes; for easy queries (> 1−K), it targets remaining failures.
This minimal change provably bounds one-step entropy change, preventing both collapse and explosion.
Empirically, QAE stabilizes entropy, sparsifies credit assignment (about 80% of responses get zero advantage), and improves pass@1 on AIME'24, AIME'25, and AMC'23 across Qwen3 models (8B, 14B, 30B-A3B) and methods (DAPO, Clip-Cov, KL-Cov, GSPO), while maintaining pass@16.
The work reframes entropy regulation as a baseline-design problem rather than token-level tuning, offering a drop-in improvement that composes with existing techniques.
02 From the paper
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) strengthens LLM reasoning, but training often oscillates between {entropy collapse} and {entropy explosion}. We trace both hazards to the mean baseline used in value-free RL (e.g., GRPO and DAPO), which improperly penalizes negative-advantage samples under reward outliers. We propose {Quantile Advantage Estimation} (QAE), replacing the mean with a group-wise K-quantile baseline. QAE induces a response-level, two-regime gate: on hard queries (p <= 1 - K) it reinforces rare successes, while on easy queries (p > 1 - K) it targets remaining failures. Under first-order softmax updates, we prove {two-sided entropy safety}, giving lower and upper bounds on one-step entropy change that curb explosion and prevent collapse. Empirically, this minimal modification stabilizes entropy, sparsifies credit assignment (with tuned K, roughly 80% of responses receive zero advantage), and yields sustained pass@1 gains on Qwen3-8B/14B-Base across AIME 2024/2025 and AMC 2023. These results identify {baseline design} -- rather than token-level heuristics -- as the primary mechanism for scaling RLVR.