Paper 2509.21880
No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping
- Published
- Sep 2025
- Research lab
- Independent
- Citations
- 41
- GitHub
- Not linked
01 In brief
Summary
The paper introduces RL-ZVP, a reinforcement learning algorithm for LLMs that exploits zero-variance prompts—where all sampled responses to a prompt receive identical rewards—which standard methods like GRPO discard due to zero advantage.
RL-ZVP assigns advantages to these prompts based on response correctness (positive for all-correct, negative for all-incorrect) and scales the magnitude by token entropy, using a hyperparameter α.
This approach generalizes GRPO, reverting to standard GRPO for non-zero-variance prompts.
Experiments on Qwen3-1.7B-Base and Qwen3-8B-Base across six math benchmarks show RL-ZVP outperforms GRPO by up to 8.61 points in accuracy and 7.77 points in pass rate, and also surpasses baselines that filter zero-variance prompts, even when those baselines use 3–5× more rollouts.
Ablations confirm the importance of both positive and negative prompt signals and entropy scaling, with α=0.1 optimal.
The work challenges the prevailing practice of discarding zero-variance prompts, demonstrating their untapped potential for improving RLVR training stability and reasoning performance.
02 From the paper
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful framework for improving the reasoning abilities of Large Language Models (LLMs). However, current methods such as GRPO rely only on problems where the model responses to the same input differ in correctness, while ignoring those where all responses receive the same reward -- so-called zero-variance prompts. In this work, we argue that such prompts are not useless but can, in fact, provide meaningful feedback for policy optimization. To this end, we introduce Reinforcement Learning with Zero-Variance Prompts (RL-ZVP), a novel algorithm that extract learning signals from zero-variance prompts. RL-ZVP directly rewards correctness and penalizes errors even without contrasting responses, modulating feedback with token-level characteristics to preserve informative, nuanced signals. Across six math reasoning benchmarks, RL-ZVP achieves significant improvements of up to 8.61 points in accuracy and 7.77 points in pass rate over GRPO, while consistently outperforming other baselines that filter out zero-variance prompts. These results highlight the untapped potential of learning from zero-variance prompts in RLVR. The project page is available at https://bltnynk.github.io/publications/rl-zvp/.