Paper 2510.03222
Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 23
- GitHub
- 44 stars
01 In brief
Summary
This paper addresses the exploration collapse in Reinforcement Learning with Verifiable Rewards (RLVR) for large language models, where performance plateaus as policy entropy collapses.
The authors identify the systematic elimination of valuable low-probability exploratory tokens, termed 'reasoning sparks' (e.g., 'wait', 'however'), as a key cause, while indiscriminate entropy control amplifies irrelevant noise tokens (e.g., 'cost', 'fine'), destabilizing training.
They propose Low-probability Regularization (Lp-Reg), which constructs a less-noisy proxy distribution by filtering out presumed noise tokens below a threshold (fixed or min-p) and renormalizing probabilities over remaining candidates.
This proxy serves as a soft regularization target via forward KL divergence, applied only to low-probability, non-noise tokens with negative advantages.
Experiments on Qwen3-14B-Base and Qwen2.5-32B-Base show Lp-Reg enables stable on-policy training for 3,000 steps (81,204 GPU-hours) where baselines collapse, achieving 60.17% average accuracy on five math benchmarks, a 2.66% improvement over prior methods.
Ablations confirm the importance of targeting low-probability tokens over high-entropy ones, noise filtering, negative-sample regularization, and forward KL over reverse KL.
02 From the paper
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has propelled Large Language Models in complex reasoning, yet its scalability is often hindered by a training bottleneck where performance plateaus as policy entropy collapses, signaling a loss of exploration. Previous methods typically address this by maintaining high policy entropy, yet the precise mechanisms that govern meaningful exploration have remained underexplored. Our analysis suggests that an unselective focus on entropy risks amplifying irrelevant tokens and destabilizing training. This paper investigates the exploration dynamics within RLVR and identifies a key issue: the gradual elimination of valuable low-probability exploratory tokens, which we term \textbf{\textit{reasoning sparks}}. We find that while abundant in pre-trained models, these sparks are systematically extinguished during RLVR due to over-penalization, leading to a degeneracy in exploration. To address this, we introduce Low-probability Regularization (Lp-Reg). Its core mechanism regularizes the policy towards a heuristic proxy distribution. This proxy is constructed by filtering out presumed noise tokens and re-normalizing the distribution over the remaining candidates. The result is a less-noisy proxy where the probability of \textit{reasoning sparks} is amplified, which then serves as a soft regularization target to shield these valuable tokens from elimination via KL divergence. Experiments show that Lp-Reg enables stable on-policy RL, sustaining continuous scaling across $3,000$ training steps and $81,204$ GPU-hours, where baseline entropy-control methods collapse. This sustained exploration leads to state-of-the-art performance, achieving a $60.17\%$ average accuracy on five math benchmarks, an improvement of $2.66\%$ over prior methods. Code is available at https://github.com/CarlanLark/Lp-Reg.