Paper 2508.14029
Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 53
- GitHub
- 55 stars
01 In brief
Summary
The paper proposes an online Self-play with Variational problem Synthesis (SvS) strategy for Reinforcement Learning with Verifiable Rewards (RLVR) to address entropy collapse and plateaued Pass@k performance in LLM reasoning.
Standard RLVR improves Pass@1 but reduces generation diversity, limiting Pass@k.
SvS uses the policy's correct solutions to underperforming training problems to synthesize variational problems that share the same reference answers, avoiding additional labeling.
The policy then solves these synthetic problems, and reward shaping ensures they remain appropriately challenging.
This self-improving loop maintains policy entropy and enables sustained exploration.
Experiments on models from 3B to 32B across 12 reasoning benchmarks show SvS consistently outperforms standard RLVR, with absolute gains of 18.3% and 22.8% in Pass@32 on AIME 24 and AIME 25, respectively.
SvS also generalizes to code generation tasks, achieving improvements with fewer training steps.
Ablations confirm that augmenting underperforming problems and maintaining diversity are crucial, and combining SvS with entropy regulation methods like Clip-Cov further improves performance.
The framework is agnostic to RLVR optimization algorithms and relies solely on the policy itself for self-improvement.
02 From the paper
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as a key paradigm for post-training Large Language Models (LLMs), particularly for complex reasoning tasks. However, vanilla RLVR training has been shown to improve Pass@1 performance at the expense of policy entropy, leading to reduced generation diversity and limiting the Pass@k performance, which typically represents the upper bound of LLM reasoning capability. In this paper, we systematically analyze the policy's generation diversity from the perspective of training problems and find that augmenting and updating training problems helps mitigate entropy collapse during training. Based on these observations, we propose an online Self-play with Variational problem Synthesis (SvS) strategy for RLVR training, which uses the policy's correct solutions to synthesize variational problems while ensuring their reference answers remain identical to the originals. This self-improving strategy effectively maintains policy entropy during training and substantially improves Pass@k compared with standard RLVR, sustaining prolonged improvements and achieving absolute gains of 18.3% and 22.8% in Pass@32 performance on the competition-level AIME24 and AIME25 benchmarks, as well as on code generation tasks. Experiments on 12 reasoning benchmarks across varying model sizes from 3B to 32B consistently demonstrate the generalizability and robustness of SvS.