Paper 2602.12036
Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models
- Published
- Feb 2026
- Research lab
- Independent
- Citations
- 3
- GitHub
- 137 stars
01 In brief
Summary
Composition-RL addresses the issue of 'solve all' prompts (pass rate 1) that become prevalent during RLVR training, reducing effective data size.
The method automatically composes multiple existing verifiable prompts into new, harder ones using Sequential Prompt Composition (SPC), which modifies one prompt with a variable from another's answer and links them.
Experiments on Qwen3 models (4B-30B) show consistent improvements over RL on original data, with gains scaling with model size (up to +14.3% on math for 30B).
A curriculum variant increasing compositional depth further boosts performance, and cross-domain composition (physics+math) outperforms simple mixing.
Ablations confirm the necessity of composition over direct concatenation and show effectiveness even on fully solvable prompts.
Analysis suggests benefits arise from compositional generalization and implicit process supervision.
02 From the paper
Abstract
Large-scale verifiable prompts underpin the success of Reinforcement Learning with Verifiable Rewards (RLVR), but they contain many uninformative examples and are costly to expand further. Recent studies focus on better exploiting limited training data by prioritizing hard prompts whose rollout pass rate is 0. However, easy prompts with a pass rate of 1 also become increasingly prevalent as training progresses, thereby reducing the effective data size. To mitigate this, we propose Composition-RL, a simple yet useful approach for better utilizing limited verifiable prompts targeting pass-rate-1 prompts. More specifically, Composition-RL automatically composes multiple problems into a new verifiable question and uses these compositional prompts for RL training. Extensive experiments across model sizes from 4B to 30B show that Composition-RL consistently improves reasoning capability over RL trained on the original dataset. Performance can be further boosted with a curriculum variant of Composition-RL that gradually increases compositional depth over training. Additionally, Composition-RL enables more effective cross-domain RL by composing prompts drawn from different domains. Codes, datasets, and models are available at https://github.com/XinXU-USTC/Composition-RL.