Paper 2511.06307
DRIVE: Data Curation Best Practices for Reinforcement Learning with Verifiable Reward in Competitive Code Generation
- Published
- Nov 2025
- Research lab
- Independent
- Citations
- 2
- GitHub
- 8 stars
01 In brief
Summary
This paper from Tencent's Hunyuan Team presents a two-stage reinforcement learning with verifiable reward (RLVR) framework for competitive programming code generation, addressing the underexplored area of data curation and curriculum design.
The method begins with SFT distilled from strong open-source models, then applies two RL stages: first, entropy expansion using a large, uniformly distributed set of ~9k problems with 8 rollouts per prompt and a 24k token window; second, a hard-focus curriculum (Pre-GRPO) on a small set of challenging problems (e.g., 72, 50, 25 hardest cases) with 64 rollouts per prompt.
Implemented on Qwen2.5-32B, the model achieves state-of-the-art performance among similar-scale models, comparable to DeepSeek V3.1 and Doubao-1.5-Thinking, with relative improvements up to 58.3% on Codeforces weekly OJ over the SFT baseline.
Ablations show that difficulty-aware training is crucial, entropy expansion improves generalization, and large rollout budgets are essential for hard problems.
Scaling experiments on an internal large-scale MoE model confirm the strategy's effectiveness, with gains up to +15.17% on LeetCode Weekly OJ.
02 From the paper
Abstract
Recent reasoning-first models (e.g., OpenAI o1, DeepSeek R1) have spurred a resurgence of interest in RLVR. Nevertheless, advances are dominated by mathematics (e.g., AIME), with competitive-programming code generation underexplored and data curation receiving less attention than RL algorithm design. We investigate how to construct RLVR datasets (i.e., RL prompts) and present practical training techniques that yield strong performance on competitive-programming code generation. Our pipeline begins with supervised fine-tuning (SFT) distilled from strong open-source models, augmented with general-purpose and reasoning-intensive data. RL then follows a two-stage process with executable, testcase-driven rewards: first, training on a large, uniformly distributed set of competitive-programming problems using Group Relative Policy Optimization (GRPO) with 8 rollouts per prompt and a relatively short response-generation window (e.g., 32k during SFT and 24k in this stage) to expand entropy and mitigate repetition and truncation; second, we perform \textbf{Pre-GRPO}: updating on a small, high-quality set of challenging problems with a large rollout budget (64 rollouts per prompt) under a hard-focus curriculum that continuously retains the most difficult instances throughout training. We implement our method on Qwen2.5-32B and evaluate on LeetCode and Codeforces weekly contests to avoid data leakage. The resulting model achieves state-of-the-art performance among models of similar scale and is comparable to leading systems such as DeepSeek v3.1 and Doubao-1.5-Thinking. We also examine scaling trends and observe strong RL scaling on an internal large-scale MoE model. Our study distills concise best practices for data curation, entropy expansion, and curriculum design in RLVR for competitive-programming code generation.