VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models

The paper introduces VCRL, a curriculum reinforcement learning framework for large language models (LLMs) that dynamically adjusts training sample difficulty based on the variance of group rewards in rollout-based RL (e.g., GRPO).

The authors observe that samples with moderate difficulty yield higher reward variance, while too-easy or too-hard samples yield low variance.

VCRL uses Variance-based Dynamic Sampling to filter samples with high normalized variance (p) and Replay Learning with a memory bank to maintain a high-value training batch.

Experiments on five mathematical benchmarks (AIME-2024, AIME-2025, MATH500, OlympiadBench, AMC23) with Qwen3-4B-Base and Qwen3-8B-Base show VCRL outperforms GRPO, DAPO, and GSPO, achieving average scores of 49.43 and 57.76 respectively, with notable gains on competition-level datasets.

Ablation studies confirm each component contributes positively.

The paper also provides a theoretical proof that VCRL's policy gradient norm is expected to be lower than GRPO's, indicating more stable training.

Training dynamics show VCRL maintains higher reward scores, longer responses, and more stable entropy compared to GRPO.

The method is implemented on the verl framework with hyperparameters including a variance threshold κ of 0.3 initially and 0.8 later, and a momentum constant α of 0.9.

The work addresses the limitation of existing RL methods that ignore sample difficulty relative to the model's current ability, aligning with human curriculum learning principles.

The authors conclude that VCRL improves training efficiency and stability, leading to state-of-the-art results on mathematical reasoning tasks.

Future work may explore broader applications beyond mathematics.

The paper is a preprint from Alibaba Cloud Computing, with corresponding…