Paper 2601.20614
Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation
- Published
- Jan 2026
- Research lab
- Independent
- Citations
- 23
- GitHub
- 128 stars
01 In brief
Summary
The paper introduces MathForge, a framework to enhance mathematical reasoning in large language models via reinforcement learning with verifiable rewards (RLVR).
It addresses a lack of emphasis on harder questions in existing methods, both algorithmically and data-wise.
Algorithmically, it proposes Difficulty-Aware Group Policy Optimization (DGPO), which rectifies an implicit imbalance in GRPO's advantage estimation (where update magnitudes peak for moderate-difficulty questions) by using mean absolute deviation instead of standard deviation, and further prioritizes harder questions via difficulty-aware question-level weighting.
Data-wise, it proposes Multi-Aspect Question Reformulation (MQR), which uses a large model to reformulate questions by adding story background, introducing abstract terms, or nesting sub-problems, while preserving the original gold answer.
Experiments on Qwen2.5-Math-7B and other models show MathForge outperforms baselines like GRPO, DAPO, and GSPO on benchmarks such as AIME24, MATH500, and Olympiad, with an average score of 42.17% vs.
37.61% for GRPO.
Ablations confirm the contributions of each component, and the method generalizes across models and to multimodal tasks.
The code and augmented data are released publicly.
The paper concludes that targeting harder questions is key to improving reasoning, encapsulated in the principle 'harder is better'.
02 From the paper
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) offers a robust mechanism for enhancing mathematical reasoning in large models. However, we identify a systematic lack of emphasis on more challenging questions in existing methods from both algorithmic and data perspectives, despite their importance for refining underdeveloped capabilities. Algorithmically, widely used Group Relative Policy Optimization (GRPO) suffers from an implicit imbalance where the magnitude of policy updates is lower for harder questions. Data-wise, augmentation approaches primarily rephrase questions to enhance diversity without systematically increasing intrinsic difficulty. To address these issues, we propose a two-dual MathForge framework to improve mathematical reasoning by targeting harder questions from both perspectives, which comprises a Difficulty-Aware Group Policy Optimization (DGPO) algorithm and a Multi-Aspect Question Reformulation (MQR) strategy. Specifically, DGPO first rectifies the implicit imbalance in GRPO via difficulty-balanced group advantage estimation, and further prioritizes harder questions by difficulty-aware question-level weighting. Meanwhile, MQR reformulates questions across multiple aspects to increase difficulty while maintaining the original gold answer. Overall, MathForge forms a synergistic loop: MQR expands the data frontier, and DGPO effectively learns from the augmented data. Extensive experiments show that MathForge significantly outperforms existing methods on various mathematical reasoning tasks. The code and augmented data are all available at https://github.com/AMAP-ML/MathForge.