Paper 2602.05261
Length-Unbiased Sequence Policy Optimization: Revealing and Controlling Response Length Variation in RLVR
- Published
- Feb 2026
- Research lab
- Independent
- Citations
- 6
- GitHub
- 8 stars
01 In brief
Summary
The paper analyzes response length variation in Reinforcement Learning with Verifiable Rewards (RLVR) for large language models, focusing on the length bias in GRPO and GSPO objectives.
GRPO averages token contributions per trajectory, causing shorter correct responses to receive larger gradient updates, while GSPO's sequence-level clipping and Clip-Higher mechanism amplify this bias, leading to response length collapse.
To address this, the authors propose Length-Unbiased Sequence Policy Optimization (LUSPO), which scales each sequence's loss by its length, eliminating the length bias.
Experiments on dense (Qwen2.5-7B-Base) and MoE (Qwen3-30B-A3B-Instruct) text models, and a vision-language model (Qwen2.5-VL-7B-Instruct), show LUSPO consistently outperforms GRPO and GSPO on benchmarks like AIME24, AIME25, MATH500, MathVista-mini, and LogicVista.
For example, LUSPO improves AIME24 accuracy by up to 2.9% over GSPO for Qwen2.5-7B-Base and 6.9% for Qwen3-30B-A3B-Instruct, and increases average response length by ~1.5x.
LUSPO also prevents response length collapse and improves training stability and accuracy rewards.
02 From the paper
Abstract
Recent applications of Reinforcement Learning with Verifiable Rewards (RLVR) to Large Language Models (LLMs) and Vision-Language Models (VLMs) have demonstrated significant success in enhancing reasoning capabilities for complex tasks. During RLVR training, an increase in response length is often regarded as a key factor contributing to the growth of reasoning ability. However, the patterns of change in response length vary significantly across different RLVR algorithms during the training process. To provide a fundamental explanation for these variations, this paper conducts an in-depth analysis of the components of mainstream RLVR algorithms. We present a theoretical analysis of the factors influencing response length and validate our theory through extensive experimentation. Building upon these theoretical findings, we propose the Length-Unbiased Sequence Policy Optimization (LUSPO) algorithm. Specifically, we rectify the length bias inherent in Group Sequence Policy Optimization (GSPO), rendering its loss function unbiased with respect to response length and thereby resolving the issue of response length collapse. We conduct extensive experiments across mathematical reasoning benchmarks and multimodal reasoning scenarios, where LUSPO consistently achieves superior performance. Empirical results demonstrate that LUSPO represents a novel, state-of-the-art optimization strategy compared to existing methods such as GRPO and GSPO.