Paper 2601.15165
The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models
- Published
- Jan 2026
- Research lab
- Independent
- Citations
- 2
- GitHub
- 259 stars
01 In brief
Summary
This paper challenges the assumption that arbitrary-order generation in Diffusion Large Language Models (dLLMs) enhances reasoning potential.
The authors find that for general reasoning tasks like mathematics and coding, arbitrary-order generation can actually limit reasoning potential compared to standard autoregressive (AR) order.
They introduce the concept of 'entropy degradation,' where dLLMs bypass high-uncertainty 'forking tokens' (e.g., logical connectives) by prioritizing easier tokens, prematurely collapsing solution space coverage.
This is measured via Pass@k, showing AR order achieves higher coverage.
Based on this, they propose JustGRPO, which applies standard Group Relative Policy Optimization (GRPO) by treating the dLLM as an AR policy during training, avoiding complex diffusion-specific RL adaptations.
JustGRPO achieves strong results (e.g., 89.1% on GSM8K) while preserving parallel decoding at inference.
The paper also introduces JustGRPO-Fast, which accelerates training by computing probability ratios only at high-entropy positions, and shows that random order does not help.
Overall, the work suggests that forgoing arbitrary-order flexibility during RL training can be more effective for eliciting reasoning capabilities.
02 From the paper
Abstract
Diffusion Large Language Models (dLLMs) break the rigid left-to-right constraint of traditional LLMs, enabling token generation in arbitrary orders. Intuitively, this flexibility implies a solution space that strictly supersets the fixed autoregressive trajectory, theoretically unlocking superior reasoning potential. However, in this paper, we find that for general reasoning tasks (e.g., mathematics and coding), arbitrary order generation may in fact limit the reasoning potential of dLLMs. We observe that dLLMs tend to exploit this order flexibility to bypass high-uncertainty tokens that are crucial for exploration, which can lead to a premature collapse of solution coverage. This observation motivates a rethink of RL approaches for dLLMs, where considerable complexities, such as handling combinatorial trajectories and intractable likelihoods, are often devoted to preserving this flexibility. We show that effective reasoning can be elicited by simply forgoing arbitrary order and applying standard Group Relative Policy Optimization (GRPO) instead. Our approach, JustGRPO, is minimalist yet surprisingly effective (e.g., 89.1% accuracy on GSM8K) while fully retaining the parallel decoding ability of dLLMs. Project page: https://nzl-thu.github.io/the-flexibility-trap