Paper 2604.00479
All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models
- Published
- Apr 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- 148 stars
01 In brief
Summary
This paper investigates the behavioral differences between Reinforcement Learning (RL) and base Vision-Language Models (VLMs).
The authors find that RL models, such as those trained with Group Relative Policy Optimization (GRPO), exhibit deeper but narrower reasoning, while base models show broader and more diverse thinking patterns.
They identify a 'diversity collapse' in GRPO, where models prematurely converge to a limited set of reasoning strategies, leading to local optima and poor scalability.
To address this, they propose Multi-Group Policy Optimization (MUPO), which partitions responses into multiple groups, computes localized advantages, and introduces a diversity reward to encourage exploration of diverse strategies.
MUPO-Thinker-7B achieves average gains of 2.5% on mathematical benchmarks and 2.3% on general-purpose benchmarks over previous best results, and demonstrates stronger test-time scaling capabilities.
The work highlights the importance of preserving divergent thinking in RL training for VLMs.
02 From the paper
Abstract
Recent studies have demonstrated that Reinforcement Learning (RL), notably Group Relative Policy Optimization (GRPO), can intrinsically elicit and enhance the reasoning capabilities of Vision-Language Models (VLMs). However, despite the promise, the underlying mechanisms that drive the effectiveness of RL models as well as their limitations remain underexplored. In this paper, we highlight a fundamental behavioral distinction between RL and base models, where the former engages in deeper yet narrow reasoning, while base models, despite less refined along individual path, exhibit broader and more diverse thinking patterns. Through further analysis of training dynamics, we show that GRPO is prone to diversity collapse, causing models to prematurely converge to a limited subset of reasoning strategies while discarding the majority of potential alternatives, leading to local optima and poor scalability. To address this, we propose Multi-Group Policy Optimization (MUPO), a simple yet effective approach designed to incentivize divergent thinking across multiple solutions, and demonstrate its effectiveness on established benchmarks. Project page: https://xytian1008.github.io/MUPO/