The Year in AI Papers
The year/Independent research

Paper 2509.07980

Parallel-R1: Towards Parallel Thinking via Reinforcement Learning

Tong Zheng, Hongming Zhang, Wenhao Yu, Xiaoyang Wang, Runpeng Dai, Rui Liu, Huiwen Bao, Chengsong Huang, Heng Huang, Dong Yu

Published
Sep 2025
Research lab
Independent
Citations
49
GitHub
Not linked

01 In brief

Summary

Parallel-R1 is the first reinforcement learning (RL) framework to train large language models (LLMs) in parallel thinking for general mathematical reasoning.

It uses a progressive curriculum: supervised fine-tuning (SFT) on easy GSM8K problems to teach the parallel thinking format, followed by RL on harder DAPO problems to generalize the skill.

The framework introduces control tags (<Parallel>, <Path>, <Summary>) for dynamic parallel exploration and summarization.

Experiments on AIME25, AIME24, AMC23, and MATH show Parallel-R1-Seen achieves an average accuracy of 48.9, an 8.4% improvement over the GRPO baseline (45.1).

Analysis reveals the model's strategy evolves from early-stage exploration to late-stage verification.

Parallel thinking also serves as a mid-training exploration scaffold, boosting AIME25 accuracy to 25.6% (a 42.9% improvement over baseline).

Reward design is critical: an alternating accuracy/parallel reward balances performance and parallel usage, while accuracy-only yields low parallel ratio (13.6) and parallel-only hurts performance.

The structured variant (Parallel-R1-Unseen) with path-window masking underperforms the causal variant, and adding RL on GSM8K hurts it.

The model, data, and code are open-sourced at https://github.com/zhengkid/Parallel-R1.

02 From the paper

Abstract

Parallel thinking has emerged as a novel approach for enhancing the reasoning capabilities of large language models (LLMs) by exploring multiple reasoning paths concurrently. However, activating such capabilities through training remains challenging, as existing methods predominantly rely on supervised fine-tuning (SFT) over synthetic data, which encourages teacher-forced imitation rather than exploration and generalization. Different from them, we propose \textbf{Parallel-R1}, the first reinforcement learning (RL) framework that enables parallel thinking behaviors for complex real-world reasoning tasks. Our framework employs a progressive curriculum that explicitly addresses the cold-start problem in training parallel thinking with RL. We first use SFT on prompt-generated trajectories from easier tasks to instill the parallel thinking ability, then transition to RL to explore and generalize this skill on harder problems. Experiments on various math benchmarks, including MATH, AMC23, and AIME, show that Parallel-R1 successfully instills parallel thinking, leading to 8.4% accuracy improvements over the sequential thinking model trained directly on challenging tasks with RL. Further analysis reveals a clear shift in the model's thinking behavior: at an early stage, it uses parallel thinking as an exploration strategy, while in a later stage, it uses the same capability for multi-perspective verification. Most significantly, we validate parallel thinking as a \textbf{mid-training exploration scaffold}, where this temporary exploratory phase unlocks a higher performance ceiling after RL, yielding a 42.9% improvement over the baseline on AIME25. Our model, data, and code will be open-source at https://github.com/zhengkid/Parallel-R1.