The Year in AI Papers
The year/Topics/RL for reasoning

Research collection

RL for reasoning

Reinforcement learning applied to train reasoning ability, including RLVR, verifiable rewards, reward design, and exploration for reasoning models.

Papers
136
Research labs
7
Official code
85

101–136 of 136 papers in this collection

101

arXiv.org

Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play

Vision-Zero is a label-free, domain-agnostic multi-agent self-play framework for self-evolving vision-language models (VLMs) through competitive visual games generated from arbitrary images. It trains VLMs in a 'Who Is the Spy?'-style game where civilians see an image and the spy sees a blank input, requiring strategic reasoning and communication. The…

Qinsi Wang, Bo Liu, Tianyi Zhou, Jing Shi, et al.
Published
Sep 2025
Citations
33
Code
165 stars
Read summary Original
102

arXiv.org

DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search

DeepSearch is a framework that integrates Monte Carlo Tree Search (MCTS) directly into reinforcement learning with verifiable rewards (RLVR) training to overcome exploration bottlenecks that cause performance plateaus. Unlike methods that use tree search only at inference, DeepSearch embeds structured search into the training loop, enabling systematic…

Fang Wu, Weihao Xuan, Heli Qi, Ximing Lu, et al.
Published
Sep 2025
Citations
10
Code
34 stars
Read summary Original
103

arXiv.org

Multiplayer Nash Preference Optimization

This paper introduces Multiplayer Nash Preference Optimization (MNPO), a framework that generalizes Nash learning from human feedback (NLHF) to n-player games for aligning large language models with human preferences. MNPO formulates alignment as a game where each policy competes against a population of opponents while being regularized toward a reference…

Fang Wu, Xu Huang, Weihao Xuan, Zhiwei Zhang, et al.
Published
Sep 2025
Citations
6
Code
35 stars
Read summary Original
104

arXiv.org

Language Models Can Learn from Verbal Feedback Without Scalar Rewards

The paper proposes Feedback-Conditional Policy (FCP), a method for training LLMs directly from verbal feedback without converting it into scalar rewards. FCP treats feedback as a conditioning signal, learning a policy πθ(o|x,c) that approximates the posterior P_off(o|x,c) ∝ π_ref(o|x)·p_env(c|x,o) via maximum likelihood on offline data. An online…

Renjie Luo, Zichen Liu, Xiangyan Liu, Chao Du, et al.
Published
Sep 2025
Citations
15
Code
65 stars
Read summary Original
105

arXiv.org

Variational Reasoning for Language Models

The paper introduces a variational reasoning framework for language models, treating thinking traces as latent variables and optimizing them via variational inference. It extends the evidence lower bound (ELBO) to a multi-trace IWAE-style objective and proposes a forward-KL formulation for stable training of the variational posterior. The authors show that…

Xiangxin Zhou, Zichen Liu, Haonan Wang, Chao Du, et al.
Published
Sep 2025
Citations
6
Code
60 stars
Read summary Original
106

Independent research

Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning

The paper introduces Quantile Advantage Estimation (QAE) to stabilize Reinforcement Learning with Verifiable Rewards (RLVR) for LLM reasoning. It identifies that mean-baseline methods like GRPO and DAPO cause entropy collapse or explosion, harming performance. QAE replaces the mean reward baseline with a group-wise K-quantile, creating a two-regime gate:…

Junkang Wu, Kexin Huang, Jiancan Wu, An Zhang, et al.
Published
Sep 2025
Citations
10
Code
29 stars
Read summary Original
107

arXiv.org

EPO: Entropy-regularized Policy Optimization for LLM Agents Reinforcement Learning

The paper identifies the exploration-exploitation cascade failure in multi-turn LLM agent reinforcement learning, where shared policy parameters across turns and sparse terminal rewards cause severe entropy oscillations that destabilize training. To address this, the authors propose Entropy-regularized Policy Optimization (EPO), which combines…

Wujiang Xu, Wentian Zhao, Zhenting Wang, Yu-Jhe Li, et al.
Published
Sep 2025
Citations
10
Code
40 stars
Read summary Original
108

Independent research

Verifying Meta-Awareness via Predictive Rewards in Reasoning Models

The paper introduces MAPR (Meta-Awareness via Predictive Reward), a reinforcement learning framework that enhances reasoning models by training them to predict their own rollout statistics—specifically solution length, pass-rate, and concepts used—and rewarding alignment with actual statistics. This self-generated meta-prediction is verified against…

Yoonjeon Kim, Doohyuk Jang, Eunho Yang
Published
Sep 2025
Citations
2
Code
12 stars
Read summary Original
109

arXiv.org

No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping

The paper introduces RL-ZVP, a reinforcement learning algorithm for LLMs that exploits zero-variance prompts—where all sampled responses to a prompt receive identical rewards—which standard methods like GRPO discard due to zero advantage. RL-ZVP assigns advantages to these prompts based on response correctness (positive for all-correct, negative for…

Thanh-Long V. Le, Myeongho Jeon, Kim Vu, Viet Lai, et al.
Published
Sep 2025
Citations
41
Code
Not linked
Read summary Original
110

arXiv.org

MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources

This paper introduces MMR1, a framework for enhancing multimodal reasoning in large models. It addresses two key limitations: the lack of open, large-scale, high-quality long chain-of-thought (CoT) data, and the instability of reinforcement learning (RL) algorithms, specifically gradient vanishing in Group Relative Policy Optimization (GRPO) when reward…

Sicong Leng, Jing Wang, Jiaxi Li, Hao Zhang, et al.
Published
Sep 2025
Citations
35
Code
217 stars
Read summary Original
111

arXiv.org

Tree Search for LLM Agent Reinforcement Learning

The paper introduces Tree-GRPO, a reinforcement learning method for LLM agents that uses tree search for rollouts, with each node representing a complete agent interaction step (Thought-Action-Observation). This approach shares common prefixes across rollouts, increasing the number of samples under a fixed token/tool-call budget. Tree-GRPO estimates group…

Yuxiang Ji, Ziyu Ma, Yong Wang, Guanhua Chen, et al.
Published
Sep 2025
Citations
52
Code
392 stars
Read summary Original
112

arXiv.org

VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models

The paper introduces VCRL, a curriculum reinforcement learning framework for large language models (LLMs) that dynamically adjusts training sample difficulty based on the variance of group rewards in rollout-based RL (e.g., GRPO). The authors observe that samples with moderate difficulty yield higher reward variance, while too-easy or too-hard samples…

Guochao Jiang, Wenfeng Feng, Guofeng Quan, Chuzhan Hao, et al.
Published
Sep 2025
Citations
27
Code
Not linked
Read summary Original
113

Annual Meeting of the Association for Computational Linguistics

Reinforcement Learning on Pre-Training Data

The paper introduces RLPT (Reinforcement Learning on Pre-Training data), a new training-time scaling paradigm for LLMs that applies reinforcement learning directly to pre-training data, eliminating the need for human annotations. RLPT uses a next-segment reasoning objective where the model predicts a subsequent text segment, and a generative reward model…

Siheng Li, Kejiao Li, Zenan Xu, Guanhua Huang, et al.
Published
Sep 2025
Citations
11
Code
Not linked
Read summary Original
114

arXiv.org

FlowRL: Matching Reward Distributions for LLM Reasoning

FlowRL is a policy optimization algorithm for large language model (LLM) reasoning that shifts from reward maximization to reward distribution matching. It uses a learnable partition function to normalize scalar rewards into a target distribution and minimizes the reverse KL divergence between the policy and this distribution, which is equivalent to a…

Xuekai Zhu, Daixuan Cheng, Dinghuai Zhang, Hengli Li, et al.
Published
Sep 2025
Citations
37
Code
Not linked
Read summary Original
115

arXiv.org

SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning

SimpleVLA-RL is an efficient reinforcement learning (RL) framework for Vision-Language-Action (VLA) models, built on veRL, that addresses data scarcity and poor generalization in robotic manipulation. It uses interactive trajectory sampling, parallel multi-environment rendering, and outcome-based rewards (1 for success, 0 for failure) with GRPO, enhanced…

Haozhan Li, Yuxin Zuo, Jiale Yu, Yuhao Zhang, et al.
Published
Sep 2025
Citations
130
Code
1.8K stars
Read summary Original
116

arXiv.org

A Survey of Reinforcement Learning for Large Reasoning Models

This survey reviews recent advances in Reinforcement Learning (RL) for Large Reasoning Models (LRMs), focusing on how RL transforms LLMs into LRMs, particularly since DeepSeek-R1. It covers foundational components: reward design (verifiable, generative, dense, unsupervised, and shaping), policy optimization (critic-based, critic-free, off-policy, and…

Kaiyan Zhang, Yuxin Zuo, Bingxiang He, Youbang Sun, et al.
Published
Sep 2025
Citations
161
Code
2.5K stars
Read summary Original
117

arXiv.org

RewardDance: Reward Scaling in Visual Generation

RewardDance is a scalable reward modeling framework for visual generation that addresses limitations in existing reward models (RMs) by using a generative paradigm. It reformulates reward scoring as the probability of a 'yes' token in a Vision-Language Model (VLM), aligning with the model's next-token prediction mechanism. This enables scaling along two…

Jie Wu, Yu Gao, Zilyu Ye, Ming Li, et al.
Published
Sep 2025
Citations
58
Code
Not linked
Read summary Original
118

arXiv.org

Sharing is Caring: Efficient LM Post-Training with Collective RL Experience Sharing

The paper introduces Swarm sAmpling Policy Optimization (SAPO), a fully decentralized and asynchronous reinforcement learning (RL) post-training algorithm for language models (LMs). SAPO enables heterogeneous compute nodes to train their own policies while sharing decoded rollouts with the swarm, avoiding synchronization bottlenecks and hardware…

Jeffrey Amico, Gabriel Passamani Andrade, John Donaghy, Ben Fielding, et al.
Published
Sep 2025
Citations
3
Code
Not linked
Read summary Original
119

arXiv.org

Parallel-R1: Towards Parallel Thinking via Reinforcement Learning

Parallel-R1 is the first reinforcement learning (RL) framework to train large language models (LLMs) in parallel thinking for general mathematical reasoning. It uses a progressive curriculum: supervised fine-tuning (SFT) on easy GSM8K problems to teach the parallel thinking format, followed by RL on harder DAPO problems to generalize the skill. The…

Tong Zheng, Hongming Zhang, Wenhao Yu, Xiaoyang Wang, et al.
Published
Sep 2025
Citations
49
Code
Not linked
Read summary Original
120

arXiv.org

Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models

The paper introduces TraceRL, a trajectory-aware reinforcement learning framework for diffusion language models (DLMs) that aligns training objectives with inference trajectories, improving reasoning performance on math and coding tasks. It includes a diffusion-based value model for training stability and supports both full-attention and block-attention…

Yinjie Wang, Ling Yang, Bowen Li, Ye Tian, et al.
Published
Sep 2025
Citations
75
Code
516 stars
Read summary Original
121

arXiv.org

Towards a Unified View of Large Language Model Post-Training

This paper introduces a unified theoretical framework for large language model (LLM) post-training, showing that Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) are not contradictory but instances of a single optimization process. The authors derive a Unified Policy Gradient Estimator (UPGE) that subsumes the gradients of various post-training…

Xingtai Lv, Yuxin Zuo, Youbang Sun, Hongyi Liu, et al.
Published
Sep 2025
Citations
39
Code
211 stars
Read summary Original
122

arXiv.org

The Landscape of Agentic Reinforcement Learning for LLMs: A Survey

This survey formalizes Agentic Reinforcement Learning (Agentic RL), a paradigm where LLMs are treated as learnable policies in sequential decision-making loops, contrasting it with conventional single-step LLM RL. It proposes a twofold taxonomy: one based on core agentic capabilities (planning, tool use, memory, reasoning, self-improvement, perception) and…

Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, et al.
Published
Sep 2025
Citations
182
Code
1.9K stars
Read summary Original
123

arXiv.org

SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning

SimpleTIR is a plug-and-play algorithm that stabilizes multi-turn Tool-Integrated Reasoning (TIR) training under the Zero RL paradigm. The authors identify that training instability and gradient explosions stem from distributional drift caused by external tool feedback, which leads to the generation of low-probability tokens that compound over turns. They…

Zhenghai Xue, Longtao Zheng, Qian Liu, Yingru Li, et al.
Published
Sep 2025
Citations
139
Code
403 stars
Read summary Original
124

arXiv.org

Reasoning Vectors: Transferring Chain-of-Thought Capabilities via Task Arithmetic

This paper introduces the concept of a reasoning vector, which captures the reasoning capability instilled by reinforcement learning (RL) in large language models (LLMs) and can be transferred to other models via simple tensor arithmetic. The authors source two publicly available QWEN2.5 models (1.5B and 7B) with identical initialization, one fine-tuned…

Mohammad Zbeeb, Hasan Abed Al Kader Hammoud, Bernard Ghanem
Published
Sep 2025
Citations
5
Code
Not linked
Read summary Original
125

arXiv.org

LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model

The paper introduces LLaVA-Critic-R1, a multimodal critic model trained via reinforcement learning (RL) on preference-labeled critic data, which surprisingly also becomes a strong policy model. By reformulating critic data into verifiable tasks and using GRPO with preference and format rewards, the model improves over its base (Qwen-2.5-VL-7B) by +5.7% on…

Xiyao Wang, Chunyuan Li, Jianwei Yang, Kai Zhang, et al.
Published
Aug 2025
Citations
30
Code
4.7K stars
Read summary Original
126

arXiv.org

R-4B: Incentivizing General-Purpose Auto-Thinking Capability in MLLMs via Bi-Mode Annealing and Reinforce Learning

The paper introduces R-4B, a multimodal large language model (MLLM) with auto-thinking capability, which adaptively decides whether to engage in step-by-step reasoning based on problem complexity. R-4B is trained in two stages: bi-mode annealing, which uses a curated dataset of reasoning and non-reasoning examples to equip the model with both thinking and…

Qi Yang, Bolin Ni, Shiming Xiang, Han Hu, et al.
Published
Aug 2025
Citations
16
Code
141 stars
Read summary Original
127

arXiv.org

Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning

The paper introduces Pref-GRPO, a reinforcement learning method for text-to-image (T2I) generation that replaces pointwise reward score maximization with pairwise preference fitting to address reward hacking. It identifies the root cause of reward hacking as an 'illusory advantage' arising from normalizing tightly clustered pointwise scores by a small…

Yibin Wang, Zhimin Li, Yuhang Zang, Yujie Zhou, et al.
Published
Aug 2025
Citations
90
Code
276 stars
Read summary Original
128

arXiv.org

rStar2-Agent: Agentic Reasoning Technical Report

rStar2-Agent is a 14B math reasoning model trained with agentic reinforcement learning (RL) to achieve frontier-level performance, rivaling or surpassing DeepSeek-R1 (671B). It uses Python coding tools within a dedicated environment, enabling advanced cognitive behaviors like careful tool use and reflection on execution feedback. Key innovations include an…

Ning Shang, Yifei Liu, Yi Zhu, Li Lyna Zhang, et al.
Published
Aug 2025
Citations
45
Code
1.4K stars
Read summary Original
129

arXiv.org

Self-Rewarding Vision-Language Model via Reasoning Decomposition

Vision-SR1 is a three-stage self-rewarding reinforcement learning method for vision-language models (VLMs) that improves visual reasoning without external visual supervision. It decomposes VLM reasoning into visual perception and language reasoning, prompting the model to first generate a self-contained visual description, then re-prompting it to answer…

Zongxia Li, Wenhao Yu, Chengsong Huang, Zhenwen Liang, et al.
Published
Aug 2025
Citations
91
Code
180 stars
Read summary Original
130

arXiv.org

TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling

TreePO is a reinforcement learning framework for large language models that replaces standard independent trajectory sampling with a heuristic tree-based rollout mechanism. It models sequence generation as a tree search with dynamic branching and fixed-length segment decoding, leveraging shared prefixes via KV-cache reuse to reduce compute. The method…

Yizhi Li, Qingshui Gu, Zhoufutu Wen, Ziniu Li, et al.
Published
Aug 2025
Citations
59
Code
65 stars
Read summary Original
131

arXiv.org

DuPO: Enabling Reliable LLM Self-Verification via Dual Preference Optimization

DuPO is a dual learning-based preference optimization framework that generates annotation-free feedback for LLM optimization. It addresses limitations of RLVR (costly labels, restricted to verifiable tasks) and traditional dual learning (strictly dual task pairs). DuPO decomposes a primal task's input into known and unknown components, then constructs a…

Shuaijie She, Yu Bao, Yu Lu, Lu Xu, et al.
Published
Aug 2025
Citations
5
Code
Not linked
Read summary Original
132

arXiv.org

Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR

The paper proposes an online Self-play with Variational problem Synthesis (SvS) strategy for Reinforcement Learning with Verifiable Rewards (RLVR) to address entropy collapse and plateaued Pass@k performance in LLM reasoning. Standard RLVR improves Pass@1 but reduces generation diversity, limiting Pass@k. SvS uses the policy's correct solutions to…

Xiao Liang, Zhongzhi Li, Yeyun Gong, Yelong Shen, et al.
Published
Aug 2025
Citations
53
Code
55 stars
Read summary Original
133

arXiv.org

SSRL: Self-Search Reinforcement Learning

The paper introduces Self-Search RL (SSRL), a method that trains LLMs to answer search-based questions using only their internal knowledge, without external search engines. The authors first quantify LLMs' intrinsic search capability via repeated sampling (Self-Search), showing strong scaling with inference budget and high pass@k on benchmarks like…

Yuchen Fan, Kaiyan Zhang, Heng Zhou, Yuxin Zuo, et al.
Published
Aug 2025
Citations
11
Code
210 stars
Read summary Original
134

arXiv.org

We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoning

WE-MATH 2.0 is a unified system to enhance multimodal large language models' (MLLMs) mathematical reasoning. It integrates a structured knowledge system, model-centric data modeling, and reinforcement learning (RL). The MathBook Knowledge System is a five-level hierarchy with 491 knowledge points and 1,819 fundamental principles. MathBook-Standard is a…

Runqi Qiao, Qiuna Tan, Peiqing Yang, Yanzi Wang, et al.
Published
Aug 2025
Citations
30
Code
170 stars
Read summary Original
135

arXiv.org

On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification

This paper analyzes the generalization gap between Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) for Large Language Models. The authors show that the SFT gradient can be interpreted as a policy gradient with an implicit, sparse reward that is inversely proportional to the model's token probability, leading to unstable updates and poor…

Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, et al.
Published
Aug 2025
Citations
112
Code
1.1K stars
Read summary Original
136

arXiv.org

R-Zero: Self-Evolving Reasoning LLM from Zero Data

R-Zero is a fully autonomous framework for training reasoning LLMs without any pre-existing tasks or labels. It initializes two models from a single base LLM: a Challenger, trained via GRPO to generate questions at the edge of the Solver's capability (using an uncertainty reward based on self-consistency, plus repetition and format penalties), and a…

Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, et al.
Published
Aug 2025
Citations
167
Code
831 stars
Read summary Original