The Year in AI Papers
The year/Topics/Reasoning

Topic area

Reasoning

Every collection across reasoning.

Papers
177
Research labs
7
Official code
113

151–177 of 177 papers in this topic area

151

arXiv.org

FlowRL: Matching Reward Distributions for LLM Reasoning

FlowRL is a policy optimization algorithm for large language model (LLM) reasoning that shifts from reward maximization to reward distribution matching. It uses a learnable partition function to normalize scalar rewards into a target distribution and minimizes the reverse KL divergence between the policy and this distribution, which is equivalent to a…

Xuekai Zhu, Daixuan Cheng, Dinghuai Zhang, Hengli Li, et al.
Published
Sep 2025
Citations
37
Code
Not linked
Read summary Original
152

arXiv.org

SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning

SimpleVLA-RL is an efficient reinforcement learning (RL) framework for Vision-Language-Action (VLA) models, built on veRL, that addresses data scarcity and poor generalization in robotic manipulation. It uses interactive trajectory sampling, parallel multi-environment rendering, and outcome-based rewards (1 for success, 0 for failure) with GRPO, enhanced…

Haozhan Li, Yuxin Zuo, Jiale Yu, Yuhao Zhang, et al.
Published
Sep 2025
Citations
130
Code
1.8K stars
Read summary Original
153

arXiv.org

A Survey of Reinforcement Learning for Large Reasoning Models

This survey reviews recent advances in Reinforcement Learning (RL) for Large Reasoning Models (LRMs), focusing on how RL transforms LLMs into LRMs, particularly since DeepSeek-R1. It covers foundational components: reward design (verifiable, generative, dense, unsupervised, and shaping), policy optimization (critic-based, critic-free, off-policy, and…

Kaiyan Zhang, Yuxin Zuo, Bingxiang He, Youbang Sun, et al.
Published
Sep 2025
Citations
161
Code
2.5K stars
Read summary Original
154

arXiv.org

RewardDance: Reward Scaling in Visual Generation

RewardDance is a scalable reward modeling framework for visual generation that addresses limitations in existing reward models (RMs) by using a generative paradigm. It reformulates reward scoring as the probability of a 'yes' token in a Vision-Language Model (VLM), aligning with the model's next-token prediction mechanism. This enables scaling along two…

Jie Wu, Yu Gao, Zilyu Ye, Ming Li, et al.
Published
Sep 2025
Citations
58
Code
Not linked
Read summary Original
155

arXiv.org

Sharing is Caring: Efficient LM Post-Training with Collective RL Experience Sharing

The paper introduces Swarm sAmpling Policy Optimization (SAPO), a fully decentralized and asynchronous reinforcement learning (RL) post-training algorithm for language models (LMs). SAPO enables heterogeneous compute nodes to train their own policies while sharing decoded rollouts with the swarm, avoiding synchronization bottlenecks and hardware…

Jeffrey Amico, Gabriel Passamani Andrade, John Donaghy, Ben Fielding, et al.
Published
Sep 2025
Citations
3
Code
Not linked
Read summary Original
156

arXiv.org

Parallel-R1: Towards Parallel Thinking via Reinforcement Learning

Parallel-R1 is the first reinforcement learning (RL) framework to train large language models (LLMs) in parallel thinking for general mathematical reasoning. It uses a progressive curriculum: supervised fine-tuning (SFT) on easy GSM8K problems to teach the parallel thinking format, followed by RL on harder DAPO problems to generalize the skill. The…

Tong Zheng, Hongming Zhang, Wenhao Yu, Xiaoyang Wang, et al.
Published
Sep 2025
Citations
49
Code
Not linked
Read summary Original
157

arXiv.org

Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models

The paper introduces TraceRL, a trajectory-aware reinforcement learning framework for diffusion language models (DLMs) that aligns training objectives with inference trajectories, improving reasoning performance on math and coding tasks. It includes a diffusion-based value model for training stability and supports both full-attention and block-attention…

Yinjie Wang, Ling Yang, Bowen Li, Ye Tian, et al.
Published
Sep 2025
Citations
75
Code
516 stars
Read summary Original
158

arXiv.org

Reverse-Engineered Reasoning for Open-Ended Generation

The paper introduces REverse-Engineered Reasoning (REER), a new paradigm for instilling deep reasoning in LLMs for open-ended, non-verifiable tasks like creative writing, where reinforcement learning (RL) and instruction distillation fail due to lack of clear rewards and high costs. REER works backwards from known good solutions, using a gradient-free…

Haozhe Wang, Haoran Que, Qixin Xu, Minghao Liu, et al.
Published
Sep 2025
Citations
33
Code
98 stars
Read summary Original
159

arXiv.org

Towards a Unified View of Large Language Model Post-Training

This paper introduces a unified theoretical framework for large language model (LLM) post-training, showing that Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) are not contradictory but instances of a single optimization process. The authors derive a Unified Policy Gradient Estimator (UPGE) that subsumes the gradients of various post-training…

Xingtai Lv, Yuxin Zuo, Youbang Sun, Hongyi Liu, et al.
Published
Sep 2025
Citations
39
Code
211 stars
Read summary Original
160

Conference on Empirical Methods in Natural Language Processing

Drivel-ology: Challenging LLMs with Interpreting Nonsense with Depth

The paper introduces Drivelology, a linguistic phenomenon of syntactically coherent but pragmatically paradoxical utterances that encode implicit meaning requiring contextual inference, moral reasoning, or emotional interpretation. The authors construct DRIVELHUB, a benchmark dataset of over 1,200 curated examples across six languages (English, Mandarin,…

Yang Wang, Chenghao Xiao, Chia-Yi Hsiao, Zi Yan Chang, et al.
Published
Sep 2025
Citations
3
Code
15 stars
Read summary Original
161

arXiv.org

The Landscape of Agentic Reinforcement Learning for LLMs: A Survey

This survey formalizes Agentic Reinforcement Learning (Agentic RL), a paradigm where LLMs are treated as learnable policies in sequential decision-making loops, contrasting it with conventional single-step LLM RL. It proposes a twofold taxonomy: one based on core agentic capabilities (planning, tool use, memory, reasoning, self-improvement, perception) and…

Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, et al.
Published
Sep 2025
Citations
182
Code
1.9K stars
Read summary Original
162

arXiv.org

SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning

SimpleTIR is a plug-and-play algorithm that stabilizes multi-turn Tool-Integrated Reasoning (TIR) training under the Zero RL paradigm. The authors identify that training instability and gradient explosions stem from distributional drift caused by external tool feedback, which leads to the generation of low-probability tokens that compound over turns. They…

Zhenghai Xue, Longtao Zheng, Qian Liu, Yingru Li, et al.
Published
Sep 2025
Citations
139
Code
403 stars
Read summary Original
163

arXiv.org

Reasoning Vectors: Transferring Chain-of-Thought Capabilities via Task Arithmetic

This paper introduces the concept of a reasoning vector, which captures the reasoning capability instilled by reinforcement learning (RL) in large language models (LLMs) and can be transferred to other models via simple tensor arithmetic. The authors source two publicly available QWEN2.5 models (1.5B and 7B) with identical initialization, one fine-tuned…

Mohammad Zbeeb, Hasan Abed Al Kader Hammoud, Bernard Ghanem
Published
Sep 2025
Citations
5
Code
Not linked
Read summary Original
164

arXiv.org

LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model

The paper introduces LLaVA-Critic-R1, a multimodal critic model trained via reinforcement learning (RL) on preference-labeled critic data, which surprisingly also becomes a strong policy model. By reformulating critic data into verifiable tasks and using GRPO with preference and format rewards, the model improves over its base (Qwen-2.5-VL-7B) by +5.7% on…

Xiyao Wang, Chunyuan Li, Jianwei Yang, Kai Zhang, et al.
Published
Aug 2025
Citations
30
Code
4.7K stars
Read summary Original
165

arXiv.org

R-4B: Incentivizing General-Purpose Auto-Thinking Capability in MLLMs via Bi-Mode Annealing and Reinforce Learning

The paper introduces R-4B, a multimodal large language model (MLLM) with auto-thinking capability, which adaptively decides whether to engage in step-by-step reasoning based on problem complexity. R-4B is trained in two stages: bi-mode annealing, which uses a curated dataset of reasoning and non-reasoning examples to equip the model with both thinking and…

Qi Yang, Bolin Ni, Shiming Xiang, Han Hu, et al.
Published
Aug 2025
Citations
16
Code
141 stars
Read summary Original
166

arXiv.org

Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning

The paper introduces Pref-GRPO, a reinforcement learning method for text-to-image (T2I) generation that replaces pointwise reward score maximization with pairwise preference fitting to address reward hacking. It identifies the root cause of reward hacking as an 'illusory advantage' arising from normalizing tightly clustered pointwise scores by a small…

Yibin Wang, Zhimin Li, Yuhang Zang, Yujie Zhou, et al.
Published
Aug 2025
Citations
90
Code
276 stars
Read summary Original
167

arXiv.org

rStar2-Agent: Agentic Reasoning Technical Report

rStar2-Agent is a 14B math reasoning model trained with agentic reinforcement learning (RL) to achieve frontier-level performance, rivaling or surpassing DeepSeek-R1 (671B). It uses Python coding tools within a dedicated environment, enabling advanced cognitive behaviors like careful tool use and reflection on execution feedback. Key innovations include an…

Ning Shang, Yifei Liu, Yi Zhu, Li Lyna Zhang, et al.
Published
Aug 2025
Citations
45
Code
1.4K stars
Read summary Original
168

arXiv.org

Self-Rewarding Vision-Language Model via Reasoning Decomposition

Vision-SR1 is a three-stage self-rewarding reinforcement learning method for vision-language models (VLMs) that improves visual reasoning without external visual supervision. It decomposes VLM reasoning into visual perception and language reasoning, prompting the model to first generate a self-contained visual description, then re-prompting it to answer…

Zongxia Li, Wenhao Yu, Chengsong Huang, Zhenwen Liang, et al.
Published
Aug 2025
Citations
91
Code
180 stars
Read summary Original
169

arXiv.org

TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling

TreePO is a reinforcement learning framework for large language models that replaces standard independent trajectory sampling with a heuristic tree-based rollout mechanism. It models sequence generation as a tree search with dynamic branching and fixed-length segment decoding, leveraging shared prefixes via KV-cache reuse to reduce compute. The method…

Yizhi Li, Qingshui Gu, Zhoufutu Wen, Ziniu Li, et al.
Published
Aug 2025
Citations
59
Code
65 stars
Read summary Original
170

Meta AI

Deep Think with Confidence

Deep Think with Confidence (DeepConf) is a test-time method that improves LLM reasoning efficiency and accuracy by using model-internal confidence signals to filter low-quality reasoning traces. It addresses the diminishing returns and high computational overhead of self-consistency with majority voting. DeepConf operates in offline mode (filtering…

Yichao Fu, Xuewei Wang, Yuandong Tian, Jiawei Zhao
Published
Aug 2025
Citations
151
Code
409 stars
Read summary Original
171

arXiv.org

DuPO: Enabling Reliable LLM Self-Verification via Dual Preference Optimization

DuPO is a dual learning-based preference optimization framework that generates annotation-free feedback for LLM optimization. It addresses limitations of RLVR (costly labels, restricted to verifiable tasks) and traditional dual learning (strictly dual task pairs). DuPO decomposes a primal task's input into known and unknown components, then constructs a…

Shuaijie She, Yu Bao, Yu Lu, Lu Xu, et al.
Published
Aug 2025
Citations
5
Code
Not linked
Read summary Original
172

arXiv.org

Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR

The paper proposes an online Self-play with Variational problem Synthesis (SvS) strategy for Reinforcement Learning with Verifiable Rewards (RLVR) to address entropy collapse and plateaued Pass@k performance in LLM reasoning. Standard RLVR improves Pass@1 but reduces generation diversity, limiting Pass@k. SvS uses the policy's correct solutions to…

Xiao Liang, Zhongzhi Li, Yeyun Gong, Yelong Shen, et al.
Published
Aug 2025
Citations
53
Code
55 stars
Read summary Original
173

arXiv.org

SSRL: Self-Search Reinforcement Learning

The paper introduces Self-Search RL (SSRL), a method that trains LLMs to answer search-based questions using only their internal knowledge, without external search engines. The authors first quantify LLMs' intrinsic search capability via repeated sampling (Self-Search), showing strong scaling with inference budget and high pass@k on benchmarks like…

Yuchen Fan, Kaiyan Zhang, Heng Zhou, Yuxin Zuo, et al.
Published
Aug 2025
Citations
11
Code
210 stars
Read summary Original
174

arXiv.org

We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoning

WE-MATH 2.0 is a unified system to enhance multimodal large language models' (MLLMs) mathematical reasoning. It integrates a structured knowledge system, model-centric data modeling, and reinforcement learning (RL). The MathBook Knowledge System is a five-level hierarchy with 491 knowledge points and 1,819 fundamental principles. MathBook-Standard is a…

Runqi Qiao, Qiuna Tan, Peiqing Yang, Yanzi Wang, et al.
Published
Aug 2025
Citations
30
Code
170 stars
Read summary Original
175

Annual Meeting of the Association for Computational Linguistics

ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability

ReasonRank is a reasoning-intensive listwise passage reranker that outperforms existing baselines on reasoning-heavy IR benchmarks. The authors address the scarcity of reasoning-intensive training data by proposing an automated synthesis framework that collects queries from diverse domains (complex QA, coding, math, web search) and uses DeepSeek-R1 to…

Wenhan Liu, Xinyu Ma, Weiwei Sun, Yutao Zhu, et al.
Published
Aug 2025
Citations
47
Code
Not linked
Read summary Original
176

arXiv.org

On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification

This paper analyzes the generalization gap between Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) for Large Language Models. The authors show that the SFT gradient can be interpreted as a policy gradient with an implicit, sparse reward that is inversely proportional to the model's token probability, leading to unstable updates and poor…

Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, et al.
Published
Aug 2025
Citations
112
Code
1.1K stars
Read summary Original
177

arXiv.org

R-Zero: Self-Evolving Reasoning LLM from Zero Data

R-Zero is a fully autonomous framework for training reasoning LLMs without any pre-existing tasks or labels. It initializes two models from a single base LLM: a Challenger, trained via GRPO to generate questions at the edge of the Solver's capability (using an uncertainty reward based on self-consistency, plus repetition and format penalties), and a…

Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, et al.
Published
Aug 2025
Citations
167
Code
831 stars
Read summary Original