The Year in AI Papers
The year/Topics/Reasoning

Topic area

Reasoning

Every collection across reasoning.

Papers
177
Research labs
7
Official code
113

101–150 of 177 papers in this topic area

101

arXiv.org

When Reasoning Meets Its Laws

The paper introduces the Laws of Reasoning (LORE), a framework formalizing desired reasoning behaviors in Large Reasoning Models (LRMs). LORE posits a compute law (reasoning compute scales linearly with question complexity) and an accuracy law (accuracy decays exponentially with complexity). Since complexity is hard to measure, the laws are approximated…

Junyu Zhang, Yifan Sun, Tianang Leng, Jingyan Shen, et al.
Published
Dec 2025
Citations
2
Code
38 stars
Read summary Original
102

arXiv.org

Seed-Prover 1.5: Mastering Undergraduate-Level Theorem Proving via Learning from Experience

Seed-Prover 1.5 is a formal theorem-proving system for Lean developed by ByteDance Seed AI4Math. It uses an agentic prover trained via large-scale reinforcement learning (RL) with tool integration (Lean verification, Mathlib search, Python execution) and a sketch model trained with rubric RL to bridge natural language proofs and Lean sketches. The system…

Jiangjie Chen, Wenxiang Chen, Jiacheng Du, Jinyi Hu, et al.
Published
Dec 2025
Citations
29
Code
Not linked
Read summary Original
103

arXiv.org

T-pro 2.0: An Efficient Russian Hybrid-Reasoning Model and Playground

T-pro 2.0 is an open-weight Russian LLM for hybrid reasoning and efficient inference, supporting direct answering and reasoning-trace generation. It uses a Cyrillic-dense tokenizer and an adapted EAGLE speculative-decoding pipeline to reduce latency. The authors release the model weights, the T-Wix 500k instruction corpus, the T-Math reasoning benchmark,…

Dmitrii Stoianov, Danil Taranets, Olga Tsymboi, Ramil Latypov, et al.
Published
Dec 2025
Citations
1
Code
Not linked
Read summary Original
104

arXiv.org

Native Parallel Reasoner: Reasoning in Parallelism via Self-Distilled Reinforcement Learning

The paper introduces Native Parallel Reasoner (NPR), a teacher-free framework that enables large language models to develop genuine parallel reasoning capabilities through self-distillation and reinforcement learning. NPR uses a three-stage progressive training paradigm: Stage 1 applies format-following RL (DAPO) to induce a structured parallel format,…

Tong Wu, Yang Liu, Jun Bai, Zixia Jia, et al.
Published
Dec 2025
Citations
11
Code
113 stars
Read summary Original
105

Qwen

Stabilizing Reinforcement Learning with LLMs: Formulation and Practices

This paper proposes a formulation for reinforcement learning (RL) with large language models (LLMs), showing that optimizing sequence-level rewards via token-level objectives is a first-order approximation that holds only when training–inference discrepancy and policy staleness are minimized. The authors introduce MiniRL, a minimalist REINFORCE-based…

Chujie Zheng, Kai Dang, Bowen Yu, Mingze Li, et al.
Published
Dec 2025
Citations
35
Code
Not linked
Read summary Original
106

DeepSeek

DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning

DeepSeekMath-V2 is a large language model for natural-language theorem proving, built on DeepSeek-V3.2-Exp-Base, that achieves self-verifiable mathematical reasoning. The authors argue that final-answer rewards are insufficient because correct answers do not guarantee correct reasoning and are inapplicable to theorem proving. They train a verifier using…

Zhihong Shao, Yuxiang Luo, Chengda Lu, Z. Z. Ren, et al.
Published
Nov 2025
Citations
60
Code
1.6K stars
Read summary Original
107

Together AI

Escaping the Verifier: Learning to Reason via Demonstrations

The paper introduces RARO (Relativistic Adversarial Reasoning Optimization), a method to train large language models (LLMs) to reason using only expert demonstrations, without task-specific verifiers or human preferences. RARO frames the problem as inverse reinforcement learning, setting up an adversarial game between a policy and a relativistic critic…

Locke Cai, Ivan Provilkov
Published
Nov 2025
Citations
5
Code
Not linked
Read summary Original
108

arXiv.org

Latent Collaboration in Multi-Agent Systems

LatentMAS is a training-free framework enabling multi-agent systems (MAS) to collaborate entirely in the continuous latent space, bypassing text-based communication. Each agent generates latent thoughts via auto-regressive last-layer hidden states, and a shared latent working memory (KV caches) transfers these representations losslessly to subsequent…

Jiaru Zou, Ruizhong Qiu, Gaotang Li, Xiyuan Yang, et al.
Published
Nov 2025
Citations
35
Code
1.1K stars
Read summary Original
109

arXiv.org

OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipe

OpenMMReasoner introduces a fully transparent two-stage recipe for training large multimodal reasoning models (LMRMs), covering supervised fine-tuning (SFT) and reinforcement learning (RL). The SFT stage uses an 874K-sample cold-start dataset built from 103K raw questions, distilled with a strong teacher model (Qwen3-VL-235B-Instruct) and scaled via ×8…

Kaichen Zhang, Keming Wu, Zuhao Yang, Bo Li, et al.
Published
Nov 2025
Citations
31
Code
165 stars
Read summary Original
110

arXiv.org

Step-Audio-R1 Technical Report

Step-Audio-R1 is the first audio reasoning model to successfully benefit from deliberate thinking, addressing the historical issue where audio language models performed worse with extended reasoning. The team identifies the root cause as 'textual surrogate reasoning,' where models reason from transcripts rather than acoustic features. They propose…

Fei Tian, Xiangyu Tony Zhang, Yuxin Zhang, Haoyang Zhang, et al.
Published
Nov 2025
Citations
35
Code
Not linked
Read summary Original
111

arXiv.org

P1: Mastering Physics Olympiads with Reinforcement Learning

The P1 technical report introduces a family of open-source physics reasoning models trained entirely via reinforcement learning (RL). P1-235B-A22B is the first open-source model to achieve gold-medal performance at the International Physics Olympiad 2025 (IPhO 2025), scoring 21.2/30 and ranking 3rd behind Gemini-2.5-Pro and GPT-5. It won 12 gold and 1…

Jiacheng Chen, Qianjia Cheng, Fangchen Yu, Haiyuan Wan, et al.
Published
Nov 2025
Citations
4
Code
89 stars
Read summary Original
112

arXiv.org

VIDEOP2R: Video Understanding from Perception to Reasoning

The paper introduces VIDEOP2R, a process-aware reinforcement fine-tuning (RFT) framework for video understanding that explicitly separates perception and reasoning. It addresses limitations of existing video RFT methods, which treat video reasoning as a single process and assign a single final reward, leading to poor credit assignment. VIDEOP2R consists of…

Yifan Jiang, Yueying Wang, Rui Zhao, Toufiq Parag, et al.
Published
Nov 2025
Citations
8
Code
Not linked
Read summary Original
113

arXiv.org

Think-at-Hard: Selective Latent Iterations to Improve Reasoning Language Models

The paper introduces Think-at-Hard (TaH), a looped transformer that selectively applies latent iterations to improve reasoning in small language models. The authors identify a 'latent overthinking' problem where always iterating can flip correct predictions into errors. An oracle policy that iterates only on mispredicted tokens boosts accuracy by up to…

Tianyu Fu, Yichen You, Zekai Chen, Guohao Dai, et al.
Published
Nov 2025
Citations
18
Code
78 stars
Read summary Original
114

arXiv.org

DRIVE: Data Curation Best Practices for Reinforcement Learning with Verifiable Reward in Competitive Code Generation

This paper from Tencent's Hunyuan Team presents a two-stage reinforcement learning with verifiable reward (RLVR) framework for competitive programming code generation, addressing the underexplored area of data curation and curriculum design. The method begins with SFT distilled from strong open-source models, then applies two RL stages: first, entropy…

Speed Zhu, Jianwei Cai, Guang Chen, Lulu Wu, et al.
Published
Nov 2025
Citations
2
Code
8 stars
Read summary Original
115

arXiv.org

Tiny Model, Big Logic: Diversity-Driven Optimization Elicits Large-Model Reasoning Ability in VibeThinker-1.5B

This technical report from Sina Weibo Inc. introduces VibeThinker-1.5B, a 1.5B-parameter dense model that challenges the assumption that small models lack robust reasoning capabilities. Developed with a post-training methodology based on the 'Spectrum-to-Signal Principle (SSP)', the approach decouples SFT and RL objectives: the SFT 'Spectrum Phase' uses…

Sen Xu, Yi Zhou, Wei Wang, Jixin Min, et al.
Published
Nov 2025
Citations
8
Code
1.5K stars
Read summary Original
116

arXiv.org

V-Thinker: Interactive Thinking with Images

V-Thinker is a general-purpose multimodal reasoning assistant that enables interactive, vision-centric thinking through end-to-end reinforcement learning. It comprises a Data Evolution Flywheel that automatically synthesizes, evolves, and verifies interactive reasoning datasets across diversity, quality, and difficulty, producing the V-Interaction-400K…

Runqi Qiao, Qiuna Tan, Minghan Yang, Guanting Dong, et al.
Published
Nov 2025
Citations
19
Code
177 stars
Read summary Original
117

arXiv.org

When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought

The paper introduces MIRA (Multimodal Imagination for Reasoning Assessment), a benchmark designed to evaluate multimodal large language models (MLLMs) on tasks that require generating or using intermediate visual images (e.g., sketches, diagrams) for successful reasoning, mirroring human 'drawing to think'. MIRA contains 546 problems across 20 task types…

Yiyang Zhou, Haoqin Tu, Zijun Wang, Zeyu Wang, et al.
Published
Nov 2025
Citations
10
Code
Not linked
Read summary Original
118

arXiv.org

ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning

ThinkMorph is a unified multimodal model fine-tuned on ~24K interleaved reasoning traces across four tasks (Jigsaw Assembly, Spatial Navigation, Visual Search, Chart Refocus) to enable complementary text-image chain-of-thought reasoning. It outperforms its base model Bagel-7B by an average of 34.74% on vision-centric benchmarks, with gains of 85.84% on…

Jiawei Gu, Yunzhuo Hao, Huichen Will Wang, Linjie Li, et al.
Published
Oct 2025
Citations
40
Code
192 stars
Read summary Original
119

arXiv.org

The End of Manual Decoding: Towards Truly End-to-End Language Models

The paper introduces AutoDeco, a novel architecture that makes LLM generation truly end-to-end by learning to predict token-level temperature and top-p values during decoding. This replaces manual hyperparameter tuning with a dynamic, self-regulating process. AutoDeco uses lightweight heads on top of frozen base models, trained with a differentiable soft…

Zhichao Wang, Dongyang Ma, Xinting Huang, Deng Cai, et al.
Published
Oct 2025
Citations
6
Code
73 stars
Read summary Original
120

Independent research

$π_\texttt{RL}$: Online RL Fine-tuning for Flow-based Vision-Language-Action Models

The paper introduces π_RL, a framework for online reinforcement learning (RL) fine-tuning of flow-based Vision-Language-Action (VLA) models such as π0 and π0.5. It addresses the challenge of intractable action log-likelihoods in flow matching by proposing two methods: Flow-Noise, which models the denoising process as a discrete-time MDP with a learnable…

Kang Chen, Zhihao Liu, Tonghe Zhang, Zhen Guo, et al.
Published
Oct 2025
Citations
3
Code
Not linked
Read summary Original
121

arXiv.org

Scaling Latent Reasoning via Looped Language Models

The paper introduces Ouro, a family of pre-trained Looped Language Models (LoopLM) that integrate iterative computation in latent space during pre-training, using an entropy-regularized objective for adaptive depth allocation. Trained on 7.7T tokens, Ouro 1.4B and 2.6B models match or exceed the performance of 4B and 8B standard transformers, achieving…

Rui-Jie Zhu, Zixuan Wang, Kai Hua, Tianyu Zhang, et al.
Published
Oct 2025
Citations
104
Code
Not linked
Read summary Original
122

arXiv.org

Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning

Video-Thinker is a framework that extends the 'Thinking with Images' paradigm to video reasoning by enabling Multimodal Large Language Models (MLLMs) to autonomously use intrinsic 'grounding' (temporal localization) and 'captioning' (visual comprehension) capabilities within chain-of-thought reasoning, eliminating the need for external tools. The authors…

Shijian Wang, Jiarui Jin, Xingjian Wang, Linxin Song, et al.
Published
Oct 2025
Citations
21
Code
160 stars
Read summary Original
123

arXiv.org

LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts

LoongRL is a data-driven reinforcement learning (RL) method for advanced long-context reasoning. It introduces KeyChain, a synthesis approach that transforms short multi-hop QA into high-difficulty long-context tasks by inserting UUID chains that hide the true question among distracting documents. Solving these tasks requires tracing the chain, identifying…

Siyuan Wang, Gaokai Zhang, Li Lyna Zhang, Ning Shang, et al.
Published
Oct 2025
Citations
21
Code
Not linked
Read summary Original
124

arXiv.org

Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model

Ring-1T is the first open-source, state-of-the-art thinking model with 1 trillion total parameters and about 50 billion activated per token, built on the Ling 2.0 architecture. Training at this scale posed challenges like train-inference misalignment, rollout inefficiencies, and RL system bottlenecks. The authors introduced three innovations: IcePop, which…

Ling Team, Anqi Shen, Baihui Li, Bin Hu, et al.
Published
Oct 2025
Citations
32
Code
Not linked
Read summary Original
125

arXiv.org

BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping

This paper introduces BAPO (Balanced Policy Optimization with Adaptive Clipping), a method to stabilize off-policy reinforcement learning for large language models (LLMs). Off-policy RL, where stale data from past policies is used, suffers from unstable optimization, entropy collapse, and gradient explosions. The authors identify two key issues: (1) an…

Zhiheng Xi, Xin Guo, Yang Nan, Enyu Zhou, et al.
Published
Oct 2025
Citations
48
Code
94 stars
Read summary Original
126

arXiv.org

Chem-R: Learning to Reason as a Chemist

Chem-R is a chemical reasoning model designed to emulate the deliberative processes of chemists, addressing three key challenges in current LLMs: lack of chemical fundamentals, unreliable reasoning trajectories, and imbalanced performance across tasks. The model is trained via a three-phase framework: (1) Chemical Foundation Training, which fine-tunes on…

Weida Wang, Benteng Chen, Di Zhang, Wanhao Liu, et al.
Published
Oct 2025
Citations
11
Code
30 stars
Read summary Original
127

Neural Information Processing Systems

A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning

This paper introduces a theoretical framework for sampling-based test-time scaling in LLM reasoning, decomposing reasoning error into estimation error and model error. It analyzes self-consistency (SC) and perplexity (PPL), finding SC has linear estimation error convergence while PPL has exponential but suffers from high model error and degradation when…

Zhi Zhou, Yuhao Tan, Zenan Li, Yuan Yao, et al.
Published
Oct 2025
Citations
11
Code
14 stars
Read summary Original
128

arXiv.org

Agentic Entropy-Balanced Policy Optimization

The paper introduces Agentic Entropy-Balanced Policy Optimization (AEPO), an agentic reinforcement learning algorithm for training multi-turn web agents. It addresses two entropy-driven challenges: 'High-Entropy Rollout Collapse' (over-branching on few trajectories due to consecutive high-entropy tool-call steps) and 'High-Entropy Token Gradient Clipping'…

Guanting Dong, Licheng Bao, Zhongyuan Wang, Kangzhi Zhao, et al.
Published
Oct 2025
Citations
26
Code
1.1K stars
Read summary Original
129

arXiv.org

Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization

This paper investigates how attention dynamics in LLMs can reveal reasoning patterns and improve reinforcement learning (RL) credit assignment. The authors analyze attention heads, classifying them as locally or globally focused. Locally focused heads show a sawtooth pattern near the diagonal, indicating phrasal chunks, while globally focused heads…

Yang Li, Zhichen Dong, Yuhan Sun, Weixun Wang, et al.
Published
Oct 2025
Citations
25
Code
Not linked
Read summary Original
130

NVIDIA

QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs

QeRL is a framework that combines NVFP4 4-bit quantization with Low-Rank Adaptation (LoRA) to accelerate and reduce memory usage in reinforcement learning (RL) for large language models (LLMs). The authors find that quantization noise increases policy entropy, which enhances exploration during RL, contrary to its detrimental effect in supervised…

Wei Huang, Yi Ge, Shuai Yang, Yicheng Xiao, et al.
Published
Oct 2025
Citations
16
Code
512 stars
Read summary Original
131

arXiv.org

MM-HELIX: Boosting Multimodal Long-Chain Reflective Reasoning with Holistic Platform and Adaptive Hybrid Policy Optimization

The paper introduces MM-HELIX, a benchmark of 1,260 samples across 42 multimodal tasks (algorithms, graphs, puzzles, games) with five difficulty levels, designed to evaluate long-chain reflective reasoning in MLLMs. Evaluation of 23 models shows significant deficits; even GPT-5 achieves only 58.1% accuracy, and open-source models lag far behind. To improve…

Xiangyu Zhao, Junming Lin, Tianhao Liang, Yifan Zhou, et al.
Published
Oct 2025
Citations
3
Code
69 stars
Read summary Original
132

arXiv.org

TaTToo: Tool-Grounded Thinking PRM for Test-Time Scaling in Tabular Reasoning

TATTOO is a novel table-grounded Process Reward Model (PRM) designed to improve test-time scaling (TTS) for large reasoning models (LRMs) on tabular reasoning tasks. The authors identify that existing PRMs fail on table-specific operations like sub-table retrieval and schema interaction, leading to performance bottlenecks. TATTOO addresses this by…

Jiaru Zou, Soumya Roy, Vinay Kumar Verma, Ziyi Wang, et al.
Published
Oct 2025
Citations
10
Code
Not linked
Read summary Original
133

arXiv (Cornell University)

Less is More: Recursive Reasoning with Tiny Networks

The paper introduces Tiny Recursive Model (TRM), a simplified recursive reasoning approach that outperforms the Hierarchical Reasoning Model (HRM) and many large language models on hard puzzle tasks. TRM uses a single tiny 2-layer network (7M parameters) that recursively improves its answer, eliminating the need for fixed-point theorems, hierarchical…

Alexia Jolicoeur-Martineau
Published
Oct 2025
Citations
4
Code
6.6K stars
Read summary Original
134

Annual Meeting of the Association for Computational Linguistics

Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward

This paper addresses the exploration collapse in Reinforcement Learning with Verifiable Rewards (RLVR) for large language models, where performance plateaus as policy entropy collapses. The authors identify the systematic elimination of valuable low-probability exploratory tokens, termed 'reasoning sparks' (e.g., 'wait', 'however'), as a key cause, while…

Guanhua Huang, Tingqiang Xu, Mingze Wang, Qi Yi, et al.
Published
Oct 2025
Citations
23
Code
44 stars
Read summary Original
135

arXiv.org

ExGRPO: Learning to Reason from Experience

The paper introduces ExGRPO, a framework for improving reinforcement learning from verifiable rewards (RLVR) for large reasoning models by reusing past rollout experiences. The authors first analyze what makes a reasoning experience valuable, identifying that medium-difficulty questions (with rollout correctness between 25% and 75%) and low-entropy…

Runzhe Zhan, Yafu Li, Zhi Wang, Xiaoye Qu, et al.
Published
Oct 2025
Citations
41
Code
Not linked
Read summary Original
136

arXiv.org

More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models

This paper investigates the dual nature of reasoning in Vision-Language Models (VLMs). The authors find that while reasoning improves logical inference, longer reasoning lengths can impair perceptual grounding, leading to errors on basic visual questions. They attribute this to 'visual forgetting,' where models increasingly ignore visual input during…

Xinyu Tian, Shu Zou, Zhaoyuan Yang, Mengqi He, et al.
Published
Sep 2025
Citations
26
Code
389 stars
Read summary Original
137

Meta AI

TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning

TruthRL is a reinforcement learning framework that directly optimizes the truthfulness of large language models (LLMs) by using a ternary reward scheme that distinguishes correct answers, hallucinations, and abstentions. Implemented with GRPO, it rewards correct answers, penalizes hallucinations, and treats abstentions neutrally, encouraging models to…

Zhepei Wei, Xiao Yang, Kai Sun, Jiaqi Wang, et al.
Published
Sep 2025
Citations
16
Code
Not linked
Read summary Original
138

arXiv.org

Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play

Vision-Zero is a label-free, domain-agnostic multi-agent self-play framework for self-evolving vision-language models (VLMs) through competitive visual games generated from arbitrary images. It trains VLMs in a 'Who Is the Spy?'-style game where civilians see an image and the spy sees a blank input, requiring strategic reasoning and communication. The…

Qinsi Wang, Bo Liu, Tianyi Zhou, Jing Shi, et al.
Published
Sep 2025
Citations
33
Code
165 stars
Read summary Original
139

arXiv.org

DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search

DeepSearch is a framework that integrates Monte Carlo Tree Search (MCTS) directly into reinforcement learning with verifiable rewards (RLVR) training to overcome exploration bottlenecks that cause performance plateaus. Unlike methods that use tree search only at inference, DeepSearch embeds structured search into the training loop, enabling systematic…

Fang Wu, Weihao Xuan, Heli Qi, Ximing Lu, et al.
Published
Sep 2025
Citations
10
Code
34 stars
Read summary Original
140

arXiv.org

Multiplayer Nash Preference Optimization

This paper introduces Multiplayer Nash Preference Optimization (MNPO), a framework that generalizes Nash learning from human feedback (NLHF) to n-player games for aligning large language models with human preferences. MNPO formulates alignment as a game where each policy competes against a population of opponents while being regularized toward a reference…

Fang Wu, Xu Huang, Weihao Xuan, Zhiwei Zhang, et al.
Published
Sep 2025
Citations
6
Code
35 stars
Read summary Original
141

arXiv.org

Language Models Can Learn from Verbal Feedback Without Scalar Rewards

The paper proposes Feedback-Conditional Policy (FCP), a method for training LLMs directly from verbal feedback without converting it into scalar rewards. FCP treats feedback as a conditioning signal, learning a policy πθ(o|x,c) that approximates the posterior P_off(o|x,c) ∝ π_ref(o|x)·p_env(c|x,o) via maximum likelihood on offline data. An online…

Renjie Luo, Zichen Liu, Xiangyan Liu, Chao Du, et al.
Published
Sep 2025
Citations
15
Code
65 stars
Read summary Original
142

arXiv.org

Variational Reasoning for Language Models

The paper introduces a variational reasoning framework for language models, treating thinking traces as latent variables and optimizing them via variational inference. It extends the evidence lower bound (ELBO) to a multi-trace IWAE-style objective and proposes a forward-KL formulation for stable training of the variational posterior. The authors show that…

Xiangxin Zhou, Zichen Liu, Haonan Wang, Chao Du, et al.
Published
Sep 2025
Citations
6
Code
60 stars
Read summary Original
143

Independent research

Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning

The paper introduces Quantile Advantage Estimation (QAE) to stabilize Reinforcement Learning with Verifiable Rewards (RLVR) for LLM reasoning. It identifies that mean-baseline methods like GRPO and DAPO cause entropy collapse or explosion, harming performance. QAE replaces the mean reward baseline with a group-wise K-quantile, creating a two-regime gate:…

Junkang Wu, Kexin Huang, Jiancan Wu, An Zhang, et al.
Published
Sep 2025
Citations
10
Code
29 stars
Read summary Original
144

arXiv.org

EPO: Entropy-regularized Policy Optimization for LLM Agents Reinforcement Learning

The paper identifies the exploration-exploitation cascade failure in multi-turn LLM agent reinforcement learning, where shared policy parameters across turns and sparse terminal rewards cause severe entropy oscillations that destabilize training. To address this, the authors propose Entropy-regularized Policy Optimization (EPO), which combines…

Wujiang Xu, Wentian Zhao, Zhenting Wang, Yu-Jhe Li, et al.
Published
Sep 2025
Citations
10
Code
40 stars
Read summary Original
145

Independent research

Verifying Meta-Awareness via Predictive Rewards in Reasoning Models

The paper introduces MAPR (Meta-Awareness via Predictive Reward), a reinforcement learning framework that enhances reasoning models by training them to predict their own rollout statistics—specifically solution length, pass-rate, and concepts used—and rewarding alignment with actual statistics. This self-generated meta-prediction is verified against…

Yoonjeon Kim, Doohyuk Jang, Eunho Yang
Published
Sep 2025
Citations
2
Code
12 stars
Read summary Original
146

arXiv.org

No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping

The paper introduces RL-ZVP, a reinforcement learning algorithm for LLMs that exploits zero-variance prompts—where all sampled responses to a prompt receive identical rewards—which standard methods like GRPO discard due to zero advantage. RL-ZVP assigns advantages to these prompts based on response correctness (positive for all-correct, negative for…

Thanh-Long V. Le, Myeongho Jeon, Kim Vu, Viet Lai, et al.
Published
Sep 2025
Citations
41
Code
Not linked
Read summary Original
147

arXiv.org

MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources

This paper introduces MMR1, a framework for enhancing multimodal reasoning in large models. It addresses two key limitations: the lack of open, large-scale, high-quality long chain-of-thought (CoT) data, and the instability of reinforcement learning (RL) algorithms, specifically gradient vanishing in Group Relative Policy Optimization (GRPO) when reward…

Sicong Leng, Jing Wang, Jiaxi Li, Hao Zhang, et al.
Published
Sep 2025
Citations
35
Code
217 stars
Read summary Original
148

arXiv.org

Tree Search for LLM Agent Reinforcement Learning

The paper introduces Tree-GRPO, a reinforcement learning method for LLM agents that uses tree search for rollouts, with each node representing a complete agent interaction step (Thought-Action-Observation). This approach shares common prefixes across rollouts, increasing the number of samples under a fixed token/tool-call budget. Tree-GRPO estimates group…

Yuxiang Ji, Ziyu Ma, Yong Wang, Guanhua Chen, et al.
Published
Sep 2025
Citations
52
Code
392 stars
Read summary Original
149

arXiv.org

VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models

The paper introduces VCRL, a curriculum reinforcement learning framework for large language models (LLMs) that dynamically adjusts training sample difficulty based on the variance of group rewards in rollout-based RL (e.g., GRPO). The authors observe that samples with moderate difficulty yield higher reward variance, while too-easy or too-hard samples…

Guochao Jiang, Wenfeng Feng, Guofeng Quan, Chuzhan Hao, et al.
Published
Sep 2025
Citations
27
Code
Not linked
Read summary Original
150

Annual Meeting of the Association for Computational Linguistics

Reinforcement Learning on Pre-Training Data

The paper introduces RLPT (Reinforcement Learning on Pre-Training data), a new training-time scaling paradigm for LLMs that applies reinforcement learning directly to pre-training data, eliminating the need for human annotations. RLPT uses a next-segment reasoning objective where the model predicts a subsequent text segment, and a generative reward model…

Siheng Li, Kejiao Li, Zenan Xu, Guanhua Huang, et al.
Published
Sep 2025
Citations
11
Code
Not linked
Read summary Original