The Year in AI Papers
The year/Topics/RL for reasoning

Research collection

RL for reasoning

Reinforcement learning applied to train reasoning ability, including RLVR, verifiable rewards, reward design, and exploration for reasoning models.

Papers
136
Research labs
7
Official code
85

51–100 of 136 papers in this collection

51

arXiv.org

BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning

The paper introduces BandPO, a method for LLM reinforcement learning that replaces the fixed clipping bounds of PPO/GRPO with dynamic, probability-aware bounds derived from f-divergence trust regions. The authors identify a bottleneck in canonical clipping: fixed bounds limit the upward update margin for low-probability actions, suppressing high-advantage…

Yuan Li, Bo Wang, Yufei Gao, Yuqian Yao, et al.
Published
Mar 2026
Citations
2
Code
49 stars
Read summary Original
52

arXiv.org

Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learning

The paper introduces GOLF, a reinforcement learning (RL) framework that leverages group-level natural language (NL) feedback to improve exploration efficiency in training large language models (LLMs). Unlike standard RL that relies solely on scalar rewards, GOLF aggregates two complementary feedback sources: external critiques (pinpointing errors or…

Lei Huang, Xiang Cheng, Chenxiao Zhao, Guobin Shen, et al.
Published
Mar 2026
Citations
2
Code
18 stars
Read summary Original
53

Independent research

Edit in 2D, Verify in 3D: Reinforcement Learning for Multi-view Consistent Scene Editing

The paper introduces RL3DEdit, a single-pass framework for multi-view consistent 3D scene editing that uses reinforcement learning (RL) to augment a 2D diffusion editor (FLUX-Kontext) with 3D consistency priors. The authors observe that while generating multi-view consistent images is challenging, verifying consistency is tractable, making RL a suitable…

Jiyuan Wang, Chunyu Lin, Lei Sun, Zhi Cao, et al.
Published
Mar 2026
Citations
7
Code
203 stars
Read summary Original
54

arXiv.org

Heterogeneous Agent Collaborative Reinforcement Learning

The paper introduces Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a new RLVR problem where heterogeneous LLM agents share verified rollouts during training to mutually improve, while executing independently at inference. Unlike LLM-based MARL, HACRL does not require coordinated deployment, and unlike distillation, it enables…

Zhixia Zhang, Zixuan Huang, Gongxun Li, Huaiyang Wang, et al.
Published
Mar 2026
Citations
12
Code
Not linked
Read summary Original
55

arXiv.org

CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation

CUDA Agent is a large-scale agentic reinforcement learning (RL) system for generating high-performance CUDA kernels. It addresses the gap between LLMs and compiler-based systems like torch.compile by combining a scalable data synthesis pipeline (crawling seed operators, LLM-based combinatorial fusion, and rubric-based filtering to create…

Weinan Dai, Hanlin Wu, Qiying Yu, Huan-ang Gao, et al.
Published
Feb 2026
Citations
32
Code
1.1K stars
Read summary Original
56

arXiv.org

Experiential Reinforcement Learning

The paper introduces Experiential Reinforcement Learning (ERL), a training paradigm for language models that embeds an explicit experience–reflection–consolidation loop into reinforcement learning. In ERL, the model first generates an initial attempt, receives environmental feedback, and then produces a self-reflection that guides a refined second attempt.…

Taiwei Shi, Sihao Chen, Bowen Jiang, Linxin Song, et al.
Published
Feb 2026
Citations
12
Code
76 stars
Read summary Original
57

arXiv.org

Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation

The paper introduces Generalized On-Policy Distillation (G-OPD), a framework that extends standard on-policy distillation (OPD) by adding a flexible reference model and a reward scaling factor λ. The authors show that OPD is a special case of dense KL-constrained RL with equal reward and KL weights. In G-OPD, λ controls the relative weight of the reward…

Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, et al.
Published
Feb 2026
Citations
87
Code
278 stars
Read summary Original
58

arXiv.org

Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models

Composition-RL addresses the issue of 'solve all' prompts (pass rate 1) that become prevalent during RLVR training, reducing effective data size. The method automatically composes multiple existing verifiable prompts into new, harder ones using Sequential Prompt Composition (SPC), which modifies one prompt with a variable from another's answer and links…

Xin Xu, Clive Bai, Kai Yang, Tianhao Chen, et al.
Published
Feb 2026
Citations
3
Code
137 stars
Read summary Original
59

arXiv.org

VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Training

VESPO addresses high variance in off-policy reinforcement learning for large language models by deriving a principled importance weight reshaping kernel from a variational formulation. It recasts weight reshaping as a measure change to an implicit proposal distribution, solving a KL-regularized objective with a variance constraint to obtain a closed-form…

Guobin Shen, Chenxiao Zhao, Xiang Cheng, Lei Huang, et al.
Published
Feb 2026
Citations
7
Code
35 stars
Read summary Original
60

arXiv.org

Weak-Driven Learning: How Weak Agents make Strong Agents Stronger

The paper introduces Weak-Driven Learning, a post-training paradigm that repurposes weak historical checkpoints as corrective signals to strengthen a strong language model, contrasting with knowledge distillation that imitates stronger teachers. The authors instantiate this with WMSS (Weak agents Make Strong agents Stronger), which combines Weak-Driven…

Zehao Chen, Gongxun Li, Tianxiang Ai, Zixuan Huang, et al.
Published
Feb 2026
Citations
4
Code
79 stars
Read summary Original
61

arXiv.org

F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

The paper introduces F-GRPO, a method to mitigate distribution sharpening in Reinforcement Learning with Verifiable Rewards (RLVR) using group-relative objectives like GRPO, DAPO, and CISPO. The authors derive a non-monotonic tail-miss probability, showing that intermediate group sizes can maximize active updates that miss rare-correct solutions, while…

Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov, Viacheslav Sinii, et al.
Published
Feb 2026
Citations
8
Code
Not linked
Read summary Original
62

arXiv.org

Length-Unbiased Sequence Policy Optimization: Revealing and Controlling Response Length Variation in RLVR

The paper analyzes response length variation in Reinforcement Learning with Verifiable Rewards (RLVR) for large language models, focusing on the length bias in GRPO and GSPO objectives. GRPO averages token contributions per trajectory, causing shorter correct responses to receive larger gradient updates, while GSPO's sequence-level clipping and Clip-Higher…

Fanfan Liu, Youyang Yin, Peng Shi, Siqi Yang, et al.
Published
Feb 2026
Citations
6
Code
8 stars
Read summary Original
63

arXiv.org

On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models

This paper establishes a theoretical framework for analyzing entropy dynamics during reinforcement fine-tuning (RFT) of large language models. The authors derive a first-order expression for entropy change from a single logit update, showing that the direction of entropy change is determined by the update direction and a discriminator score S* = p_k(H +…

Shumin Wang, Yuexiang Xie, Wenhao Zhang, Yuchang Sun, et al.
Published
Feb 2026
Citations
7
Code
Not linked
Read summary Original
64

NVIDIA

Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Text

The paper introduces Golden Goose, a method to synthesize unlimited Reinforcement Learning with Verifiable Rewards (RLVR) tasks from unverifiable internet text by converting them into multiple-choice fill-in-the-middle questions. Given a source text, an LLM masks a contiguous span of crucial reasoning steps (the ground-truth answer) and generates diverse,…

Ximing Lu, David Acuna, Jaehun Jung, Jian Hu, et al.
Published
Jan 2026
Citations
6
Code
Not linked
Read summary Original
65

arXiv.org

Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation

The paper introduces MathForge, a framework to enhance mathematical reasoning in large language models via reinforcement learning with verifiable rewards (RLVR). It addresses a lack of emphasis on harder questions in existing methods, both algorithmically and data-wise. Algorithmically, it proposes Difficulty-Aware Group Policy Optimization (DGPO), which…

Yanqi Dai, Yuxiang Ji, Xiao Zhang, Yong Wang, et al.
Published
Jan 2026
Citations
23
Code
128 stars
Read summary Original
66

Together AI

Learning to Discover at Test Time

The paper introduces TTT-Discover, a method that performs reinforcement learning at test time to solve scientific discovery problems. Unlike prior test-time scaling methods that use a frozen LLM for search, TTT-Discover continues to train the LLM on the specific test problem, using an entropic objective and PUCT-based state reuse to prioritize the most…

Mert Yuksekgonul, Daniel Koceja, Xinhao Li, Federico Bianchi, et al.
Published
Jan 2026
Citations
71
Code
613 stars
Read summary Original
67

Annual Meeting of the Association for Computational Linguistics

Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs

The paper introduces Uniqueness-Aware Reinforcement Learning (UARL) to address exploration collapse in RL-trained LLMs, where policies prematurely converge to a few dominant reasoning patterns, improving pass@1 but limiting pass@k. UARL operates at the rollout level, using an LLM judge to cluster solutions by high-level strategy and reweighting GRPO…

Zhiyuan Hu, Yucheng Wang, Yufei He, Jiaying Wu, et al.
Published
Jan 2026
Citations
9
Code
Not linked
Read summary Original
68

arXiv.org

Your Group-Relative Advantage Is Biased

This paper identifies a fundamental bias in group-relative advantage estimation used by RLVR algorithms like GRPO. The authors prove that this estimator systematically underestimates advantages for hard prompts and overestimates them for easy prompts, with bias increasing as prompt difficulty deviates from 0.5 and with smaller group sizes. To address this,…

Fengkai Yang, Zherui Chen, Xiaohan Wang, Xiaodong Lu, et al.
Published
Jan 2026
Citations
27
Code
Not linked
Read summary Original
69

Qwen

ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Ranking

ArenaRL is a reinforcement learning framework for open-ended LLM agents that replaces pointwise scalar rewards with tournament-based relative ranking to overcome discriminative collapse, where reward models fail to distinguish subtle differences among high-quality trajectories. It introduces a process-aware pairwise evaluation mechanism and a seeded…

Qiang Zhang, Boli Chen, Fanrui Zhang, Ruixue Ding, et al.
Published
Jan 2026
Citations
9
Code
274 stars
Read summary Original
70

Annual Meeting of the Association for Computational Linguistics

PaCoRe: Learning to Scale Test-Time Compute with Parallel Coordinated Reasoning

PaCoRe (Parallel Coordinated Reasoning) is a training-and-inference framework that scales test-time compute (TTC) beyond sequential reasoning limits by using parallel exploration coordinated via message passing. It runs multiple rounds: each round generates parallel reasoning trajectories, compacts them into context-bounded messages, and synthesizes these…

Jingcheng Hu, Yinmin Zhang, Shijie Shang, Xiaobo Yang, et al.
Published
Jan 2026
Citations
16
Code
338 stars
Read summary Original
71

NVIDIA

GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization

The paper identifies a flaw in applying Group Relative Policy Optimization (GRPO) to multi-reward reinforcement learning: normalizing the summed reward causes distinct reward combinations to collapse into identical advantage values, reducing training signal resolution and causing suboptimal convergence or early failure. To address this, the authors propose…

Shih-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, et al.
Published
Jan 2026
Citations
114
Code
495 stars
Read summary Original
72

arXiv.org

Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting

The paper investigates why Supervised Fine-Tuning (SFT) causes catastrophic forgetting while on-policy Reinforcement Learning (RL) preserves general capabilities. The authors identify a distributional gap: SFT data contains 'Confident Conflicts'—tokens with low probability and low entropy, where the model is confident but forced to learn a divergent label,…

Muxi Diao, Lele Yang, Wuxuan Gong, Yutong Zhang, et al.
Published
Jan 2026
Citations
12
Code
105 stars
Read summary Original
73

Google DeepMind

Emergent temporal abstractions in autoregressive models enable hierarchical reinforcement learning

This paper introduces a method for hierarchical reinforcement learning (RL) in autoregressive models by discovering and using temporally-abstract actions within the model's internal representations. The authors show that autoregressive models pretrained on next-token prediction learn latent representations of subgoals in their residual stream activations.…

Seijin Kobayashi, Yanick Schimpf, Maximilian Schlegel, Angelika Steger, et al.
Published
Dec 2025
Citations
5
Code
Not linked
Read summary Original
74

arXiv.org

Bottom-up Policy Optimization: Your Language Model Policy Secretly Contains Internal Policies

This paper introduces Bottom-up Policy Optimization (BuPO), a novel reinforcement learning (RL) paradigm for large language models (LLMs) that optimizes internal layer policies before the full language model policy. The authors decompose the LLM policy into Internal Layer Policies and Internal Modular Policies via the Transformer's residual stream. Entropy…

Yuqiao Tan, Minzheng Wang, Shizhu He, Huanxuan Liao, et al.
Published
Dec 2025
Citations
21
Code
60 stars
Read summary Original
75

arXiv.org

Seed-Prover 1.5: Mastering Undergraduate-Level Theorem Proving via Learning from Experience

Seed-Prover 1.5 is a formal theorem-proving system for Lean developed by ByteDance Seed AI4Math. It uses an agentic prover trained via large-scale reinforcement learning (RL) with tool integration (Lean verification, Mathlib search, Python execution) and a sketch model trained with rubric RL to bridge natural language proofs and Lean sketches. The system…

Jiangjie Chen, Wenxiang Chen, Jiacheng Du, Jinyi Hu, et al.
Published
Dec 2025
Citations
29
Code
Not linked
Read summary Original
76

arXiv.org

Native Parallel Reasoner: Reasoning in Parallelism via Self-Distilled Reinforcement Learning

The paper introduces Native Parallel Reasoner (NPR), a teacher-free framework that enables large language models to develop genuine parallel reasoning capabilities through self-distillation and reinforcement learning. NPR uses a three-stage progressive training paradigm: Stage 1 applies format-following RL (DAPO) to induce a structured parallel format,…

Tong Wu, Yang Liu, Jun Bai, Zixia Jia, et al.
Published
Dec 2025
Citations
11
Code
113 stars
Read summary Original
77

Qwen

Stabilizing Reinforcement Learning with LLMs: Formulation and Practices

This paper proposes a formulation for reinforcement learning (RL) with large language models (LLMs), showing that optimizing sequence-level rewards via token-level objectives is a first-order approximation that holds only when training–inference discrepancy and policy staleness are minimized. The authors introduce MiniRL, a minimalist REINFORCE-based…

Chujie Zheng, Kai Dang, Bowen Yu, Mingze Li, et al.
Published
Dec 2025
Citations
35
Code
Not linked
Read summary Original
78

DeepSeek

DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning

DeepSeekMath-V2 is a large language model for natural-language theorem proving, built on DeepSeek-V3.2-Exp-Base, that achieves self-verifiable mathematical reasoning. The authors argue that final-answer rewards are insufficient because correct answers do not guarantee correct reasoning and are inapplicable to theorem proving. They train a verifier using…

Zhihong Shao, Yuxiang Luo, Chengda Lu, Z. Z. Ren, et al.
Published
Nov 2025
Citations
60
Code
1.6K stars
Read summary Original
79

Together AI

Escaping the Verifier: Learning to Reason via Demonstrations

The paper introduces RARO (Relativistic Adversarial Reasoning Optimization), a method to train large language models (LLMs) to reason using only expert demonstrations, without task-specific verifiers or human preferences. RARO frames the problem as inverse reinforcement learning, setting up an adversarial game between a policy and a relativistic critic…

Locke Cai, Ivan Provilkov
Published
Nov 2025
Citations
5
Code
Not linked
Read summary Original
80

arXiv.org

OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipe

OpenMMReasoner introduces a fully transparent two-stage recipe for training large multimodal reasoning models (LMRMs), covering supervised fine-tuning (SFT) and reinforcement learning (RL). The SFT stage uses an 874K-sample cold-start dataset built from 103K raw questions, distilled with a strong teacher model (Qwen3-VL-235B-Instruct) and scaled via ×8…

Kaichen Zhang, Keming Wu, Zuhao Yang, Bo Li, et al.
Published
Nov 2025
Citations
31
Code
165 stars
Read summary Original
81

arXiv.org

Step-Audio-R1 Technical Report

Step-Audio-R1 is the first audio reasoning model to successfully benefit from deliberate thinking, addressing the historical issue where audio language models performed worse with extended reasoning. The team identifies the root cause as 'textual surrogate reasoning,' where models reason from transcripts rather than acoustic features. They propose…

Fei Tian, Xiangyu Tony Zhang, Yuxin Zhang, Haoyang Zhang, et al.
Published
Nov 2025
Citations
35
Code
Not linked
Read summary Original
82

arXiv.org

P1: Mastering Physics Olympiads with Reinforcement Learning

The P1 technical report introduces a family of open-source physics reasoning models trained entirely via reinforcement learning (RL). P1-235B-A22B is the first open-source model to achieve gold-medal performance at the International Physics Olympiad 2025 (IPhO 2025), scoring 21.2/30 and ranking 3rd behind Gemini-2.5-Pro and GPT-5. It won 12 gold and 1…

Jiacheng Chen, Qianjia Cheng, Fangchen Yu, Haiyuan Wan, et al.
Published
Nov 2025
Citations
4
Code
89 stars
Read summary Original
83

arXiv.org

VIDEOP2R: Video Understanding from Perception to Reasoning

The paper introduces VIDEOP2R, a process-aware reinforcement fine-tuning (RFT) framework for video understanding that explicitly separates perception and reasoning. It addresses limitations of existing video RFT methods, which treat video reasoning as a single process and assign a single final reward, leading to poor credit assignment. VIDEOP2R consists of…

Yifan Jiang, Yueying Wang, Rui Zhao, Toufiq Parag, et al.
Published
Nov 2025
Citations
8
Code
Not linked
Read summary Original
84

arXiv.org

DRIVE: Data Curation Best Practices for Reinforcement Learning with Verifiable Reward in Competitive Code Generation

This paper from Tencent's Hunyuan Team presents a two-stage reinforcement learning with verifiable reward (RLVR) framework for competitive programming code generation, addressing the underexplored area of data curation and curriculum design. The method begins with SFT distilled from strong open-source models, then applies two RL stages: first, entropy…

Speed Zhu, Jianwei Cai, Guang Chen, Lulu Wu, et al.
Published
Nov 2025
Citations
2
Code
8 stars
Read summary Original
85

arXiv.org

Tiny Model, Big Logic: Diversity-Driven Optimization Elicits Large-Model Reasoning Ability in VibeThinker-1.5B

This technical report from Sina Weibo Inc. introduces VibeThinker-1.5B, a 1.5B-parameter dense model that challenges the assumption that small models lack robust reasoning capabilities. Developed with a post-training methodology based on the 'Spectrum-to-Signal Principle (SSP)', the approach decouples SFT and RL objectives: the SFT 'Spectrum Phase' uses…

Sen Xu, Yi Zhou, Wei Wang, Jixin Min, et al.
Published
Nov 2025
Citations
8
Code
1.5K stars
Read summary Original
86

arXiv.org

V-Thinker: Interactive Thinking with Images

V-Thinker is a general-purpose multimodal reasoning assistant that enables interactive, vision-centric thinking through end-to-end reinforcement learning. It comprises a Data Evolution Flywheel that automatically synthesizes, evolves, and verifies interactive reasoning datasets across diversity, quality, and difficulty, producing the V-Interaction-400K…

Runqi Qiao, Qiuna Tan, Minghan Yang, Guanting Dong, et al.
Published
Nov 2025
Citations
19
Code
177 stars
Read summary Original
87

Independent research

$π_\texttt{RL}$: Online RL Fine-tuning for Flow-based Vision-Language-Action Models

The paper introduces π_RL, a framework for online reinforcement learning (RL) fine-tuning of flow-based Vision-Language-Action (VLA) models such as π0 and π0.5. It addresses the challenge of intractable action log-likelihoods in flow matching by proposing two methods: Flow-Noise, which models the denoising process as a discrete-time MDP with a learnable…

Kang Chen, Zhihao Liu, Tonghe Zhang, Zhen Guo, et al.
Published
Oct 2025
Citations
3
Code
Not linked
Read summary Original
88

arXiv.org

Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning

Video-Thinker is a framework that extends the 'Thinking with Images' paradigm to video reasoning by enabling Multimodal Large Language Models (MLLMs) to autonomously use intrinsic 'grounding' (temporal localization) and 'captioning' (visual comprehension) capabilities within chain-of-thought reasoning, eliminating the need for external tools. The authors…

Shijian Wang, Jiarui Jin, Xingjian Wang, Linxin Song, et al.
Published
Oct 2025
Citations
21
Code
160 stars
Read summary Original
89

arXiv.org

LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts

LoongRL is a data-driven reinforcement learning (RL) method for advanced long-context reasoning. It introduces KeyChain, a synthesis approach that transforms short multi-hop QA into high-difficulty long-context tasks by inserting UUID chains that hide the true question among distracting documents. Solving these tasks requires tracing the chain, identifying…

Siyuan Wang, Gaokai Zhang, Li Lyna Zhang, Ning Shang, et al.
Published
Oct 2025
Citations
21
Code
Not linked
Read summary Original
90

arXiv.org

Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model

Ring-1T is the first open-source, state-of-the-art thinking model with 1 trillion total parameters and about 50 billion activated per token, built on the Ling 2.0 architecture. Training at this scale posed challenges like train-inference misalignment, rollout inefficiencies, and RL system bottlenecks. The authors introduced three innovations: IcePop, which…

Ling Team, Anqi Shen, Baihui Li, Bin Hu, et al.
Published
Oct 2025
Citations
32
Code
Not linked
Read summary Original
91

arXiv.org

BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping

This paper introduces BAPO (Balanced Policy Optimization with Adaptive Clipping), a method to stabilize off-policy reinforcement learning for large language models (LLMs). Off-policy RL, where stale data from past policies is used, suffers from unstable optimization, entropy collapse, and gradient explosions. The authors identify two key issues: (1) an…

Zhiheng Xi, Xin Guo, Yang Nan, Enyu Zhou, et al.
Published
Oct 2025
Citations
48
Code
94 stars
Read summary Original
92

arXiv.org

Chem-R: Learning to Reason as a Chemist

Chem-R is a chemical reasoning model designed to emulate the deliberative processes of chemists, addressing three key challenges in current LLMs: lack of chemical fundamentals, unreliable reasoning trajectories, and imbalanced performance across tasks. The model is trained via a three-phase framework: (1) Chemical Foundation Training, which fine-tunes on…

Weida Wang, Benteng Chen, Di Zhang, Wanhao Liu, et al.
Published
Oct 2025
Citations
11
Code
30 stars
Read summary Original
93

arXiv.org

Agentic Entropy-Balanced Policy Optimization

The paper introduces Agentic Entropy-Balanced Policy Optimization (AEPO), an agentic reinforcement learning algorithm for training multi-turn web agents. It addresses two entropy-driven challenges: 'High-Entropy Rollout Collapse' (over-branching on few trajectories due to consecutive high-entropy tool-call steps) and 'High-Entropy Token Gradient Clipping'…

Guanting Dong, Licheng Bao, Zhongyuan Wang, Kangzhi Zhao, et al.
Published
Oct 2025
Citations
26
Code
1.1K stars
Read summary Original
94

arXiv.org

Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization

This paper investigates how attention dynamics in LLMs can reveal reasoning patterns and improve reinforcement learning (RL) credit assignment. The authors analyze attention heads, classifying them as locally or globally focused. Locally focused heads show a sawtooth pattern near the diagonal, indicating phrasal chunks, while globally focused heads…

Yang Li, Zhichen Dong, Yuhan Sun, Weixun Wang, et al.
Published
Oct 2025
Citations
25
Code
Not linked
Read summary Original
95

NVIDIA

QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs

QeRL is a framework that combines NVFP4 4-bit quantization with Low-Rank Adaptation (LoRA) to accelerate and reduce memory usage in reinforcement learning (RL) for large language models (LLMs). The authors find that quantization noise increases policy entropy, which enhances exploration during RL, contrary to its detrimental effect in supervised…

Wei Huang, Yi Ge, Shuai Yang, Yicheng Xiao, et al.
Published
Oct 2025
Citations
16
Code
512 stars
Read summary Original
96

arXiv.org

TaTToo: Tool-Grounded Thinking PRM for Test-Time Scaling in Tabular Reasoning

TATTOO is a novel table-grounded Process Reward Model (PRM) designed to improve test-time scaling (TTS) for large reasoning models (LRMs) on tabular reasoning tasks. The authors identify that existing PRMs fail on table-specific operations like sub-table retrieval and schema interaction, leading to performance bottlenecks. TATTOO addresses this by…

Jiaru Zou, Soumya Roy, Vinay Kumar Verma, Ziyi Wang, et al.
Published
Oct 2025
Citations
10
Code
Not linked
Read summary Original
97

Annual Meeting of the Association for Computational Linguistics

Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward

This paper addresses the exploration collapse in Reinforcement Learning with Verifiable Rewards (RLVR) for large language models, where performance plateaus as policy entropy collapses. The authors identify the systematic elimination of valuable low-probability exploratory tokens, termed 'reasoning sparks' (e.g., 'wait', 'however'), as a key cause, while…

Guanhua Huang, Tingqiang Xu, Mingze Wang, Qi Yi, et al.
Published
Oct 2025
Citations
23
Code
44 stars
Read summary Original
98

arXiv.org

ExGRPO: Learning to Reason from Experience

The paper introduces ExGRPO, a framework for improving reinforcement learning from verifiable rewards (RLVR) for large reasoning models by reusing past rollout experiences. The authors first analyze what makes a reasoning experience valuable, identifying that medium-difficulty questions (with rollout correctness between 25% and 75%) and low-entropy…

Runzhe Zhan, Yafu Li, Zhi Wang, Xiaoye Qu, et al.
Published
Oct 2025
Citations
41
Code
Not linked
Read summary Original
99

arXiv.org

More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models

This paper investigates the dual nature of reasoning in Vision-Language Models (VLMs). The authors find that while reasoning improves logical inference, longer reasoning lengths can impair perceptual grounding, leading to errors on basic visual questions. They attribute this to 'visual forgetting,' where models increasingly ignore visual input during…

Xinyu Tian, Shu Zou, Zhaoyuan Yang, Mengqi He, et al.
Published
Sep 2025
Citations
26
Code
389 stars
Read summary Original
100

Meta AI

TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning

TruthRL is a reinforcement learning framework that directly optimizes the truthfulness of large language models (LLMs) by using a ternary reward scheme that distinguishes correct answers, hallucinations, and abstentions. Implemented with GRPO, it rewards correct answers, penalizes hallucinations, and treats abstentions neutrally, encouraging models to…

Zhepei Wei, Xiao Yang, Kai Sun, Jiaqi Wang, et al.
Published
Sep 2025
Citations
16
Code
Not linked
Read summary Original