The Year in AI Papers
The year/Independent research

Paper 2509.08827

A Survey of Reinforcement Learning for Large Reasoning Models

Kaiyan Zhang, Yuxin Zuo, Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jia, Pengfei Li, Yu Fu, Xingtai Lv, et al.

Published
Sep 2025
Research lab
Independent
Citations
161
GitHub
2.5K stars

01 In brief

Summary

This survey reviews recent advances in Reinforcement Learning (RL) for Large Reasoning Models (LRMs), focusing on how RL transforms LLMs into LRMs, particularly since DeepSeek-R1.

It covers foundational components: reward design (verifiable, generative, dense, unsupervised, and shaping), policy optimization (critic-based, critic-free, off-policy, and regularization), and sampling strategies.

It discusses open problems such as RL's role (sharpening vs.

discovery), RL vs.

SFT (generalization vs.

memorization), model priors, training recipes, and reward types.

Training resources include static corpora, dynamic environments, and RL infrastructure.

Applications span coding, agentic tasks, multimodal tasks, multi-agent systems, robotics, and medical tasks.

Future directions include continual RL, memory-based RL, model-based RL, efficient reasoning, latent space reasoning, RL for pre-training, diffusion-based LLMs, scientific discovery, and architecture-algorithm co-design.

The survey aims to identify strategies for scaling RL toward Artificial SuperIntelligence (ASI).

02 From the paper

Abstract

In this paper, we survey recent advances in Reinforcement Learning (RL) for reasoning with Large Language Models (LLMs). RL has achieved remarkable success in advancing the frontier of LLM capabilities, particularly in addressing complex logical tasks such as mathematics and coding. As a result, RL has emerged as a foundational methodology for transforming LLMs into LRMs. With the rapid progress of the field, further scaling of RL for LRMs now faces foundational challenges not only in computational resources but also in algorithm design, training data, and infrastructure. To this end, it is timely to revisit the development of this domain, reassess its trajectory, and explore strategies to enhance the scalability of RL toward Artificial SuperIntelligence (ASI). In particular, we examine research applying RL to LLMs and LRMs for reasoning abilities, especially since the release of DeepSeek-R1, including foundational components, core problems, training resources, and downstream applications, to identify future opportunities and directions for this rapidly evolving area. We hope this review will promote future research on RL for broader reasoning models. Github: https://github.com/TsinghuaC3I/Awesome-RL-for-LRMs