Paper 2508.20722
rStar2-Agent: Agentic Reasoning Technical Report
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 45
- GitHub
- 1.4K stars
01 In brief
Summary
rStar2-Agent is a 14B math reasoning model trained with agentic reinforcement learning (RL) to achieve frontier-level performance, rivaling or surpassing DeepSeek-R1 (671B).
It uses Python coding tools within a dedicated environment, enabling advanced cognitive behaviors like careful tool use and reflection on execution feedback.
Key innovations include an efficient RL infrastructure on 64 MI300X GPUs handling 45K concurrent tool calls with 0.3s latency, GRPO-RoC (Resample-on-Correct) to filter noisy trajectories, and a training recipe starting with non-reasoning SFT followed by multi-stage RL (8K→12K lengths).
The model achieves 80.6% on AIME24 and 69.8% on AIME25 in just 510 RL steps, with shorter responses than baselines.
It also generalizes to science reasoning (GPQA-Diamond) and agentic tool use (BFCL v3).
Analysis reveals high-entropy tokens for exploration and reflection on tool responses, driving smarter reasoning.
The code and recipes are open-sourced at https://github.com/microsoft/rStar.
The model demonstrates that efficient agentic RL can unlock advanced reasoning capabilities with minimal compute, outperforming larger models and pure CoT approaches.
Future work will extend to broader domains and tools.
The approach highlights the value of tool-augmented reasoning and environment feedback in RL training, offering a practical path for efficient reasoning model development.
The model's success underscores the importance of high-quality positive trajectory selection and infrastructure scalability in agentic RL.
The findings suggest that agentic RL can effectively complement internal self-reflection, leading to more robust and efficient problem-solving.
The training recipe's efficiency (one week) makes it accessible for further research and application.
The model's generalization beyond math indicates transferable reasoning skills…
02 From the paper
Abstract
We introduce rStar2-Agent, a 14B math reasoning model trained with agentic reinforcement learning to achieve frontier-level performance. Beyond current long CoT, the model demonstrates advanced cognitive behaviors, such as thinking carefully before using Python coding tools and reflecting on code execution feedback to autonomously explore, verify, and refine intermediate steps in complex problem-solving. This capability is enabled through three key innovations that makes agentic RL effective at scale: (i) an efficient RL infrastructure with a reliable Python code environment that supports high-throughput execution and mitigates the high rollout costs, enabling training on limited GPU resources (64 MI300X GPUs); (ii) GRPO-RoC, an agentic RL algorithm with a Resample-on-Correct rollout strategy that addresses the inherent environment noises from coding tools, allowing the model to reason more effectively in a code environment; (iii) An efficient agent training recipe that starts with non-reasoning SFT and progresses through multi-RL stages, yielding advanced cognitive abilities with minimal compute cost. To this end, rStar2-Agent boosts a pre-trained 14B model to state of the art in only 510 RL steps within one week, achieving average pass@1 scores of 80.6% on AIME24 and 69.8% on AIME25, surpassing DeepSeek-R1 (671B) with significantly shorter responses. Beyond mathematics, rStar2-Agent-14B also demonstrates strong generalization to alignment, scientific reasoning, and agentic tool-use tasks. Code and training recipes are available at https://github.com/microsoft/rStar.