Paper 2509.02479
SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning
- Published
- Sep 2025
- Research lab
- Independent
- Citations
- 139
- GitHub
- 403 stars
01 In brief
Summary
SimpleTIR is a plug-and-play algorithm that stabilizes multi-turn Tool-Integrated Reasoning (TIR) training under the Zero RL paradigm.
The authors identify that training instability and gradient explosions stem from distributional drift caused by external tool feedback, which leads to the generation of low-probability tokens that compound over turns.
They propose filtering out trajectories containing 'void turns' (responses with neither a complete code block nor a final answer) to block harmful high-magnitude gradients and correct misaligned credit assignment.
Starting from Qwen2.5-7B base model, SimpleTIR achieves state-of-the-art AIME24 score of 50.5, up from a text-only baseline of 22.1, and also improves scores on other benchmarks (e.g., MATH500 88.4, AMC23 79.1).
Ablations show that alternative filtering criteria (high importance ratio, low token probability) fail to stabilize training, while SimpleTIR maintains stable gradient norms.
The Zero RL approach also encourages diverse reasoning patterns like cross-validation, progressive reasoning, and error correction, as evidenced by higher frequencies compared to cold-start methods like ReTool.
02 From the paper
Abstract
Large Language Models (LLMs) can significantly improve their reasoning capabilities by interacting with external tools, a paradigm known as Tool-Integrated Reasoning (TIR). However, extending TIR to multi-turn scenarios using Reinforcement Learning (RL) is often hindered by training instability and performance collapse. We identify that such instability is primarily caused by a distributional drift from external tool feedback, leading to the generation of low-probability tokens. This issue compounds over successive turns, causing catastrophic gradient norm explosions that derail the training process. To address this challenge, we introduce SimpleTIR , a plug-and-play algorithm that stabilizes multi-turn TIR training. Its core strategy is to identify and filter out trajectories containing void turns, i.e., turns that yield neither a code block nor a final answer. By removing these problematic trajectories from the policy update, SimpleTIR effectively blocks the harmful, high-magnitude gradients, thus stabilizing the learning dynamics. Extensive experiments show that SimpleTIR achieves state-of-the-art performance on challenging math reasoning benchmarks, notably elevating the AIME24 score from a text-only baseline of 22.1 to 50.5 when starting from the Qwen2.5-7B base model. Furthermore, by avoiding the constraints of supervised fine-tuning, SimpleTIR encourages the model to discover diverse and sophisticated reasoning patterns, such as self-correction and cross-validation.