The Year in AI Papers
The year/Independent research

Paper 2508.03501

Training Long-Context, Multi-Turn Software Engineering Agents with Reinforcement Learning

Alexander Golubev, Maria Trofimova, Sergei Polezhaev, Ibragim Badertdinov, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Sergey Abramov, Andrei Andriushchenko, Filipp Fisin, Sergei Skvortsov, Boris Yangel

Published
Aug 2025
Research lab
Independent
Citations
27
GitHub
Not linked

01 In brief

Summary

The paper presents a two-phase training pipeline for software engineering (SWE) agents using reinforcement learning (RL).

Starting from Qwen2.5-72B-Instruct, the authors first apply rejection fine-tuning (RFT) on successful trajectories from SWE-rebench tasks, improving Pass@1 on SWE-bench Verified from 11% to 20%.

Then, they apply a synchronous RL pipeline using DAPO, with two stages: Stage 1 at 65k context length and Stage 2 at 131k context with more turns.

This boosts Pass@1 to 39.0% on SWE-bench Verified, and 35.0% and 31.7% on SWE-rebench May and June splits, respectively, competitive with larger models like DeepSeek-V3-0324.

Key findings include the importance of not filtering long trajectories (to avoid biasing against looping behavior) and the risk of decoding parameter changes causing distribution mismatch and training instability.

The work demonstrates that RL can train capable multi-turn interactive agents from open-weight models without teacher distillation.

02 From the paper

Abstract

Research on applications of reinforcement learning (RL) to large language models has mostly been focused on single-turn problems, such as mathematical reasoning or single-shot code generation. While these problems can be viewed as token-level multi-turn Markov decision processes (MDPs), this view corresponds to a degenerate case of multi-turn interaction where the environment provides no feedback. This contrasts with many real-world domains, such as software engineering (SWE), which require rich multi-turn interactions with a stateful environment that responds to each action with a non-trivial observation. To bridge this gap, we demonstrate the successful application of RL to this general regime. Our methodology begins with rejection fine-tuning (RFT) using execution feedback to train a policy to follow instructions and formatting effectively, followed by a synchronous RL pipeline using DAPO for iterative improvement. Applying this pipeline to Qwen2.5-72B-Instruct, we increase its Pass@1 on the SWE-bench Verified benchmark from 11% to 39%, substantially improving upon the 20% RFT baseline. On the May and June splits of SWE-rebench, the resulting agent achieves Pass@1 of 35% and 31% respectively, competitive with even larger models such as DeepSeek-V3-0324 or Qwen3-235B-A22B, demonstrating that our methodology offers a practical approach for training capable agents for multi-turn interactive tasks using open-weight models.