The Year in AI Papers
The year/Independent research

Paper 2601.21558

ASTRA: Automated Synthesis of agentic Trajectories and Reinforcement Arenas

Xiaoyu Tian, Haotian Wang, Shuaiting Chen, Hao Zhou, Kaichi Yu, Yudian Zhang, Jade Ouyang, Junxi Yin, Jiong Chen, Baoyan Guo, Lei Zhang, Junjie Tao, et al.

Published
Jan 2026
Research lab
Independent
Citations
3
GitHub
151 stars

01 In brief

Summary

ASTRA is a fully automated, end-to-end framework for training tool-augmented language model agents.

It combines a trajectory synthesis pipeline that uses the static topology of tool-call graphs to generate diverse, multi-turn tool-use trajectories for supervised fine-tuning (SFT), with an environment synthesis framework that converts decomposed question-answer traces into code-executable, rule-verifiable Python environments for online reinforcement learning (RL).

The training methodology integrates SFT with online, multi-turn RL using an F1-style trajectory-level reward that balances task completion and interaction efficiency, and includes irrelevant-tool mixing to improve tool discrimination.

Experiments on BFCL-MT, τ²-Bench, and ACEBench show that ASTRA-trained models achieve state-of-the-art performance at comparable scales, approaching closed-source systems while preserving core reasoning ability on AIME benchmarks.

The framework is open-sourced.

02 From the paper

Abstract

Large language models (LLMs) are increasingly used as tool-augmented agents for multi-step decision making, yet training robust tool-using agents remains challenging. Existing methods still require manual intervention, depend on non-verifiable simulated environments, rely exclusively on either supervised fine-tuning (SFT) or reinforcement learning (RL), and struggle with stable long-horizon, multi-turn learning. To address these challenges, we introduce ASTRA, a fully automated end-to-end framework for training tool-augmented language model agents via scalable data synthesis and verifiable reinforcement learning. ASTRA integrates two complementary components. First, a pipeline that leverages the static topology of tool-call graphs synthesizes diverse, structurally grounded trajectories, instilling broad and transferable tool-use competence. Second, an environment synthesis framework that captures the rich, compositional topology of human semantic reasoning converts decomposed question-answer traces into independent, code-executable, and rule-verifiable environments, enabling deterministic multi-turn RL. Based on this method, we develop a unified training methodology that integrates SFT with online RL using trajectory-level rewards to balance task completion and interaction efficiency. Experiments on multiple agentic tool-use benchmarks demonstrate that ASTRA-trained models achieve state-of-the-art performance at comparable scales, approaching closed-source systems while preserving core reasoning ability. We release the full pipelines, environments, and trained models at https://github.com/LianjiaTech/astra.