Paper 2510.05592
In-the-Flow Agentic System Optimization for Effective Planning and Tool Use
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 51
- GitHub
- 2K stars
01 In brief
Summary
The paper introduces AGENTFLOW, a trainable agentic framework for planning and tool use that coordinates four specialized modules—planner, executor, verifier, and generator—through an evolving memory.
Unlike monolithic tool-integrated reasoning models, AGENTFLOW optimizes its planner on-policy within the multi-turn loop.
To address long-horizon credit assignment with sparse rewards, the authors propose Flow-based Group Refined Policy Optimization (Flow-GRPO), which broadcasts a single verifiable trajectory-level outcome to every turn, converting multi-turn RL into tractable single-turn updates with group-normalized advantages.
Experiments across ten benchmarks show AGENTFLOW with a 7B backbone outperforms top baselines, with average accuracy gains of 14.9% on search, 14.0% on agentic, 14.5% on mathematical, and 4.1% on scientific tasks, even surpassing GPT-4o.
Ablations show in-the-flow RL is crucial, as offline SFT causes performance collapse.
Analyses confirm improved planning, tool-calling reliability, and positive scaling with model size and turn budgets.
02 From the paper
Abstract
Outcome-driven reinforcement learning has advanced reasoning in large language models (LLMs), but prevailing tool-augmented approaches train a single, monolithic policy that interleaves thoughts and tool calls under full context; this scales poorly with long horizons and diverse tools and generalizes weakly to new scenarios. Agentic systems offer a promising alternative by decomposing work across specialized modules, yet most remain training-free or rely on offline training decoupled from the live dynamics of multi-turn interaction. We introduce AgentFlow, a trainable, in-the-flow agentic framework that coordinates four modules (planner, executor, verifier, generator) through an evolving memory and directly optimizes its planner inside the multi-turn loop. To train on-policy in live environments, we propose Flow-based Group Refined Policy Optimization (Flow-GRPO), which tackles long-horizon, sparse-reward credit assignment by converting multi-turn optimization into a sequence of tractable single-turn policy updates. It broadcasts a single, verifiable trajectory-level outcome to every turn to align local planner decisions with global success and stabilizes learning with group-normalized advantages. Across ten benchmarks, AgentFlow with a 7B-scale backbone outperforms top-performing baselines with average accuracy gains of 14.9% on search, 14.0% on agentic, 14.5% on mathematical, and 4.1% on scientific tasks, even surpassing larger proprietary models like GPT-4o. Further analyses confirm the benefits of in-the-flow optimization, showing improved planning, enhanced tool-calling reliability, and positive scaling with model size and reasoning turns.