Paper 2509.13310
Scaling Agents via Continual Pre-training
01 In brief
Summary
The paper introduces Agentic Continual Pre-training (Agentic CPT), a new training stage between pre-training and post-training, to build agentic foundation models for deep research agents.
The authors argue that post-training on general-purpose models creates optimization conflicts, as models must simultaneously learn agentic behaviors and align to expert demonstrations.
They propose AgentFounder, a 30B model built from Qwen3-30B-A3B-Base, using two data synthesis methods: First-order Action Synthesis (FAS) which generates planning and reasoning data from knowledge-to-question transformations without external tool calls, and Higher-order Action Synthesis (HAS) which expands trajectories into multi-step decision-making problems.
A two-stage training strategy uses 200B tokens with 32K context, then 100B tokens with 128K context.
AgentFounder-30B achieves state-of-the-art results on 10 benchmarks, including 39.9% on BrowseComp-en, 43.3% on BrowseComp-zh, 72.8% on GAIA, 31.5% on HLE, and 73.0% on xbench-DeepSearch, outperforming open-source and some commercial agents.
Ablations show the two-stage strategy and both data types contribute gains, and scaling laws hold for data volume and model size.
The model also retains general tool-use ability, scoring 70.0 on ACEBench, surpassing the base model's 67.2.
02 From the paper
Abstract
Large language models (LLMs) have evolved into agentic systems capable of autonomous tool use and multi-step reasoning for complex problem-solving. However, post-training approaches building upon general-purpose foundation models consistently underperform in agentic tasks, particularly in open-source implementations. We identify the root cause: the absence of robust agentic foundation models forces models during post-training to simultaneously learn diverse agentic behaviors while aligning them to expert demonstrations, thereby creating fundamental optimization tensions. To this end, we are the first to propose incorporating Agentic Continual Pre-training (Agentic CPT) into the deep research agents training pipeline to build powerful agentic foundational models. Based on this approach, we develop a deep research agent model named AgentFounder. We evaluate our AgentFounder-30B on 10 benchmarks and achieve state-of-the-art performance while retains strong tool-use ability, notably 39.9% on BrowseComp-en, 43.3% on BrowseComp-zh, and 31.5% Pass@1 on HLE.