The Year in AI Papers
The year/NVIDIA

Paper 2605.28774

Agent Explorative Policy Optimization for Multimodal Agentic Reasoning

Minki Kang, Shizhe Diao, Ryo Hachiuma, Sung Ju Hwang, Pavlo Molchanov, Yu-Chiang Frank Wang, Byung-Kwan Lee

Published
May 2026
Research lab
NVIDIA
Citations
2
GitHub
Not linked

01 In brief

Summary

The paper introduces AXPO (Agent eXplorative Policy Optimization) to address the Thinking-Acting Gap in multimodal agentic reasoning, where tool use is under-trained compared to thinking.

Under GRPO, tool use occurs in only ~30% of rollouts, and tool-using subgroups are all-wrong on ~40% of questions, suppressing learning signals.

AXPO fixes the thinking prefix of failed tool-using rollouts and resamples the tool call and continuation, using uncertainty-based prefix selection.

Experiments on nine multimodal benchmarks with Qwen3-VL-Thinking (2B/4B/8B) show SFT+AXPO outperforms SFT+GRPO by +1.8 pp Pass@1 and +1.8 pp Pass@4 on average at 8B, and the 8B model surpasses the 32B Base on Pass@4 with 4x fewer parameters.

Ablations confirm all components matter, and AXPO outperforms alternative RL recipes and reward shaping.

Training dynamics show AXPO increases tool-use rate and reduces all-wrong rates, improving tool interaction quality across all tool families.

02 From the paper

Abstract

Vision-language models with extended reasoning succeed on complex problems, but many real-world problems require external tools that internal reasoning alone often cannot resolve. Agentic reasoning therefore interleaves two behaviors with a structural asymmetry: thinking (the self-contained default) and tool use (a high-variance auxiliary acting). We refer to this asymmetry as the Thinking-Acting Gap. Under standard RL recipes like GRPO, the gap manifests as two diagnostic symptoms during training: tool use is attempted on only ~30% of rollouts, and when attempted, the tool-using rollouts within a group are all-wrong on ~40% of questions, suppressing the learning signal at the tool calls that needed it. We propose AXPO (Agent eXplorative Policy Optimization): for each all-wrong tool-using subgroup, AXPO fixes the thinking prefix and resamples the tool call and its continuation, paired with uncertainty-based prefix selection. Across nine multimodal benchmarks and three scales of Qwen3-VL-Thinking, SFT+AXPO outperforms SFT+GRPO at average (+1.8pp Pass@1 and +1.8pp Pass@4 at 8B on average) and 8B with SFT+AXPO surpasses the 32B Base on Pass@4 with 4 times fewer parameters.