The Year in AI Papers
The year/Independent research

Paper 2510.25889

$π_\texttt{RL}$: Online RL Fine-tuning for Flow-based Vision-Language-Action Models

Kang Chen, Zhihao Liu, Tonghe Zhang, Zhen Guo, Si Xu, Hao Lin, Hongzhi Zang, Xiang Li, Quanlu Zhang, Zhaofei Yu, Guoliang Fan, Tiejun Huang, et al.

Published
Oct 2025
Research lab
Independent
Citations
3
GitHub
Not linked

01 In brief

Summary

The paper introduces π_RL, a framework for online reinforcement learning (RL) fine-tuning of flow-based Vision-Language-Action (VLA) models such as π0 and π0.5.

It addresses the challenge of intractable action log-likelihoods in flow matching by proposing two methods: Flow-Noise, which models the denoising process as a discrete-time MDP with a learnable noise network for exact log-likelihood computation, and Flow-SDE, which converts the ODE denoising into an SDE and formulates a two-layer MDP for efficient exploration.

Both methods are optimized with PPO.

Experiments on LIBERO, ManiSkill, MetaWorld, and CALVIN show significant performance gains over SFT baselines, with π0 achieving up to +29.2% and π0.5 up to +31.0% average success rate improvement.

Few-shot SFT with RL outperforms full-dataset SFT baselines (e.g., 98.3% vs 96.9% on LIBERO).

OOD evaluations show gains transfer to visual and execution variations but not to novel task objectives.

Ablations cover critic design, MDP formulation, noise injection, and hyperparameters.

The framework also enables zero-shot sim-to-real transfer using a 3D Gaussian Splatting simulator.

02 From the paper

Abstract

Vision-Language-Action (VLA) models enable robots to understand and perform complex tasks from multimodal input. Although recent work explores using reinforcement learning (RL) to automate the laborious data collection process in scaling supervised fine-tuning (SFT), applying RL to large-scale flow-based VLAs (\eg, $π_0$, $π_{0.5}$) remains challenging due to intractable action log-likelihoods raised from flow matching. We address this challenge with $π_{\texttt{RL}}$, featuring two technical approaches: (1) \textbf{Flow-Noise} models the denoising process as a discrete-time MDP with a learnable noise network for exact log-likelihood computation. (2) \textbf{Flow-SDE} integrates denoising with agent-environment interaction, formulating a two-layer MDP that employs ODE-to-SDE conversion for efficient RL exploration. We evaluate $π_{\texttt{RL}}$ across various benchmarks, with experiments demonstrating that RL yields significant performance improvements in both in-distribution and out-of-distribution settings.