Paper 2510.00406
VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 50
- GitHub
- 163 stars
01 In brief
Summary
VLA-RFT is a reinforcement fine-tuning framework for Vision-Language-Action (VLA) models that uses a data-driven world model as a simulator to provide verified rewards, avoiding costly real-world interactions and sim-to-real gaps.
The world model, pretrained on offline data, predicts future visual observations conditioned on actions, enabling policy rollouts and dense rewards based on trajectory similarity.
The VLA policy, using a flow-matching action head, is optimized via GRPO with an SDE-based stochastic formulation for exploration.
Experiments on LIBERO show that with only 400 fine-tuning steps, VLA-RFT improves average success rate from 86.6% to 91.1%, outperforming supervised baselines and matching online RL methods with far fewer steps.
It also demonstrates robustness under perturbations (e.g., object/goal position shifts, robot state changes), with gains up to +6.7% in major goal perturbations.
Ablations show that trajectory-level rewards within the world model (Reward Type 3) are crucial, outperforming action-level or direct image comparison rewards.
Limitations include reliance on expert demonstrations, world model capacity, and lack of explicit planning integration.
02 From the paper
Abstract
Vision-Language-Action (VLA) models enable embodied decision-making but rely heavily on imitation learning, leading to compounding errors and poor robustness under distribution shift. Reinforcement learning (RL) can mitigate these issues yet typically demands costly real-world interactions or suffers from sim-to-real gaps. We introduce VLA-RFT, a reinforcement fine-tuning framework that leverages a data-driven world model as a controllable simulator. Trained from real interaction data, the simulator predicts future visual observations conditioned on actions, allowing policy rollouts with dense, trajectory-level rewards derived from goal-achieving references. This design delivers an efficient and action-aligned learning signal, drastically lowering sample requirements. With fewer than 400 fine-tuning steps, VLA-RFT surpasses strong supervised baselines and achieves greater efficiency than simulator-based RL. Moreover, it exhibits strong robustness under perturbed conditions, sustaining stable task execution. Our results establish world-model-based RFT as a practical post-training paradigm to enhance the generalization and robustness of VLA models. For more details, please refer to https://vla-rft.github.io/.