Paper 2509.09674
SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning
- Published
- Sep 2025
- Research lab
- Independent
- Citations
- 130
- GitHub
- 1.8K stars
01 In brief
Summary
SimpleVLA-RL is an efficient reinforcement learning (RL) framework for Vision-Language-Action (VLA) models, built on veRL, that addresses data scarcity and poor generalization in robotic manipulation.
It uses interactive trajectory sampling, parallel multi-environment rendering, and outcome-based rewards (1 for success, 0 for failure) with GRPO, enhanced by dynamic sampling, higher clipping (0.8 to 1.28), and higher rollout temperature (1.6).
Applied to OpenVLA-OFT, it achieves state-of-the-art results on LIBERO (99.1% average success rate, up from 91.0% with SFT) and RoboTwin1.0 (70.4% vs 39.8%) and RoboTwin2.0 (68.8% vs 38.3%), outperforming π0 and RDT.
With only one demonstration per task, RL boosts LIBERO-Long success from 17.3% to 91.7%, surpassing full-trajectory SFT.
RL also improves generalization across spatial, object, and goal tasks, and sim-to-real transfer (real-world success from 17.5% to 38.5%).
A novel 'pushcut' phenomenon emerges where the policy discovers pushing as a shortcut instead of grasp-move-place.
RL fails when initial model capability is zero, and effectiveness depends on a performance threshold.
02 From the paper
Abstract
Vision-Language-Action (VLA) models have recently emerged as a powerful paradigm for robotic manipulation. Despite substantial progress enabled by large-scale pretraining and supervised fine-tuning (SFT), these models face two fundamental challenges: (i) the scarcity and high cost of large-scale human-operated robotic trajectories required for SFT scaling, and (ii) limited generalization to tasks involving distribution shift. Recent breakthroughs in Large Reasoning Models (LRMs) demonstrate that reinforcement learning (RL) can dramatically enhance step-by-step reasoning capabilities, raising a natural question: Can RL similarly improve the long-horizon step-by-step action planning of VLA? In this work, we introduce SimpleVLA-RL, an efficient RL framework tailored for VLA models. Building upon veRL, we introduce VLA-specific trajectory sampling, scalable parallelization, multi-environment rendering, and optimized loss computation. When applied to OpenVLA-OFT, SimpleVLA-RL achieves SoTA performance on LIBERO and even outperforms $π_0$ on RoboTwin 1.0\&2.0 with the exploration-enhancing strategies we introduce. SimpleVLA-RL not only reduces dependence on large-scale data and enables robust generalization, but also remarkably surpasses SFT in real-world tasks. Moreover, we identify a novel phenomenon ``pushcut'' during RL training, wherein the policy discovers previously unseen patterns beyond those seen in the previous training process. Github: https://github.com/PRIME-RL/SimpleVLA-RL