Paper 2606.17200
ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining
- Published
- Jun 2026
- Research lab
- Independent
- Citations
- 1
- GitHub
- 37 stars
01 In brief
Summary
ACE-EGO-0 is a unified Vision-Language-Action (VLA) pretraining framework that jointly trains on heterogeneous embodied data, including egocentric human videos, multi-embodiment robot demonstrations, and simulation rollouts.
It addresses representation heterogeneity via a unified action representation: camera-space actions (spatial alignment), cross-embodiment morphology conditioning via URDF-encoded robot tokens and learned human surrogate embeddings (structural alignment), and time-aligned action chunking based on physical duration (temporal alignment).
To handle supervision-quality mismatch, it uses a reliability-aware training objective where robot data supervises the primary flow-matching loss, while noisy human pseudo-actions contribute through an auxiliary loss weighted by channel-level priors and step-level smoothness factors.
A five-stage pipeline converts raw egocentric videos into 1,478 hours of pseudo-action-labeled data, combined with 4,534.8+ hours of robot and simulation data.
ACE-EGO-0 achieves state-of-the-art results: 72.8% average success on RoboCasa GR1 TableTop, 91.12%/90.62% on RoboTwin 2.0 Easy/Hard, and 78.3% on real-world bimanual tasks, outperforming baselines.
Ablations confirm each component contributes, and human data improves fine-tuning in data-scarce regimes.
02 From the paper
Abstract
Vision-Language-Action (VLA) models benefit from large-scale and diverse embodied data, yet scaling robot trajectory collection is costly and labor-intensive. Recent advances show that large-scale egocentric human videos provide complementary real-world supervision in pretraining. However, joint training on human and robot data remains challenging due to divergences in action spaces, embodiment structures, temporal dynamics, and supervision quality. We introduce ACE-EGO-0, a unified VLA pretraining framework jointly leveraging heterogeneous data sources. To extract large-scale pretraining supervision from egocentric human videos, we build a scalable egocentric video-to-action pipeline that converts raw human videos into robot-format pseudo-action trajectories. To make these labels comparable with robot demonstrations, ACE-EGO-0 uses a unified action representation based on camera-space actions, morphology conditioning, and time-aligned action chunking. To robustly leverage noisy pseudo-action supervision from egocentric human videos, we formulate a reliability-aware training objective with a human auxiliary loss that concentrates supervision on reliable signals. We instantiate ACE-EGO-0 on 4.53K hours of robot and simulation data, together with 1.48K hours of pseudo-action-labeled egocentric human data. Experiments show that incorporating large-scale human supervision under reliability-aware weighting consistently improves both unified joint pretraining and supervised fine-tuning. ACE-EGO-0 achieves state-of-the-art performance on RoboCasa GR1 TableTop and RoboTwin 2.0, while demonstrating strong transfer to real-world bimanual manipulation.