Paper 2607.14183
Open-AoE: An Open Egocentric Manipulation Dataset and Toolchain for Embodied Learning
- Published
- Jul 2026
- Research lab
- Independent
- Citations
- 1
- GitHub
- 135 stars
01 In brief
Summary
Open-AoE is an open, community-oriented egocentric manipulation dataset and toolchain from Ant Group, covering the full pipeline from smartphone capture to model training.
Its first release includes approximately 2,000 hours of manipulation video collected in natural environments by 500+ contributors using 400+ smartphone models, spanning 400+ scenes and 8,000+ tasks.
The dataset provides text annotations, MANO-based hand poses, camera trajectories, and temporally localized atomic actions.
The data processing pipeline includes edge-side online detection, offline quality checking and scene labeling, reconstruction and annotation (camera trajectory, hand reconstruction, atomic action labeling), and quality inspection.
The downstream toolchain supports visualization, cross-embodiment retargeting, model-specific data conversion, and training recipes for VLA policies, WAMs, and World Models.
Analysis shows Open-AoE has the highest visual diversity among compared datasets (OpenEgo, EgoDex, EgoXtreme) across six CLIP-based metrics, near-complete temporal annotation coverage (99.99%), high image-annotation consistency (4.58/5), and high training-window retention (97.8% of theoretical ceiling).
The dataset is released with privacy protections, including consent, local review, and anonymization.
02 From the paper
Abstract
Egocentric videos of human manipulation provide scalable supervision for embodied intelligence, yet existing resources rarely combine low-cost continuous capture, manipulation-level structured annotations, and reusable tools for robot learning. We present Open-AoE, an open, community-oriented egocentric manipulation dataset and toolchain spanning the full pipeline from smartphone capture to model training. Its first release contains approximately 2,000 hours of manipulation video collected in natural environments by 500+ contributors using 400+ smartphones. The dataset provides text annotations, MANO-based hand poses, camera trajectories, and temporally localized atomic actions. Open-AoE further includes a data processing pipeline that transforms raw recordings into structured samples through temporal action segmentation, semantic annotation, hand reconstruction, and camera trajectory reconstruction. Meanwhile, we provide a separate downstream toolchain supports visualization, cross-embodiment retargeting, model-specific data conversion, and training recipes for VLA policies, WAMs, and World Models. By integrating scalable capture, structured processing, and downstream adaptation, Open-AoE reduces the barriers to both data contribution and reuse, providing practical open infrastructure for embodied model training, human-to-robot transfer, and world modeling.