Paper 2605.06747
HumanNet: Scaling Human-centric Video Learning to One Million Hours
- Published
- May 2026
- Research lab
- Independent
- Citations
- 6
- GitHub
- 281 stars
01 In brief
Summary
HumanNet is a one-million-hour human-centric video corpus designed to scale embodied learning by capturing how humans interact with the physical world.
It includes both first-person and third-person perspectives, covering fine-grained activities, human-object interactions, tool use, and long-horizon behaviors across diverse environments.
The dataset provides rich annotations such as captions, motion descriptions, and hand/body signals, and is organized by a multi-axis taxonomy.
A systematic curation pipeline handles data collection, processing, and annotation, including filtering, viewpoint characterization, and privacy review.
In a controlled vision-language-action post-training study, initializing from 1,000 hours of egocentric video from HumanNet matched or slightly surpassed 100 hours of real-robot data from Magic Cobot, and substantially closed the gap to a 20,000-hour real-robot baseline, suggesting that egocentric human video is a scalable and cost-effective substitute for robot data.
The dataset supports applications in video and VLM pretraining, world-action models, motion-aware representation learning, and human-to-robot transfer.
Limitations include the embodiment gap, noise from scale, uneven coverage, and privacy concerns.
02 From the paper
Abstract
Progress in embodied intelligence increasingly depends on scalable data infrastructure. While vision and language have scaled with internet corpora, learning physical interaction remains constrained by the lack of large, diverse, and richly annotated human activity data. We present HumanNet, a one-million-hour human-centric video corpus that captures how humans interact with the physical world at scale. HumanNet spans both first-person and third-person perspectives and covers fine-grained activities, human-object interactions, tool use, and long-horizon behaviors across diverse real-world environments. Beyond raw video, the dataset provides interaction-centric annotations, including captions, motion descriptions, and hand and body-related signals, enabling motion-aware and interaction-aware learning. Beyond scale, HumanNet introduces a systematic data curation paradigm for embodied learning, where human-centric filtering, temporal structuring, viewpoint diversity, and annotation enrichment are treated as first-class design principles. This design transforms unstructured internet video into a scalable substrate for representation learning, activity understanding, motion generation, and human-to-robot transfer. We conduct a first-step validation on the value of this design through controlled vision-language-action ablation: under a fixed set of validation data, continued training from the Qwen VLM model with 1000 hours of egocentric video drawn from HumanNet surpasses the continued training with 100 hours of real-robot data from Magic Cobot, indicating that egocentric human video could be a scalable and cost-effective substitute for robot data. By building this project, we aim to explore the opportunity to scale embodied foundation models using human-centric videos, rather than relying solely on robot-specific data.