Paper 2510.05684
D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 0
- GitHub
- 90 stars
01 In brief
Summary
The paper introduces D2E, a framework that uses desktop interactions (screen, keyboard, mouse) as a pretraining substrate for embodied AI.
It comprises three components: the OWA Toolkit for scalable data collection and compression (152x), the Generalist-IDM for pseudo-labeling YouTube videos via timestamp-based next-event prediction (NEP-tau), and VAPT for transferring desktop-pretrained representations to robotics.
Using 1.3K+ hours of data (259 human, 1K+ pseudo-labeled), a 1B-parameter model achieves 96.6% success on LIBERO manipulation and 83.3% on CANVAS navigation, matching or surpassing larger models like pi0 (3.3B) and OpenVLA (7B).
The Generalist-IDM shows zero-shot generalization to unseen games, and pseudo-labels improve navigation but not manipulation.
The framework reduces data collection costs and storage, making large-scale pretraining accessible.
All resources are public at https://worv-ai.github.io/d2e/.
02 From the paper
Abstract
Large language models leverage internet-scale text data, yet embodied AI remains constrained by the prohibitive costs of physical trajectory collection. Desktop environments -- particularly gaming -- offer a compelling alternative: they provide rich sensorimotor interactions at scale while maintaining the structured observation-action coupling essential for embodied learning. We present D2E (Desktop to Embodied AI), a framework that demonstrates desktop interactions can serve as an effective pretraining substrate for robotics embodied AI tasks. Unlike prior work that remained domain-specific (e.g., VPT for Minecraft) or kept data proprietary (e.g., SIMA), D2E establishes a complete pipeline from scalable desktop data collection to verified transfer in embodied domains. Our framework comprises three components: (1) the OWA Toolkit that unifies diverse desktop interactions into a standardized format with 152x compression, (2) the Generalist-IDM that achieves strong zero-shot generalization across unseen games through timestamp-based event prediction, enabling internet-scale pseudo-labeling, and (3) VAPT that transfers desktop-pretrained representations to physical manipulation and navigation. Using 1.3K+ hours of data (259 hours of human demonstrations and 1K+ hours of pseudo-labeled gameplay), our 1B-parameter model achieves 96.6% success on LIBERO manipulation and 83.3% on CANVAS navigation, matching or surpassing models up to 7x larger, such as π_{0} (3.3B) and OpenVLA (7B). These results demonstrate that sensorimotor primitives learned from digital interactions transfer effectively to real-world physical tasks, establishing desktop pretraining as a practical paradigm for embodied AI. All resources are publicly available at https://worv-ai.github.io/d2e.