Paper 2607.06558
RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation
- Published
- Jul 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- Not linked
01 In brief
Summary
RynnWorld-Teleop introduces digital teleoperation, a paradigm that replaces physical robots with a generative world model for scalable robot data collection.
An operator's hand-pose stream drives a robot-centric world model to synthesize high-fidelity egocentric videos from a single reference image, with the pose stream serving as an embodiment-agnostic action label transferable to any robot via retargeting.
The system integrates depth-aware skeletal conditioning, progressive human-to-robot training on a video Diffusion Transformer, and streaming autoregressive distillation, enabling 40+ FPS real-time generation on a single H100 GPU.
Policies trained exclusively on RynnWorld-Teleop-generated data achieve zero-shot Sim2Real transfer across dexterous bimanual tasks, and augmenting real datasets with generated data consistently improves success rates.
The model outperforms baselines in visual quality metrics (e.g., FVD 550 vs.
1223 for SFT baseline) and supports out-of-distribution generalization to unseen objects and backgrounds.
Ablations confirm the importance of human pretraining, additive conditioning, and sequential distillation.
Limitations include struggles with fine-grained liquid dynamics and the need for per-platform fine-tuning.
02 From the paper
Abstract
Scaling robot learning requires massive, diverse trajectory data, yet collection is currently bottlenecked by physical teleoperation, where every demonstration binds operator time to specific hardware and workspaces. We introduce digital teleoperation, a paradigm that decouples data collection from physical constraints by replacing the real robot with a generative world model. In this framework, an operator's hand-pose stream drives a robot-centric generative world model to synthesize high-fidelity egocentric videos from a single reference image. The recorded pose stream serves as an embodiment-agnostic action label transferable to any target robot via standard retargeting, yielding complete state-action trajectories for imitation learning independent of physical hardware. We instantiate this paradigm in RynnWorld-Teleop, a system that integrates depth-aware skeletal conditioning, progressive human-to-robot training on a video Diffusion Transformer, and streaming autoregressive distillation. This pipeline compresses the generative process into a single-pass inference, enabling 40+ FPS, real-time interactive generation on a single H100 GPU. Policies trained exclusively on RynnWorld-Teleop-generated data achieve effective zero-shot Sim2Real transfer across dexterous and diverse bimanual tasks. Moreover, augmenting real-world datasets with our digitally teleoperated data consistently improves success rates, demonstrating that RynnWorld-Teleop serves as a high-fidelity, scalable data engine for the next generation of robotic agents.