Paper 2510.18135
World-in-World: World Models in a Closed-Loop World
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 35
- GitHub
- 181 stars
01 In brief
Summary
The paper introduces World-in-World, the first open benchmark for evaluating generative world models (WMs) in a closed-loop embodied setting, moving beyond visual quality to task success.
It provides a unified online planning strategy and a standardized action API to integrate diverse WMs into four embodied tasks: Active Recognition, Image-Goal Navigation, Active Embodied Question Answering, and Robotic Manipulation.
The benchmark evaluates leading video generators (e.g., Wan, SVD, Cosmos) and task-focused models, with post-training on action-observation data.
Key findings reveal three surprises: (1) visual quality alone does not guarantee task success—controllability matters more; (2) scaling post-training with action-observation data is more effective than upgrading pretrained video generators; and (3) increasing inference-time compute via online planning substantially improves closed-loop performance.
The study also presents a data scaling law for world models in embodied settings, showing that more post-training data consistently improves performance.
The authors conclude that world models should be judged by their closed-loop success, not just visual realism, and highlight challenges in generalization, long-horizon planning, and precise dynamics modeling.
02 From the paper
Abstract
Generative world models (WMs) can now simulate worlds with striking visual realism, which naturally raises the question of whether they can endow embodied agents with predictive perception for decision making. Progress on this question has been limited by fragmented evaluation: most existing benchmarks adopt open-loop protocols that emphasize visual quality in isolation, leaving the core issue of embodied utility unresolved, i.e., do WMs actually help agents succeed at embodied tasks? To address this gap, we introduce World-in-World, the first open platform that benchmarks WMs in a closed-loop world that mirrors real agent-environment interactions. World-in-World provides a unified online planning strategy and a standardized action API, enabling heterogeneous WMs for decision making. We curate four closed-loop environments that rigorously evaluate diverse WMs, prioritize task success as the primary metric, and move beyond the common focus on visual quality; we also present the first data scaling law for world models in embodied settings. Our study uncovers three surprises: (1) visual quality alone does not guarantee task success, controllability matters more; (2) scaling post-training with action-observation data is more effective than upgrading the pretrained video generators; and (3) allocating more inference-time compute allows WMs to substantially improve closed-loop performance.