Paper 2509.07996
3D and 4D World Modeling: A Survey
- Published
- Sep 2025
- Research lab
- Independent
- Citations
- 65
- GitHub
- 960 stars
01 In brief
Summary
This survey provides the first comprehensive review of 3D and 4D world modeling, addressing the lack of standardized definitions and the fragmented literature that often focuses on 2D generative methods.
It establishes precise definitions and a hierarchical taxonomy categorizing methods into video-based (VideoGen), occupancy-based (OccGen), and LiDAR-based (LiDARGen) approaches.
The survey systematically summarizes datasets and evaluation metrics tailored for 3D/4D settings, covering generation, forecasting, planning, reconstruction, and downstream tasks.
It also discusses practical applications in autonomous driving, robotics, video games/XR, digital twins, and other emerging domains.
Key challenges identified include standardized benchmarking, high-fidelity long-horizon generation, transitioning from pseudo-4D to continuous 4D dynamics, physical fidelity, computational efficiency, and cross-modal coherence.
The survey aims to provide a coherent foundation and roadmap for advancing robust, interpretable, and generalizable world models for embodied AI.
02 From the paper
Abstract
World modeling has become a cornerstone in AI research, enabling agents to understand, represent, and predict the dynamic environments they inhabit. While prior work largely emphasizes generative methods for 2D image and video data, they overlook the rapidly growing body of work that leverages native 3D and 4D representations such as RGB-D imagery, occupancy grids, and LiDAR point clouds for large-scale scene modeling. At the same time, the absence of a standardized definition and taxonomy for "world models" has led to fragmented and sometimes inconsistent claims in the literature. This survey addresses these gaps by presenting the first comprehensive review explicitly dedicated to 3D and 4D world modeling and generation. We establish precise definitions, introduce a structured taxonomy spanning video-based (VideoGen), occupancy-based (OccGen), and LiDAR-based (LiDARGen) approaches, and systematically summarize datasets and evaluation metrics tailored to 3D/4D settings. We further discuss practical applications, identify open challenges, and highlight promising research directions, aiming to provide a coherent and foundational reference for advancing the field. A systematic summary of existing literature is available at https://github.com/worldbench/awesome-3d-4d-world-models