Paper 2508.08086
Matrix-3D: Omnidirectional Explorable 3D World Generation
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 28
- GitHub
- 777 stars
01 In brief
Summary
Matrix-3D is a framework for generating omnidirectional, explorable 3D worlds from a single image or text prompt.
It uses panoramic representations to overcome the limited field of view of perspective-based methods.
The pipeline first generates a panorama image, then a trajectory-guided panoramic video using a diffusion model conditioned on scene mesh renders (instead of point clouds) to ensure geometric consistency.
The video is then lifted to 3D via two methods: an optimization-based pipeline using 3D Gaussian Splatting for high quality, and a feed-forward large panorama reconstruction model for speed.
The authors introduce the Matrix-Pano dataset, a synthetic collection of 116K panoramic video sequences with depth, camera poses, and text annotations, created in Unreal Engine.
Experiments show state-of-the-art performance in panoramic video generation and 3D world reconstruction, with better visual quality and camera controllability than baselines like 360DVD, Imagine360, GenEx, ViewCrafter, and TrajectoryCrafter.
Limitations include slow inference, occasional depth artifacts, and challenges in depth estimation from video latents.
Future work includes generating unseen areas, enhancing editability, and extending to dynamic scenes.
02 From the paper
Abstract
Explorable 3D world generation from a single image or text prompt forms a cornerstone of spatial intelligence. Recent works utilize video model to achieve wide-scope and generalizable 3D world generation. However, existing approaches often suffer from a limited scope in the generated scenes. In this work, we propose Matrix-3D, a framework that utilize panoramic representation for wide-coverage omnidirectional explorable 3D world generation that combines conditional video generation and panoramic 3D reconstruction. We first train a trajectory-guided panoramic video diffusion model that employs scene mesh renders as condition, to enable high-quality and geometrically consistent scene video generation. To lift the panorama scene video to 3D world, we propose two separate methods: (1) a feed-forward large panorama reconstruction model for rapid 3D scene reconstruction and (2) an optimization-based pipeline for accurate and detailed 3D scene reconstruction. To facilitate effective training, we also introduce the Matrix-Pano dataset, the first large-scale synthetic collection comprising 116K high-quality static panoramic video sequences with depth and trajectory annotations. Extensive experiments demonstrate that our proposed framework achieves state-of-the-art performance in panoramic video generation and 3D world generation. See more in https://matrix-3d.github.io.