Paper 2603.19235
Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding
- Published
- Mar 2026
- Research lab
- Independent
- Citations
- 10
- GitHub
- 421 stars
01 In brief
Summary
VEGA-3D is a plug-and-play framework that repurposes pre-trained video generation models as Latent World Simulators to provide implicit 3D priors for Multimodal Large Language Models (MLLMs), addressing their spatial blindness.
The method extracts spatiotemporal features from intermediate noise levels of a frozen video diffusion model (e.g., Wan2.1-T2V) and integrates them with semantic features via a token-level adaptive gated fusion mechanism, without explicit 3D supervision.
Experiments show consistent improvements over state-of-the-art baselines on 3D scene understanding benchmarks (ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, SQA3D), spatial reasoning (VSI-Bench), and robotic manipulation (LIBERO), with gains attributed to the complementary nature of generative and semantic features.
The most informative priors come from intermediate DiT layers and mid-denoising timesteps, and multi-view correspondence strongly correlates with downstream performance.
The framework is scalable, as advances in video generation directly translate to stronger 3D priors, though it incurs additional inference cost.
02 From the paper
Abstract
While Multimodal Large Language Models demonstrate impressive semantic capabilities, they often suffer from spatial blindness, struggling with fine-grained geometric reasoning and physical dynamics. Existing solutions typically rely on explicit 3D modalities or complex geometric scaffolding, which are limited by data scarcity and generalization challenges. In this work, we propose a paradigm shift by leveraging the implicit spatial prior within large-scale video generation models. We posit that to synthesize temporally coherent videos, these models inherently learn robust 3D structural priors and physical laws. We introduce VEGA-3D (Video Extracted Generative Awareness), a plug-and-play framework that repurposes a pre-trained video diffusion model as a Latent World Simulator. By extracting spatiotemporal features from intermediate noise levels and integrating them with semantic representations via a token-level adaptive gated fusion mechanism, we enrich MLLMs with dense geometric cues without explicit 3D supervision. Extensive experiments across 3D scene understanding, spatial reasoning, and embodied manipulation benchmarks demonstrate that our method outperforms state-of-the-art baselines, validating that generative priors provide a scalable foundation for physical-world understanding. Code is publicly available at https://github.com/H-EmbodVis/VEGA-3D.