Paper 2508.20470
Droplet3D: Commonsense Priors from Videos Facilitate 3D Generation
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 2
- GitHub
- 43 stars
01 In brief
Summary
Droplet3D addresses 3D data scarcity by leveraging commonsense priors from videos for 3D generation.
The authors introduce Droplet3D-4M, a large-scale dataset of 4 million 3D models, each with an 85-frame 360-degree orbital rendering video and dense multi-view-level text captions averaging 260 words.
They also present Droplet3D, a generative model fine-tuned from the video diffusion model DropletVideo, which accepts both an image and dense text input to generate multi-view images.
The model uses a 3D causal VAE and a modality-expert transformer, with modules for text rewriting and canonical view alignment.
Experiments show Droplet3D outperforms LGM and MVControl on the GSO dataset in PSNR, LPIPS, MSE, and CLIP-S, and demonstrates controllable creativity, stylized image lifting, and scene-level generation.
The authors open-sourced the dataset, code, and model weights.
02 From the paper
Abstract
Scaling laws have validated the success and promise of large-data-trained models in creative generation across text, image, and video domains. However, this paradigm faces data scarcity in the 3D domain, as there is far less of it available on the internet compared to the aforementioned modalities. Fortunately, there exist adequate videos that inherently contain commonsense priors, offering an alternative supervisory signal to mitigate the generalization bottleneck caused by limited native 3D data. On the one hand, videos capturing multiple views of an object or scene provide a spatial consistency prior for 3D generation. On the other hand, the rich semantic information contained within the videos enables the generated content to be more faithful to the text prompts and semantically plausible. This paper explores how to apply the video modality in 3D asset generation, spanning datasets to models. We introduce Droplet3D-4M, the first large-scale video dataset with multi-view level annotations, and train Droplet3D, a generative model supporting both image and dense text input. Extensive experiments validate the effectiveness of our approach, demonstrating its ability to produce spatially consistent and semantically plausible content. Moreover, in contrast to the prevailing 3D solutions, our approach exhibits the potential for extension to scene-level applications. This indicates that the commonsense priors from the videos significantly facilitate 3D creation. We have open-sourced all resources including the dataset, code, technical framework, and model weights: https://dropletx.github.io/.