Paper 2601.00393
NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos
- Published
- Jan 2026
- Research lab
- Independent
- Citations
- 28
- GitHub
- 647 stars
01 In brief
Summary
NeoVerse is a 4D world model that reconstructs dynamic 4D Gaussian Splatting (4DGS) from monocular videos in a feed-forward, pose-free manner, enabling novel-trajectory video generation and downstream applications.
It addresses scalability limitations of prior methods by avoiding expensive multi-view data and offline preprocessing.
Key innovations include a bidirectional motion-encoding branch for temporal interpolation, efficient on-the-fly reconstruction from sparse key frames, and online monocular degradation simulation (visibility-based Gaussian culling, average geometry filter) to generate training pairs.
The generation model uses a frozen video diffusion transformer with a trainable control branch, conditioned on degraded renderings (RGB, depth, masks, Plücker embeddings).
Training uses 18 public datasets plus over 1M in-the-wild monocular videos.
NeoVerse achieves state-of-the-art results on static (VRNeRF, Scannet++) and dynamic (ADT, DyCheck) reconstruction benchmarks, and on VBench for novel view generation, outperforming TrajectoryCrafter and ReCamMaster.
It also supports 3D tracking, video editing, stabilization, and super-resolution.
Limitations include difficulty with 2D data lacking 3D structure and occasional text rendering failures.
Source code will be released.
02 From the paper
Abstract
In this paper, we propose NeoVerse, a versatile 4D world model that is capable of 4D reconstruction, novel-trajectory video generation, and rich downstream applications. We first identify a common limitation of scalability in current 4D world modeling methods, caused either by expensive and specialized multi-view 4D data or by cumbersome training pre-processing. In contrast, our NeoVerse is built upon a core philosophy that makes the full pipeline scalable to diverse in-the-wild monocular videos. Specifically, NeoVerse features pose-free feed-forward 4D reconstruction, online monocular degradation pattern simulation, and other well-aligned techniques. These designs empower NeoVerse with versatility and generalization to various domains. Meanwhile, NeoVerse achieves state-of-the-art performance in standard reconstruction and generation benchmarks. Our project page is available at https://neoverse-4d.github.io.