Paper 2606.09828
Latent Spatial Memory for Video World Models
- Published
- Jun 2026
- Research lab
- Independent
- Citations
- 2
- GitHub
- 297 stars
01 In brief
Summary
This paper introduces latent spatial memory, a persistent 3D cache for video world models that stores scene information directly in the diffusion latent space, avoiding the pixel-space round trip of RGB point-cloud memories.
The authors propose Mirage, a framework that constructs the memory by lifting latent tokens into 3D via depth-guided back-projection and queries it through direct latent-space warping.
This design eliminates information loss and computational overhead, achieving up to 10.57× faster end-to-end video generation and 55× lower GPU memory usage than RGB-cache baselines.
Mirage attains state-of-the-art performance on WorldScore and strong reconstruction quality on RealEstate10K.
The method uses a ControlNet-style side branch for conditioning and a two-stage training process with LoRA adapters.
Ablations confirm the benefits of latent over RGB memory, feature downsampling, dynamic object filtering, and two-stage training.
The paper also discusses limitations, such as not persisting dynamic actors across chunks.
02 From the paper
Abstract
Video world models that maintain 3D spatial consistency across generated frames typically rely on explicit point cloud memory constructed in RGB space. This design is both computationally expensive, requiring repeated rendering and VAE encoding, and inherently lossy, as the round trip through pixel space discards rich features of the learned latent representation. In this paper, we introduce \emph{latent spatial memory} for video world models, a persistent 3D cache that stores scene information directly in the diffusion latent space, avoiding pixel-space reconstruction. Building on this, we propose Mirage, a latent-space spatial memory framework that constructs the memory by lifting latent tokens into 3D via depth-guided back-projection and queries it by synthesizing novel views through direct latent-space warping. This unified formulation eliminates both the information loss of pixel-space reconstruction and the computational burden of repeated encoding and rendering. Experiments show that latent spatial memory achieves up to \textbf{10.57}$\times$ faster end-to-end video generation and \textbf{55}$\times$ reduction in memory footprint relative to explicit 3D baselines. Leveraging the geometric prior of the diffusion model, Mirage attains state-of-the-art performance on WorldScore and strong reconstruction quality on RealEstate10K.