Paper 2603.17117
MosaicMem: Hybrid Spatial Memory for Controllable Video World Models
- Published
- Mar 2026
- Research lab
- Independent
- Citations
- 10
- GitHub
- Not linked
01 In brief
Summary
MosaicMem is a hybrid spatial memory mechanism for video world models that combines explicit 3D structure with implicit, attention-based conditioning.
It lifts video patches into 3D for precise localization and retrieval, then composes them in the queried view via a patch-and-compose interface, allowing the model to preserve persistent elements while inpainting dynamic changes.
The method integrates PRoPE camera conditioning and two alignment mechanisms (Warped RoPE and Warped Latent) to improve pose adherence and dynamic modeling.
Experiments on a new benchmark (MosaicMem-World) show that MosaicMem outperforms both explicit (e.g., GEN3C) and implicit (e.g., Context-as-Memory) baselines in camera control, visual quality, consistency, and dynamic score.
It also enables minute-level navigation, memory-based scene editing, and autoregressive generation via Mosaic Forcing, achieving real-time 16 FPS generation with superior quality and consistency compared to RELIC and Matrix-Game.
02 From the paper
Abstract
Video diffusion models are moving beyond short, plausible clips toward world simulators that must remain consistent under camera motion, revisits, and intervention. Yet spatial memory remains a key bottleneck: explicit 3D structures can improve reprojection-based consistency but struggle to depict moving objects, while implicit memory often produces inaccurate camera motion even with correct poses. We propose Mosaic Memory (MosaicMem), a hybrid spatial memory that lifts patches into 3D for reliable localization and targeted retrieval, while exploiting the model's native conditioning to preserve prompt-following generation. MosaicMem composes spatially aligned patches in the queried view via a patch-and-compose interface, preserving what should persist while allowing the model to inpaint what should evolve. With PRoPE camera conditioning and two new memory alignment methods, experiments show improved pose adherence compared to implicit memory and stronger dynamic modeling than explicit baselines. MosaicMem further enables minute-level navigation, memory-based scene editing, and autoregressive rollout.