Paper 2510.08555
VideoCanvas: Unified Video Completion from Arbitrary Spatiotemporal Patches via In-Context Conditioning
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 4
- GitHub
- 68 stars
01 In brief
Summary
VideoCanvas is a unified framework for arbitrary spatio-temporal video completion, where users specify patches at any spatial location and timestamp, and the model generates a coherent video.
The paper identifies that causal video VAEs compress multiple frames into a single latent slot, creating temporal ambiguity, and that zero-padding in video mode causes out-of-distribution artifacts.
To address this, VideoCanvas uses a hybrid conditioning strategy: spatial zero-padding in image mode for in-distribution VAE encoding, and Temporal RoPE Interpolation to assign fractional temporal positions to condition tokens for precise frame-level alignment.
This approach works on frozen VAEs without retraining or new parameters.
The authors introduce VideoCanvasBench, a benchmark with over 2,000 test cases covering AnyP2V, AnyI2V, and AnyV2V tasks.
Experiments show state-of-the-art performance across tasks like image-to-video, interpolation, inpainting, and outpainting, with ablations confirming the effectiveness of RoPE interpolation over alternatives.
The framework also enables applications like video extension, looping, and camera control.
02 From the paper
Abstract
Existing controllable video generation methods are typically designed for rigid, task-specific settings, such as first-frame image-to-video, inpainting, or interpolation, treating spatio-temporal control as a set of isolated problems. We formalize a unified task, arbitrary spatio-temporal video completion, where a model generates a coherent video from user-specified patches placed at any spatial location and timestamp. However, realizing such a unified framework within modern latent video diffusion models is non-trivial: causal video VAEs compress multiple frames into a single latent slot, making frame-level conditioning fundamentally ill-posed, and directly feeding sparsely populated, zero-padded video inputs into the VAE leads to severe out-of-distribution artifacts. To address these challenges, we propose VideoCanvas, a simple yet effective framework that adapts the In-Context Conditioning paradigm to arbitrary spatio-temporal completion without modifying or retraining the VAE. Our key idea is a hybrid conditioning strategy that decouples spatial and temporal control: spatially, we encode zero-padded full-frame canvases in image mode to keep VAE inputs in-distribution, and temporally we use Temporal RoPE Interpolation to assign each condition a continuous fractional index in the latent sequence for precise frame-level alignment. To evaluate this capability, we develop VideoCanvasBench, the first benchmark for arbitrary spatio-temporal video completion, covering both intra-scene fidelity and inter-scene creativity. Extensive experiments demonstrate that VideoCanvas achieves state-of-the-art performance across a diverse range of video generation tasks under a single, unified framework.