Paper 2606.13432
OmniDirector: General Multi-Shot Camera Cloning without Cross-Paired Data
- Published
- Jun 2026
- Research lab
- Independent
- Citations
- 5
- GitHub
- 79 stars
01 In brief
Summary
OmniDirector is a framework for cloning camera motion from reference videos to animate source images, supporting multi-shot sequences without requiring cross-paired training data.
It introduces a 'camera grid' representation, which renders camera parameters as a grid motion video within an empty 3D scene, decoupling camera motion from content and enabling scalable training on a million-scale dataset of camera grid-video pairs.
The framework is built on Multi-Modal Diffusion Transformers (MMDiTs), injecting the camera grid via token concatenation.
A hierarchical Prompt Expansion (PE) Agent at inference integrates camera motion, user prompts, and reference images into a unified text prompt, with inter-shot and intra-shot descriptions.
An adaptive classifier-free guidance strategy injects camera grid features during high-noise denoising stages.
Experiments show OmniDirector outperforms baselines (CamCloneMaster, Seedance2.0, LTX-LoRA) in camera accuracy (e.g., T-Pre 72.74% vs.
52.21% for CamCloneMaster), transition accuracy (Tem-Pre 96.52%, Sem-Pre 83.79%), and lower leakage rates (0.51% frame-level).
Ablations confirm the importance of semantic fusion, inter-shot prompts, and adaptive CFG.
The model also demonstrates emergent zero-shot camera control when conditioned on raw videos or Canny edges.
02 From the paper
Abstract
Cloning camera motion from reference videos is an important task in video generation, as videos provide intuitive and precise control. Existing methods either directly use parametric representations that fail to handle multi-shot generation or synthesize cross-paired data, which suffer from data scarcity, resulting in poor performance in complicated camera motion cloning. To address these issues, we introduce a general camera motion representation that encodes cameras as grid motion videos. This camera grid represents the camera parameters visually and supports the integration of diverse trajectories for multi-shot video generation. Building upon this, we propose OmniDirector, a unified framework trained on a million-scale camera grid-video pairs that coordinates characters, actions, and cameras to provide director-level control for multimodal diffusion transformers. Furthermore, we design a novel hierarchical prompt expansion agent that harmoniously integrates different control signals by systematically describing camera motion and visual content through understanding signal relationships. Extensive experiments demonstrate the superior performance and outstanding controllability of our framework. Project page: https://ymlinfeng.github.io/OmniDirector.github.io/