Paper 2508.07981
Omni-Effects: Unified and Spatially-Controllable Visual Effects Generation
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 28
- GitHub
- 175 stars
01 In brief
Summary
Omni-Effects is a unified framework for generating spatially controllable visual effects (VFX) in videos, addressing limitations of existing per-effect LoRA training.
It introduces two key innovations: LoRA-based Mixture of Experts (LoRA-MoE) to integrate diverse effects in a single model while mitigating cross-task interference, and Spatial-Aware Prompt (SAP) with an Independent-Information Flow (IIF) module for precise spatial control and prevention of effect blending.
The authors constructed the Omni-VFX dataset with 55 effect categories using a pipeline combining image editing and First-Last Frame-to-Video synthesis, and developed an evaluation framework with metrics like Regional Dynamic Degree (RDD), Effect Occurrence Rate (EOR), and Effect Controllability Rate (ECR).
Experiments show Omni-Effects outperforms baselines (CogVideoX, LTX-Video, Wan2.1, CogV+CN) in single- and multi-VFX generation, achieving high EOR (0.97) and ECR (0.88) for single-VFX control, and demonstrating generalization to more than two effects.
The framework enables users to specify both the category and location of desired effects, with applications in film, games, and advertising.
02 From the paper
Abstract
Visual effects (VFX) are essential visual enhancements fundamental to modern cinematic production. Although video generation models offer cost-efficient solutions for VFX production, current methods are constrained by per-effect LoRA training, which limits generation to single effects. This fundamental limitation impedes applications that require spatially controllable composite effects, i.e., the concurrent generation of multiple effects at designated locations. However, integrating diverse effects into a unified framework faces major challenges: interference from effect variations and spatial uncontrollability during multi-VFX joint training. To tackle these challenges, we propose Omni-Effects, a first unified framework capable of generating prompt-guided effects and spatially controllable composite effects. The core of our framework comprises two key innovations: (1) LoRA-based Mixture of Experts (LoRA-MoE), which employs a group of expert LoRAs, integrating diverse effects within a unified model while effectively mitigating cross-task interference. (2) Spatial-Aware Prompt (SAP) incorporates spatial mask information into the text token, enabling precise spatial control. Furthermore, we introduce an Independent-Information Flow (IIF) module integrated within the SAP, isolating the control signals corresponding to individual effects to prevent any unwanted blending. To facilitate this research, we construct a comprehensive VFX dataset Omni-VFX via a novel data collection pipeline combining image editing and First-Last Frame-to-Video (FLF2V) synthesis, and introduce a dedicated VFX evaluation framework for validating model performance. Extensive experiments demonstrate that Omni-Effects achieves precise spatial control and diverse effect generation, enabling users to specify both the category and location of desired effects.