The Year in AI Papers
The year/Independent research

Paper 2605.00658

UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priors

Houyuan Chen, Hong Li, Xianghao Kong, Tianrui Zhu, Shaocong Xu, Weiqing Xiao, Yuwei Guo, Chongjie Ye, Lvmin Zhang, Hao Zhao, Anyi Rao

Published
May 2026
Research lab
Independent
Citations
0
GitHub
249 stars

01 In brief

Summary

UniVidX is a unified multimodal framework for versatile video generation that repurposes video diffusion model (VDM) priors to handle diverse tasks within a single model.

It addresses limitations of existing approaches that train separate models for fixed input-output mappings, ignoring cross-modal correlations.

UniVidX introduces three key designs: Stochastic Condition Masking (SCM) randomly partitions modalities into clean conditions and noisy targets during training, enabling omni-directional generation; Decoupled Gated LoRA (DGL) assigns per-modality LoRAs activated only when a modality is a generation target, preserving VDM priors and preventing parameter interference; and Cross-Modal Self-Attention (CMSA) shares keys/values across modalities while keeping queries modality-specific, ensuring cross-modal consistency.

The framework is instantiated in two models: UniVid-Intrinsic for RGB videos and intrinsic maps (albedo, irradiance, normal), and UniVid-Alpha for blended RGB and RGBA layers.

Both support three paradigms (Text→X, X→X, Text&X→X) covering 15 tasks.

Experiments show competitive performance with state-of-the-art methods, superior temporal stability, and robust generalization to in-the-wild scenarios despite training on fewer than 1k videos.

The framework also enables downstream applications like video relighting, retexturing, matting, and inpainting.

02 From the paper

Abstract

Recent progress has shown that video diffusion models (VDMs) can be repurposed for diverse multimodal graphics tasks. However, existing methods often train separate models for each problem setting, which fixes the input-output mapping and limits the modeling of correlations across modalities. We present UniVidX, a unified multimodal framework that leverages VDM priors for versatile video generation. UniVidX formulates pixel-aligned tasks as conditional generation in a shared multimodal space, adapts to modality-specific distributions while preserving the backbone's native priors, and promotes cross-modal consistency during synthesis. It is built on three key designs. Stochastic Condition Masking (SCM) randomly partitions modalities into clean conditions and noisy targets during training, enabling omni-directional conditional generation instead of fixed mappings. Decoupled Gated LoRA (DGL) introduces per-modality LoRAs that are activated when a modality serves as the generation target, preserving the strong priors of the VDM. Cross-Modal Self-Attention (CMSA) shares keys and values across modalities while keeping modality-specific queries, facilitating information exchange and inter-modal alignment. We instantiate UniVidX in two domains: UniVid-Intrinsic, for RGB videos and intrinsic maps including albedo, irradiance, and normal; and UniVid-Alpha, for blended RGB videos and their constituent RGBA layers. Experiments show that both models achieve performance competitive with state-of-the-art methods across distinct tasks and generalize robustly to in-the-wild scenarios, even when trained on fewer than 1,000 videos. Project page: https://houyuanchen111.github.io/UniVidX.github.io/