Paper 2511.03334
UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions
- Published
- Nov 2025
- Research lab
- Independent
- Citations
- 28
- GitHub
- 57 stars
01 In brief
Summary
UniAVGen is a unified framework for human-centric joint audio and video generation, addressing limitations in existing methods like poor lip synchronization and semantic inconsistency.
It uses a dual-branch architecture with two parallel Diffusion Transformers (DiTs) for video and audio, enabling a cohesive cross-modal latent space.
The core innovation is an Asymmetric Cross-Modal Interaction mechanism with audio-to-video (A2V) and video-to-audio (V2A) aligners that use temporal context windows and interpolation for precise alignment.
A Face-Aware Modulation (FAM) module dynamically predicts facial masks to focus cross-modal interactions on salient regions, with a decaying loss weight to relax constraints over training.
During inference, Modality-Aware Classifier-Free Guidance (MA-CFG) amplifies cross-modal correlation signals.
UniAVGen supports multiple tasks (joint generation, dubbing, audio-driven synthesis) within one model.
Experiments show it outperforms existing methods in audio-video synchronization, timbre, and emotion consistency, using far fewer training samples (1.3M vs.
30.1M).
02 From the paper
Abstract
Due to the lack of effective cross-modal modeling, existing open-source audio-video generation methods often exhibit compromised lip synchronization and insufficient semantic consistency. To mitigate these drawbacks, we propose UniAVGen, a unified framework for joint audio and video generation. UniAVGen is anchored in a dual-branch joint synthesis architecture, incorporating two parallel Diffusion Transformers (DiTs) to build a cohesive cross-modal latent space. At its heart lies an Asymmetric Cross-Modal Interaction mechanism, which enables bidirectional, temporally aligned cross-attention, thus ensuring precise spatiotemporal synchronization and semantic consistency. Furthermore, this cross-modal interaction is augmented by a Face-Aware Modulation module, which dynamically prioritizes salient regions in the interaction process. To enhance generative fidelity during inference, we additionally introduce Modality-Aware Classifier-Free Guidance, a novel strategy that explicitly amplifies cross-modal correlation signals. Notably, UniAVGen's robust joint synthesis design enables seamless unification of pivotal audio-video tasks within a single model, such as joint audio-video generation and continuation, video-to-audio dubbing, and audio-driven video synthesis. Comprehensive experiments validate that, with far fewer training samples (1.3M vs. 30.1M), UniAVGen delivers overall advantages in audio-video synchronization, timbre consistency, and emotion consistency.