The Year in AI Papers
The year/Independent research

Paper 2604.11804

OmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generation

Donghao Zhou, Guisheng Liu, Hao Yang, Jiatong Li, Jingyu Lin, Xiaohu Huang, Yichen Liu, Xin Gao, Cunjian Chen, Shilei Wen, Chi-Wing Fu, Pheng-Ann Heng

Published
Apr 2026
Research lab
Independent
Citations
3
GitHub
464 stars

01 In brief

Summary

OmniShow is an end-to-end framework for Human-Object Interaction Video Generation (HOIVG), synthesizing videos conditioned on text, reference images, audio, and pose.

It introduces Unified Channel-wise Conditioning to inject reference images and pose via channel concatenation with pseudo-frame tokens, and Gated Local-Context Attention for precise audio-visual synchronization using masked attention and adaptive gating.

To address data scarcity, a Decoupled-Then-Joint Training strategy trains specialized R2V and A2V models, then merges them via weight interpolation before joint fine-tuning.

The authors also establish HOIVG-Bench, a benchmark with 135 curated samples and metrics across text alignment, reference consistency, pose accuracy, audio-visual sync, and video quality.

Experiments show OmniShow achieves state-of-the-art or competitive performance in R2V, RA2V, and RP2V settings, and is the only model supporting full RAP2V generation.

Ablations validate each component, and human evaluations favor OmniShow.

The model is built on Waver 1.0 (12B MMDiT) and adds only ~2.5% parameters.

02 From the paper

Abstract

In this work, we study Human-Object Interaction Video Generation (HOIVG), which aims to synthesize high-quality human-object interaction videos conditioned on text, reference images, audio, and pose. This task holds significant practical value for automating content creation in real-world applications, such as e-commerce demonstrations, short video production, and interactive entertainment. However, existing approaches fail to accommodate all these requisite conditions. We present OmniShow, an end-to-end framework tailored for this practical yet challenging task, capable of harmonizing multimodal conditions and delivering industry-grade performance. To overcome the trade-off between controllability and quality, we introduce Unified Channel-wise Conditioning for efficient image and pose injection, and Gated Local-Context Attention to ensure precise audio-visual synchronization. To effectively address data scarcity, we develop a Decoupled-Then-Joint Training strategy that leverages a multi-stage training process with model merging to efficiently harness heterogeneous sub-task datasets. Furthermore, to fill the evaluation gap in this field, we establish HOIVG-Bench, a dedicated and comprehensive benchmark for HOIVG. Extensive experiments demonstrate that OmniShow achieves overall state-of-the-art performance across various multimodal conditioning settings, setting a solid standard for the emerging HOIVG task.