The Year in AI Papers
The year/Independent research

Paper 2509.08519

HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning

Liyang Chen, Tianxiang Ma, Jiawei Liu, Bingchuan Li, Zhuowei Chen, Lijie Liu, Xu He, Gen Li, Qian He, Zhiyong Wu

Published
Sep 2025
Research lab
Independent
Citations
45
GitHub
1.3K stars

01 In brief

Summary

HuMo is a unified framework for Human-Centric Video Generation (HCVG) that enables collaborative control from text, reference images, and audio.

It addresses two challenges: data scarcity and difficulty in coordinating sub-tasks of subject preservation and audio-visual sync.

To overcome data scarcity, HuMo constructs a high-quality dataset with paired text, reference images, and audio via a two-stage processing pipeline.

For training, it proposes a progressive paradigm: Stage 1 uses a minimal-invasive image injection strategy (concatenating reference latents, updating only self-attention layers) for subject preservation; Stage 2 adds audio cross-attention layers and a focus-by-predicting strategy (a mask predictor with BCE loss) for audio-visual sync, while progressively increasing the audio task weight.

During inference, a time-adaptive Classifier-Free Guidance (CFG) dynamically adjusts guidance weights across denoising steps.

Experiments show HuMo surpasses specialized state-of-the-art methods in both sub-tasks, with models of 1.7B and 17B parameters, and supports flexible input compositions (text-image, text-audio, text-image-audio).

02 From the paper

Abstract

Human-Centric Video Generation (HCVG) methods seek to synthesize human videos from multimodal inputs, including text, image, and audio. Existing methods struggle to effectively coordinate these heterogeneous modalities due to two challenges: the scarcity of training data with paired triplet conditions and the difficulty of collaborating the sub-tasks of subject preservation and audio-visual sync with multimodal inputs. In this work, we present HuMo, a unified HCVG framework for collaborative multimodal control. For the first challenge, we construct a high-quality dataset with diverse and paired text, reference images, and audio. For the second challenge, we propose a two-stage progressive multimodal training paradigm with task-specific strategies. For the subject preservation task, to maintain the prompt following and visual generation abilities of the foundation model, we adopt the minimal-invasive image injection strategy. For the audio-visual sync task, besides the commonly adopted audio cross-attention layer, we propose a focus-by-predicting strategy that implicitly guides the model to associate audio with facial regions. For joint learning of controllabilities across multimodal inputs, building on previously acquired capabilities, we progressively incorporate the audio-visual sync task. During inference, for flexible and fine-grained multimodal control, we design a time-adaptive Classifier-Free Guidance strategy that dynamically adjusts guidance weights across denoising steps. Extensive experimental results demonstrate that HuMo surpasses specialized state-of-the-art methods in sub-tasks, establishing a unified framework for collaborative multimodal-conditioned HCVG. Project Page: https://phantom-video.github.io/HuMo.