Paper 2604.19636
CoInteract: Physically-Consistent Human-Object Interaction Video Synthesis via Spatially-Structured Co-Generation
- Published
- Apr 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- 165 stars
01 In brief
Summary
CoInteract is an end-to-end framework for speech-driven human-object interaction (HOI) video synthesis, conditioned on a person reference image, a product reference image, text prompts, and speech audio.
It addresses two common failures in diffusion models: structural instability in hands and faces, and physically implausible contact (e.g., hand-object interpenetration).
The framework introduces two key components into a Diffusion Transformer (DiT) backbone: a Human-Aware Mixture-of-Experts (MoE) that routes tokens to region-specialized experts (head, hand, base) via spatially supervised routing, improving fine-grained structural fidelity with minimal parameter overhead; and a Spatially-Structured Co-Generation paradigm, a dual-stream training approach that jointly models an RGB appearance stream and an auxiliary HOI structure stream (silhouette-like rendering) to inject interaction geometry priors.
During training, the HOI stream attends to RGB tokens and its supervision regularizes shared backbone weights; at inference, the HOI branch is removed for zero-overhead RGB generation.
Experiments on a curated dataset (12K clips) show CoInteract outperforms existing methods (AnchorCrafter, Phantom, Humo, VACE, InteractAvatar, SkyReels-V3) in interaction plausibility (VLM-QA 0.72), hand quality (HQ 0.724), identity preservation (DINO_id 0.671, FaceSim 0.696), and temporal coherence (Smooth 0.9951).
Ablations confirm the importance of both components, with the HOI stream providing the largest gain in interaction plausibility and the MoE improving hand/face fidelity.
02 From the paper
Abstract
Synthesizing human--object interaction (HOI) videos has broad practical value in e-commerce, digital advertising, and virtual marketing. However, current diffusion models, despite their photorealistic rendering capability, still frequently fail on (i) the structural stability of sensitive regions such as hands and faces and (ii) physically plausible contact (e.g., avoiding hand--object interpenetration). We present CoInteract, an end-to-end framework for HOI video synthesis conditioned on a person reference image, a product reference image, text prompts, and speech audio. CoInteract introduces two complementary designs embedded into a Diffusion Transformer (DiT) backbone. First, we propose a Human-Aware Mixture-of-Experts (MoE) that routes tokens to lightweight, region-specialized experts via spatially supervised routing, improving fine-grained structural fidelity with minimal parameter overhead. Second, we propose Spatially-Structured Co-Generation, a dual-stream training paradigm that jointly models an RGB appearance stream and an auxiliary HOI structure stream to inject interaction geometry priors. During training, the HOI stream attends to RGB tokens and its supervision regularizes shared backbone weights; at inference, the HOI branch is removed for zero-overhead RGB generation. Experimental results demonstrate that CoInteract significantly outperforms existing methods in structural stability, logical consistency, and interaction realism.