The Year in AI Papers
The year/Independent research

Paper 2508.03694

LongVie: Multimodal-Guided Controllable Ultra-Long Video Generation

Jianxiong Gao, Zhaoxi Chen, Xian Liu, Jianfeng Feng, Chenyang Si, Yanwei Fu, Yu Qiao, Ziwei Liu

Published
Aug 2025
Research lab
Independent
Citations
19
GitHub
Not linked

01 In brief

Summary

LongVie is a framework for controllable ultra-long video generation, addressing temporal inconsistency and visual degradation in autoregressive generation.

It identifies three key issues: separate noise initialization, independent control signal normalization, and single-modality guidance limitations.

LongVie introduces unified noise initialization and global control signal normalization for temporal consistency, and a multi-modal control framework integrating dense (depth) and sparse (point) signals with a degradation-aware training strategy to balance their contributions.

The authors also introduce LongVGenBench, a benchmark of 100 high-resolution videos lasting over one minute.

Experiments show LongVie achieves state-of-the-art performance in controllability, consistency, and quality, and it supports downstream tasks like video editing, motion/scene transfer, and mesh-to-video generation.

Training uses 130,000 videos, and inference takes about 45 minutes for a one-minute video on a single A100 GPU.

02 From the paper

Abstract

Controllable ultra-long video generation is a fundamental yet challenging task. Although existing methods are effective for short clips, they struggle to scale due to issues such as temporal inconsistency and visual degradation. In this paper, we initially investigate and identify three key factors: separate noise initialization, independent control signal normalization, and the limitations of single-modality guidance. To address these issues, we propose LongVie, an end-to-end autoregressive framework for controllable long video generation. LongVie introduces two core designs to ensure temporal consistency: 1) a unified noise initialization strategy that maintains consistent generation across clips, and 2) global control signal normalization that enforces alignment in the control space throughout the entire video. To mitigate visual degradation, LongVie employs 3) a multi-modal control framework that integrates both dense (e.g., depth maps) and sparse (e.g., keypoints) control signals, complemented by 4) a degradation-aware training strategy that adaptively balances modality contributions over time to preserve visual quality. We also introduce LongVGenBench, a comprehensive benchmark consisting of 100 high-resolution videos spanning diverse real-world and synthetic environments, each lasting over one minute. Extensive experiments show that LongVie achieves state-of-the-art performance in long-range controllability, consistency, and quality.