The Year in AI Papers
The year/NVIDIA

Paper 2605.18739

LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation

Yukang Chen, Luozhou Wang, Wei Huang, Shuai Yang, Bohan Zhang, Yicheng Xiao, Ruihang Chu, Weian Mao, Qixin Hu, Shaoteng Liu, Yuyang Zhao, Huizi Mao, et al.

Published
May 2026
Research lab
NVIDIA
Citations
11
GitHub
2.5K stars

01 In brief

Summary

LongLive-2.0 is an NVFP4-based parallel infrastructure for long video generation, co-designing training and inference.

For training, it introduces Balanced SP, a sequence-parallel autoregressive (AR) training method that pairs clean-history and noisy-target temporal chunks on each GPU, enabling a natural teacher-forcing mask and SP-aware chunked VAE encoding.

Combined with NVFP4 precision, this reduces GPU memory and accelerates GEMM computation, yielding up to 2.15x training speedup (e.g., 64s training iteration time drops from 1372.9 ms to 639.5 ms).

The infrastructure enables a clean pipeline: directly fine-tuning a diffusion model into a long, multi-shot, interactive AR model, with few-step distillation via standalone LoRA weights.

For inference on Blackwell GPUs, it enables W4A4 NVFP4 inference, quantizes the KV cache to NVFP4 (reducing memory from 35.4 GB to 19.4 GB), and uses asynchronous streaming VAE decoding.

On non-Blackwell GPUs, SP inference matches speed, with quantized KV cache reducing inter-GPU communication.

LongLive-2.0-5B achieves 45.7 FPS inference (2-step) and strong benchmark performance, being the first end-to-end NVFP4 training and inference system for long video generation.

02 From the paper

Abstract

We present LongLive-2.0, an NVFP4-based parallel infrastructure throughout the full training and inference workflow of long video generation, addressing speed and memory bottlenecks. For training, we introduce sequence-parallel autoregressive (AR) training, instantiated as Balanced SP, which co-designs the efficient teacher-forcing layout with SP execution by pairing clean-history and noisy-target temporal chunks on each rank, enabling a natural teacher-forcing mask with SP-aware chunked VAE encoding. Combined with NVFP4 precision, it reduces GPU memory cost and accelerates GEMM computation during training, the proportion of which increases as video length grows. Moreover, we show that a high-quality infrastructure and dataset enable a remarkably clean training pipeline. Unlike existing Self-Forcing series methods that rely on ODE initialization and subsequent distribution matching distillation (DMD), LongLive-2.0 directly tunes a diffusion model into a long, multi-shot, interactive auto-regressive (AR) diffusion model. It can be further converted to real-time generation (4 to 2 denoising steps) with standalone LoRA weights. For inference on Blackwell GPUs, we enable W4A4 NVFP4 inference, quantize KV cache into NVFP4 for memory savings, and boost end-to-end throughput with asynchronous streaming VAE decoding. On non-Blackwell GPU architectures, we deploy SP inference to match the speed on Blackwell GPUs, while the quantized KV cache can lower inter-GPU communication of SP. Experiments show up to 2.15x speedup in training, and 1.84x in inference. LongLive-2.0-5B achieves 45.7 FPS inference while attaining strong performance on benchmarks. To our knowledge, LongLive-2.0 is the first NVFP4 training and inference system for long video generation.