The Year in AI Papers
The year/Independent research

Paper 2512.20619

SemanticGen: Video Generation in Semantic Space

Jianhong Bai, Xiaoshi Wu, Xintao Wang, Xiao Fu, Yuanxing Zhang, Qinghe Wang, Xiaoyu Shi, Menghan Xia, Zuozhu Liu, Haoji Hu, Pengfei Wan, Kun Gai

Published
Dec 2025
Research lab
Independent
Citations
7
GitHub
Not linked

01 In brief

Summary

SemanticGen is a novel video generation framework that operates in a compact semantic space rather than directly in the VAE latent space.

It uses a two-stage process: first, a diffusion model generates compressed semantic video features (using Qwen-2.5-VL as the semantic encoder) that define the global layout; second, another diffusion model generates VAE latents conditioned on these features to produce the final video.

A lightweight MLP compresses the semantic space to improve convergence and sampling efficiency.

For long videos, full attention is applied only in the semantic space, while shifted-window attention is used in the VAE space to reduce computational cost.

Experiments show SemanticGen achieves faster convergence than VAE-space modeling and outperforms state-of-the-art baselines (e.g., HunyuanVideo, Wan2.1, SkyReels-V2, Self-Forcing, LongLive) on VBench metrics for both short and long video generation, with lower drift (ΔM_drift=3.58%).

Limitations include loss of high-frequency temporal details due to low-fps semantic sampling and inconsistent textures in long videos.

02 From the paper

Abstract

State-of-the-art video generative models typically learn the distribution of video latents in the VAE space and map them to pixels using a VAE decoder. While this approach can generate high-quality videos, it suffers from slow convergence and is computationally expensive when generating long videos. In this paper, we introduce SemanticGen, a novel solution to address these limitations by generating videos in the semantic space. Our main insight is that, due to the inherent redundancy in videos, the generation process should begin in a compact, high-level semantic space for global planning, followed by the addition of high-frequency details, rather than directly modeling a vast set of low-level video tokens using bi-directional attention. SemanticGen adopts a two-stage generation process. In the first stage, a diffusion model generates compact semantic video features, which define the global layout of the video. In the second stage, another diffusion model generates VAE latents conditioned on these semantic features to produce the final output. We observe that generation in the semantic space leads to faster convergence compared to the VAE latent space. Our method is also effective and computationally efficient when extended to long video generation. Extensive experiments demonstrate that SemanticGen produces high-quality videos and outperforms state-of-the-art approaches and strong baselines.