The Year in AI Papers
The year/Google DeepMind

Paper 2603.26599

VGGRPO: Towards World-Consistent Video Generation with 4D Latent Reward

Zhaochong An, Orest Kupyn, Théo Uscidda, Andrea Colaco, Karan Ahuja, Serge Belongie, Mar Gonzalez-Franco, Marta Tintore Gazulla

Published
Mar 2026
Research lab
Google DeepMind
Citations
14
GitHub
Not linked

01 In brief

Summary

VGGRPO is a framework for geometry-aware post-training of video diffusion models, addressing geometric drift and unstable camera motion.

It introduces a Latent Geometry Model (LGM) that stitches video diffusion latents to a geometry foundation model (e.g., Any4D) via a lightweight connector, enabling direct prediction of 4D scene geometry (camera poses, depth, point maps, scene flow) from latents without RGB decoding.

VGGRPO then performs latent-space Group Relative Policy Optimization (GRPO) with two complementary rewards: a camera motion smoothness reward (penalizing jittery trajectories) and a geometry reprojection consistency reward (enforcing cross-view coherence).

Experiments on static and dynamic benchmarks show VGGRPO improves camera stability, geometric consistency, and overall quality over baselines (SFT, Epipolar-DPO, VideoGPA) on Wan2.1-1B and Wan2.2-5B models.

It also reduces compute and memory (e.g., reward computation time from 54.73s to 41.33s, peak memory from 76.80GB to 68.57GB) and generalizes to standard VBench captions.

The method supports dynamic scenes, overcoming static-scene limitations of prior work.

02 From the paper

Abstract

Large-scale video diffusion models achieve impressive visual quality, yet often fail to preserve geometric consistency. Prior approaches improve consistency either by augmenting the generator with additional modules or applying geometry-aware alignment. However, architectural modifications can compromise the generalization of internet-scale pretrained models, while existing alignment methods are limited to static scenes and rely on RGB-space rewards that require repeated VAE decoding, incurring substantial compute overhead and failing to generalize to highly dynamic real-world scenes. To preserve the pretrained capacity while improving geometric consistency, we propose VGGRPO (Visual Geometry GRPO), a latent geometry-guided framework for geometry-aware video post-training. VGGRPO introduces a Latent Geometry Model (LGM) that stitches video diffusion latents to geometry foundation models, enabling direct decoding of scene geometry from the latent space. By constructing LGM from a geometry model with 4D reconstruction capability, VGGRPO naturally extends to dynamic scenes, overcoming the static-scene limitations of prior methods. Building on this, we perform latent-space Group Relative Policy Optimization with two complementary rewards: a camera motion smoothness reward that penalizes jittery trajectories, and a geometry reprojection consistency reward that enforces cross-view geometric coherence. Experiments on both static and dynamic benchmarks show that VGGRPO improves camera stability, geometry consistency, and overall quality while eliminating costly VAE decoding, making latent-space geometry-guided reinforcement an efficient and flexible approach to world-consistent video generation.