Paper 2604.02329
Generative World Renderer
- Published
- Apr 2026
- Research lab
- Independent
- Citations
- 1
- GitHub
- 709 stars
01 In brief
Summary
The paper introduces a large-scale dataset for generative world rendering, curated from two AAA games (Cyberpunk 2077 and Black Myth: Wukong) to address the domain gap in inverse and forward rendering.
The dataset includes over 4 million frames at 720p/30 FPS, with synchronized RGB and five G-buffer channels (depth, normals, albedo, metallic, roughness), captured via a novel dual-screen stitched method and graphics API interception.
It covers diverse scenes, weathers, and motion-blur variants.
The authors propose a VLM-based evaluation protocol for real-world scenarios without ground truth, assessing semantic, spatial, and temporal consistency.
Experiments show that fine-tuning DiffusionRenderer on this dataset improves cross-dataset generalization and controllable generation, with VLM evaluations correlating with human judgment.
The dataset also enables G-buffer-guided video generation and game editing via text prompts.
The work includes a toolkit for data curation and is released under CC BY-NC-SA 4.0 with gated access.
02 From the paper
Abstract
Scaling generative inverse and forward rendering to real-world scenarios is bottlenecked by the limited realism and temporal coherence of existing synthetic datasets. To bridge this persistent domain gap, we introduce a large-scale, dynamic dataset curated from visually complex AAA games. Using a novel dual-screen stitched capture method, we extracted 4M continuous frames (720p/30 FPS) of synchronized RGB and five G-buffer channels across diverse scenes, visual effects, and environments, including adverse weather and motion-blur variants. This dataset uniquely advances bidirectional rendering: enabling robust in-the-wild geometry and material decomposition, and facilitating high-fidelity G-buffer-guided video generation. Furthermore, to evaluate the real-world performance of inverse rendering without ground truth, we propose a novel VLM-based assessment protocol measuring semantic, spatial, and temporal consistency. Experiments demonstrate that inverse renderers fine-tuned on our data achieve superior cross-dataset generalization and controllable generation, while our VLM evaluation strongly correlates with human judgment. Combined with our toolkit, our forward renderer enables users to edit styles of AAA games from G-buffers using text prompts.