Paper 2512.14691
MMGR: Multi-Modal Generative Reasoning
- Published
- Dec 2025
- Research lab
- Independent
- Citations
- 9
- GitHub
- Not linked
01 In brief
Summary
The paper introduces MMGR (Multi-Modal Generative Reasoning), a benchmark suite to evaluate the reasoning capabilities of video and image generation models across five core abilities: Physical, Logical, 3D Spatial, 2D Spatial, and Temporal reasoning.
It comprises three domains: Abstract Reasoning (Maze, Sudoku, ARC-AGI, Math), Embodied Navigation (four tasks), and Physical Commonsense (Physical Concepts, Sports), totaling 1,853 samples.
Evaluations of models like Veo-3, Sora-2, Wan-2.2, Nano-banana, and GPT-4o-image reveal that while models perform moderately on Physical Commonsense (e.g., Sora-2 70% overall), they fail on Abstract Reasoning (e.g., <10% on ARC-AGI) and struggle with long-horizon spatial planning.
Human evaluations show that VLM-based auto-evaluation often overestimates performance, especially in detecting transient physical violations.
Key limitations identified include training data imbalance (lack of symbolic reasoning data), architectural weaknesses in maintaining global consistency, and optimization objectives that favor visual plausibility over causal correctness.
The benchmark provides a diagnostic framework for improving world models toward physical and logical consistency.
02 From the paper
Abstract
Video foundation models generate visually realistic and temporally coherent content, but their reliability as world simulators depends on whether they capture physical, logical, and spatial constraints. Existing metrics such as Frechet Video Distance (FVD) emphasize perceptual quality and overlook reasoning failures, including violations of causality, physics, and global consistency. We introduce MMGR (Multi-Modal Generative Reasoning Evaluation and Benchmark), a principled evaluation framework based on five reasoning abilities: Physical, Logical, 3D Spatial, 2D Spatial, and Temporal. MMGR evaluates generative reasoning across three domains: Abstract Reasoning (ARC-AGI, Sudoku), Embodied Navigation (real-world 3D navigation and localization), and Physical Commonsense (sports and compositional interactions). MMGR applies fine-grained metrics that require holistic correctness across both video and image generation. We benchmark leading video models (Veo-3, Sora-2, Wan-2.2) and image models (Nano-banana, Nano-banana Pro, GPT-4o-image, Qwen-image), revealing strong performance gaps across domains. Models show moderate success on Physical Commonsense tasks but perform poorly on Abstract Reasoning (below 10 percent accuracy on ARC-AGI) and struggle with long-horizon spatial planning in embodied settings. Our analysis highlights key limitations in current models, including overreliance on perceptual data, weak global state consistency, and objectives that reward visual plausibility over causal correctness. MMGR offers a unified diagnostic benchmark and a path toward reasoning-aware generative world models.