Paper 2511.16668
V-ReasonBench: Toward Unified Reasoning Benchmark Suite for Video Generation Models
- Published
- Nov 2025
- Research lab
- Independent
- Citations
- 17
- GitHub
- 36 stars
01 In brief
Summary
V-ReasonBench is a benchmark for evaluating reasoning in generative video models under the Chain-of-Frame paradigm, where the final frame represents the model's answer.
It covers four reasoning dimensions: structured problem-solving (arithmetic, code execution, Sudoku, Tic-Tac-Toe), spatial cognition (shape fitting, visual symmetry, color connection), pattern-based inference (sequence completion, analogy solving, rule following), and physical dynamics (block sliding, communicating vessels, temperature-induced deformation).
The benchmark uses 326 instances with initial-final image pairs, generating 9,780 videos across six models.
Evaluation combines mask-based, grid-based, and VLM-based scoring, achieving 97.09% human alignment.
Results show Sora-2 leads in structured (72.00), spatial (36.76), and pattern-based (40.00) reasoning, while Hailuo-02 and Vidu-Q2 excel in physical dynamics (36.67).
The study finds that longer video durations do not consistently improve reasoning, video models outperform image models on physical and procedural tasks, and models often hallucinate intermediate frames or add visual embellishments that break structural accuracy.
V-ReasonBench provides a reproducible framework for measuring video reasoning beyond visual fidelity.
02 From the paper
Abstract
Recent progress in generative video models, such as Veo-3, has shown surprising zero-shot reasoning abilities, creating a growing need for systematic and reliable evaluation. We introduce V-ReasonBench, a benchmark designed to assess video reasoning across four key dimensions: structured problem-solving, spatial cognition, pattern-based inference, and physical dynamics. The benchmark is built from both synthetic and real-world image sequences and provides a diverse set of answer-verifiable tasks that are reproducible, scalable, and unambiguous. Evaluations of six state-of-the-art video models reveal clear dimension-wise differences, with strong variation in structured, spatial, pattern-based, and physical reasoning. We further compare video models with strong image models, analyze common hallucination behaviors, and study how video duration affects Chain-of-Frames reasoning. Overall, V-ReasonBench offers a unified and reproducible framework for measuring video reasoning and aims to support the development of models with more reliable, human-aligned reasoning skills.