Paper 2604.05015
Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding
- Published
- Apr 2026
- Research lab
- Independent
- Citations
- 23
- GitHub
- 369 stars
01 In brief
Summary
Video-MME-v2 is a new benchmark for evaluating video multimodal large language models (MLLMs), designed to address the gap between inflated leaderboard scores and real-world capabilities.
It introduces a progressive three-level hierarchy (visual information aggregation, temporal dynamics modeling, and complex reasoning) and a group-based evaluation strategy with non-linear scoring to enforce consistency and reasoning coherence.
The dataset includes 800 videos and 3,200 questions, curated with 3,300 human-hours from 12 annotators and 50 reviewers.
Results show a large gap between the best model (Gemini-3-Pro, 49.4) and human experts (90.7), with errors in lower-level perception propagating to higher-level reasoning.
Thinking modes improve performance with subtitles but can degrade it without, indicating over-reliance on language priors.
The benchmark aims to provide a more faithful assessment of video MLLM capabilities and drive future development.
02 From the paper
Abstract
With the rapid advancement of video understanding, existing benchmarks are becoming increasingly saturated, exposing a critical discrepancy between inflated leaderboard scores and real-world model capabilities. To address this widening gap, we introduce Video-MME-v2, a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding. To systematically evaluate model capabilities, we design a \textbf{progressive tri-level hierarchy} that incrementally increases the complexity of video comprehension, ranging from multi-point visual information aggregation, to temporal dynamics modeling, and ultimately to complex multimodal reasoning. Besides, in contrast to conventional per-question accuracy, we propose a \textbf{group-based non-linear evaluation} strategy that enforces both consistency across related queries and coherence in multi-step reasoning. It penalizes fragmented or guess-based correctness and assigns credit only to answers supported by valid reasoning. To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline, involving 12 annotators and 50 independent reviewers. Backed by \textbf{3,300 human-hours} and up to \textbf{5 rounds} of quality assurance, Video-MME-v2 aims to serve as one of the most authoritative video benchmarks. Extensive experiments reveal a substantial gap between current best model Gemini-3-Pro and human experts, and uncover a clear hierarchical bottleneck where errors in visual information aggregation and temporal modeling propagate to limit high-level reasoning. We further find that thinking-based reasoning is highly dependent on textual cues, improving performance with subtitles but sometimes degrading it in purely visual settings. By exposing these limitations, Video-MME-v2 establishes a demanding new testbed for the development of next-generation video MLLMs.