Paper 2603.29616
Video-Oasis: Rethinking Evaluation of Video Understanding
- Published
- Mar 2026
- Research lab
- Independent
- Citations
- 1
- GitHub
- 34 stars
01 In brief
Summary
The paper introduces Video-Oasis, a diagnostic suite for auditing video understanding benchmarks.
It finds that 55% of samples in 14 existing benchmarks can be solved without visual or temporal context, and that state-of-the-art Video-LLMs perform only marginally above random chance on the remaining video-native challenges.
The suite includes visual-dependency tests (Blind, Audio, Summary), temporal-dependency tests (Center-Frame, Frame Shuffling, Bag-of-Frames), and ambiguity verification (Consistency, Redundancy, Sensitivity).
After filtering shortcuts, 11,033 QA pairs remain, categorized into five challenge types: Fine-Grained Perception, Spatial World Understanding, Temporal Dynamics & Tracking, Causality & Logical Reasoning, and Global Narrative.
Evaluations show that Gemini-2.5-Pro achieves the highest overall accuracy (46.7%), while most open-source models score below 37%.
Ablation studies reveal that temporal grounding improves performance, adaptive reasoning (choosing when to think) is as impactful as model scale, and that SFT and RLVR offer complementary strengths.
The work provides practical guidelines for constructing rigorous video benchmarks and designing future models.
02 From the paper
Abstract
The inherent complexity of video understanding makes it difficult to determine whether Video-LLM benchmark performance stems from visual perception, linguistic reasoning, or knowledge priors. While many benchmarks have emerged to assess high-level reasoning, shared criteria for evaluating video understanding remain largely overlooked. Instead of introducing yet another benchmark, we take a step back to re-examine the criteria for evaluating video understanding. In this work, we introduce Video-Oasis, a sustainable diagnostic suite for systematically auditing existing video understanding benchmarks. This audit reveals that 55\% of existing benchmark samples are solvable without visual input or temporal context. After filtering these shortcuts, the remaining video-native challenges expose a substantial capability gap: state-of-the-art models perform only marginally above random guessing. Building on these findings, we use the distilled challenges as a testbed to investigate which algorithmic design choices contribute to robust video understanding. We hope our work provides a practical foundation for constructing rigorous video benchmarks and evaluating future Video-LLMs. Code is available at https://github.com/sejong-rcv/Video-Oasis.