The Year in AI Papers
The year/Topics/Video understanding

Research collection

Video understanding

Comprehending existing video: video question answering, temporal grounding, video reasoning, and video understanding benchmarks.

Papers
29
Research labs
1
Official code
25

1–29 of 29 papers in this collection

01

Independent research

TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs

TimeLens2 introduces a generalist video temporal grounding model that predicts variable-cardinality sets of evidence intervals across diverse video lengths, domains, query forms, and viewpoints. It addresses two structural mismatches: unreliable long-video supervision and optimization that lacks interval-level geometry. The TimeLens2-93K dataset pipeline…

Yuhan Zhu, Changlian Ma, Xiangyu Zeng, Xinhao Li, et al.
Published
Jul 2026
Citations
0
Code
117 stars
Read summary Original
02

Independent research

VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding

VideoChat3 is a fully open, efficient, and generalist video multimodal large language model (MLLM) with 4B parameters, designed to address limitations in existing open-source models: limited cross-domain generalization, high computational demands, and incomplete openness. It introduces two key architectural innovations: the Inflated 3D Vision Transformer…

Xinhao Li, Yuhan Zhu, Xiangyu Zeng, Yuhao Dong, et al.
Published
Jul 2026
Citations
0
Code
377 stars
Read summary Original
03

Independent research

Video Generation Models are General-Purpose Vision Learners

The paper introduces GenCeption, a general-purpose vision model that uses large-scale text-to-video generation as a pre-training paradigm. By repurposing a pre-trained video diffusion backbone (WAN 2.1) into a feed-forward model, GenCeption performs multiple vision tasks—depth, surface normal, camera pose, segmentation, and 3D keypoint estimation—steered…

Letian Wang, Chuhan Zhang, Rishabh Kabra, Jasper Uijlings, et al.
Published
Jul 2026
Citations
1
Code
Not linked
Read summary Original
04

arXiv.org

YoCausal: How Far is Video Generation from World Model? A Causality Perspective

YoCausal is a two-level benchmark for evaluating causal cognition in video diffusion models (VDMs), inspired by the Violation of Expectation paradigm. It uses temporally reversed real-world videos as counterfactual samples, avoiding synthetic data and the sim-to-real gap. Level 1 introduces the Reverse Surprise Index (RSI), measuring arrow-of-time…

You-Zhe Xie, Yu-Hsuan Li, Jie-Ying Lee, Kaipeng Zhang, et al.
Published
May 2026
Citations
1
Code
36 stars
Read summary Original
05

arXiv.org

Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality?

This paper introduces Grounded Personality Reasoning (GPR) and the MM-OCEAN benchmark to evaluate whether Multimodal Large Language Models (MLLMs) perceive personality through behavioral evidence or merely prejudge via superficial patterns. The benchmark includes 1,104 videos and 5,320 cue-grounding MCQs, built via a multi-agent human-collaborative…

Caixin Kang, Tianyu Yan, Sitong Gong, Mingfang Zhang, et al.
Published
May 2026
Citations
0
Code
11 stars
Read summary Original
06

arXiv.org

When Vision Speaks for Sound

The paper identifies a 'Clever Hans effect' in video-capable multimodal LLMs, where models appear to understand audio but actually rely on visual cues to hallucinate or infer sounds without verifying the audio stream. This is demonstrated across open-source and closed-source models. To systematically study this, the authors introduce THUD, a diagnostic…

Xiaofei Wen, Wenjie Jacky Mo, Xingyu Fu, Rui Cai, et al.
Published
May 2026
Citations
2
Code
98 stars
Read summary Original
07

arXiv.org

HumanNet: Scaling Human-centric Video Learning to One Million Hours

HumanNet is a one-million-hour human-centric video corpus designed to scale embodied learning by capturing how humans interact with the physical world. It includes both first-person and third-person perspectives, covering fine-grained activities, human-object interactions, tool use, and long-horizon behaviors across diverse environments. The dataset…

Yufan Deng, Daquan Zhou
Published
May 2026
Citations
6
Code
281 stars
Read summary Original
08

arXiv.org

ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning

The paper introduces ReVSI, a benchmark for evaluating vision-language models' (VLMs) 3D spatial reasoning, addressing validity issues in the existing VSI-Bench. Two key pitfalls are identified: annotation-to-video ground-truth drift (errors from point-cloud-based annotations) and scene-observability mismatch (questions unanswerable under sparse frame…

Yiming Zhang, Jiacheng Chen, Jiaqi Tan, Yongsen Mao, et al.
Published
Apr 2026
Citations
7
Code
83 stars
Read summary Original
09

Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers

Video Analysis and Generation via a Semantic Progress Function

This paper introduces the Semantic Progress Function (SPF), a one-dimensional representation that captures how semantic meaning evolves over time in video sequences. The SPF is computed by measuring pairwise semantic distances between frames using a pretrained embedder (SigLIP) and fitting a smooth curve via regularized least squares. Departures from a…

Gal Metzer, Sagi Polaczek, Ali Mahdavi-Amiri, Raja Giryes, et al.
Published
Apr 2026
Citations
1
Code
38 stars
Read summary Original
10

arXiv.org

Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding

Video-MME-v2 is a new benchmark for evaluating video multimodal large language models (MLLMs), designed to address the gap between inflated leaderboard scores and real-world capabilities. It introduces a progressive three-level hierarchy (visual information aggregation, temporal dynamics modeling, and complex reasoning) and a group-based evaluation…

Chaoyou Fu, Haozhi Yuan, Yuhao Dong, Yi-Fan Zhang, et al.
Published
Apr 2026
Citations
23
Code
369 stars
Read summary Original
11

arXiv.org

A Simple Baseline for Streaming Video Understanding

The paper introduces SIMPLESTREAM, a minimal baseline for streaming video understanding that feeds only the most recent N frames to an off-the-shelf VLM, without any memory, retrieval, or compression mechanisms. Evaluated on OVO-Bench and StreamingBench, SIMPLESTREAM with Qwen3-VL-8B and 4 frames achieves 67.7% average accuracy on OVO-Bench and 80.59% on…

Yujiao Shen, Shulin Tian, Jingkang Yang, Ziwei Liu
Published
Apr 2026
Citations
13
Code
157 stars
Read summary Original
12

arXiv.org

Video-Oasis: Rethinking Evaluation of Video Understanding

The paper introduces Video-Oasis, a diagnostic suite for auditing video understanding benchmarks. It finds that 55% of samples in 14 existing benchmarks can be solved without visual or temporal context, and that state-of-the-art Video-LLMs perform only marginally above random chance on the remaining video-native challenges. The suite includes…

Geuntaek Lim, Sungjune Park, Jaeyun Lee, Inwoong Lee, et al.
Published
Mar 2026
Citations
1
Code
34 stars
Read summary Original
13

Independent research

Demystifying Video Reasoning

This paper challenges the Chain-of-Frames (CoF) hypothesis for reasoning in diffusion-based video models, proposing instead that reasoning primarily emerges along the diffusion denoising steps, termed Chain-of-Steps (CoS). Through qualitative analysis and noise perturbation experiments, the authors show that models explore multiple candidate solutions in…

Ruisi Wang, Zhongang Cai, Fanyi Pu, Junxiang Xu, et al.
Published
Mar 2026
Citations
6
Code
47 stars
Read summary Original
14

arXiv.org

Towards Universal Video MLLMs with Attribute-Structured and Quality-Verified Instructions

The paper introduces ASID-1M, an open-source dataset of one million attribute-structured audiovisual instruction annotations, along with ASID-Verify, a multi-stage data curation pipeline for automatic annotation, verification, and refinement, and ASID-Captioner, a video understanding model trained via supervised fine-tuning on this dataset. The pipeline…

Yunheng Li, Hengrui Zhang, Meng-Hao Guo, Wenzhao Gao, et al.
Published
Feb 2026
Citations
5
Code
68 stars
Read summary Original
15

arXiv.org

VidVec: Unlocking Video MLLM Embeddings for Video-Text Retrieval

VidVec introduces a method to leverage generative Multimodal Large Language Models (MLLMs) for video–text retrieval without visual fine-tuning. The authors show that intermediate layers of off-the-shelf video MLLMs (e.g., VideoLLaMA3-7B) encode substantial retrieval-relevant information, outperforming final-layer embeddings in zero-shot settings. They…

Issar Tzachor, Dvir Samuel, Rami Ben-Ari
Published
Feb 2026
Citations
3
Code
Not linked
Read summary Original
16

arXiv.org

Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition

This paper addresses object-driven shortcuts in Zero-Shot Compositional Action Recognition (ZS-CAR), where models predict verbs by relying on object class labels rather than temporal evidence. The authors identify two root causes: sparse and skewed compositional supervision, and asymmetric learning difficulty (objects are easier to recognize than verbs).…

Geo Ahn, Inwoong Lee, Taeoh Kim, Minho Shim, et al.
Published
Jan 2026
Citations
0
Code
9 stars
Read summary Original
17

Annual Meeting of the Association for Computational Linguistics

HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding

HERMES is a training-free framework for efficient streaming video understanding in Multimodal Large Language Models (MLLMs). It conceptualizes the KV cache as a hierarchical memory system based on a mechanistic attention analysis, where shallow layers act as sensory memory with recency bias, deep layers as long-term memory focusing on frame-level anchor…

Haowei Zhang, Shudong Yang, Jinlan Fu, See-Kiong Ng, et al.
Published
Jan 2026
Citations
22
Code
94 stars
Read summary Original
18

arXiv.org

Taming Hallucinations: Boosting MLLMs' Video Understanding via Counterfactual Video Generation

The paper addresses visual ungrounded hallucinations in Multimodal Large Language Models (MLLMs), which over-rely on language priors when processing counterfactual videos that defy common sense. To mitigate this, the authors introduce DualityForge, a framework using diffusion-based controllable video editing to transform real-world videos into…

Zhe Huang, Hao Wen, Aiming Hao, Bingze Song, et al.
Published
Dec 2025
Citations
6
Code
55 stars
Read summary Original
19

Volume 1

LongVideoAgent: Multi-Agent Reasoning with Long Videos

LongVideoAgent is a multi-agent framework for long-video question answering. A master LLM coordinates a grounding agent to localize question-relevant segments and a vision agent to extract targeted visual observations. The master agent plans with a step limit and is trained with reinforcement learning (GRPO) to encourage concise, correct, and efficient…

Runtao Liu, Ziyi Liu, Jiaqi Tang, Yue Ma, et al.
Published
Dec 2025
Citations
21
Code
126 stars
Read summary Original
20

arXiv.org

MMGR: Multi-Modal Generative Reasoning

The paper introduces MMGR (Multi-Modal Generative Reasoning), a benchmark suite to evaluate the reasoning capabilities of video and image generation models across five core abilities: Physical, Logical, 3D Spatial, 2D Spatial, and Temporal reasoning. It comprises three domains: Abstract Reasoning (Maze, Sudoku, ARC-AGI, Math), Embodied Navigation (four…

Zefan Cai, Haoyi Qiu, Tianyi Ma, Haozhe Zhao, et al.
Published
Dec 2025
Citations
9
Code
Not linked
Read summary Original
21

Independent research

VideoASMR-Bench: Can AI-Generated ASMR Videos Fool VLMs and Humans?

VideoASMR-Bench is a new benchmark for evaluating AI-generated ASMR videos, focusing on fine-grained audio-visual perception and sensory immersion. It includes 1,500 real ASMR videos from social media and 2,235 synthetic videos from nine video generation models (VGMs) under four settings. The benchmark introduces an adversarial evaluation framework where…

Jiaqi Wang, Weijia Wu, Yi Zhan, Rui Zhao, et al.
Published
Dec 2025
Citations
2
Code
23 stars
Read summary Original
22

arXiv.org

LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Calling

LongVT is an end-to-end agentic framework that enables large multimodal models (LMMs) to reason over long videos by interleaving multimodal Chain-of-Tool-Thought (iMCoTT) with native video cropping tool calls. It mimics human global-to-local viewing: the model first skims the video, then invokes a crop_video tool to inspect specific temporal windows, and…

Zuhao Yang, Sudong Wang, Kaichen Zhang, Keming Wu, et al.
Published
Nov 2025
Citations
51
Code
259 stars
Read summary Original
23

arXiv.org

V-ReasonBench: Toward Unified Reasoning Benchmark Suite for Video Generation Models

V-ReasonBench is a benchmark for evaluating reasoning in generative video models under the Chain-of-Frame paradigm, where the final frame represents the model's answer. It covers four reasoning dimensions: structured problem-solving (arithmetic, code execution, Sudoku, Tic-Tac-Toe), spatial cognition (shape fitting, visual symmetry, color connection),…

Yang Luo, Xuanlei Zhao, Baijiong Lin, Lingting Zhu, et al.
Published
Nov 2025
Citations
17
Code
36 stars
Read summary Original
24

arXiv.org

Reasoning via Video: The First Evaluation of Video Models' Reasoning Abilities through Maze-Solving Tasks

This paper introduces VR-Bench, a benchmark for evaluating the reasoning abilities of video generation models through maze-solving tasks. It comprises 7,920 procedurally generated videos across five maze types (Regular, Irregular, 3D, Trapfield, Sokoban) with varying difficulty and textures. The authors propose a 'reasoning via video' paradigm, where…

Cheng Yang, Haiyuan Wan, Yiran Peng, Xin Cheng, et al.
Published
Nov 2025
Citations
15
Code
2 stars
Read summary Original
25

arXiv.org

Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence

Open-o3-Video is a non-agent framework that integrates explicit spatio-temporal evidence into video reasoning by highlighting key timestamps, objects, and bounding boxes. The authors construct two datasets, STGR-CoT-30k and STGR-RL-36k, combining existing temporal and spatial grounding resources with 5.9k newly annotated spatio-temporal samples. They adopt…

Jiahao Meng, Xiangtai Li, Haochen Wang, Yue Tan, et al.
Published
Oct 2025
Citations
43
Code
159 stars
Read summary Original
26

arXiv.org

StreamingVLM: Real-Time Understanding for Infinite Video Streams

StreamingVLM is a framework for real-time understanding of infinite video streams, addressing the limitations of full attention (quadratic cost, poor long-video performance) and sliding window methods (coherence loss or high latency). It maintains a compact KV cache with attention sinks, a short vision window, and a long text window, using contiguous RoPE…

Ruyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He, et al.
Published
Oct 2025
Citations
73
Code
1.1K stars
Read summary Original
27

Google DeepMind

Video models are zero-shot learners and reasoners

This paper investigates whether generative video models, like Veo 3, can act as zero-shot learners and reasoners for general-purpose vision tasks, similar to how LLMs transformed NLP. The authors analyzed 18,384 generated videos across 62 qualitative and 7 quantitative tasks, finding that Veo 3 can solve tasks it wasn't explicitly trained for, including…

Thaddäus Wiedemer, Yuxuan Li, Paul Vicol, Shixiang Shane Gu, et al.
Published
Sep 2025
Citations
194
Code
Not linked
Read summary Original
28

arXiv.org

ELV-Halluc: Benchmarking Semantic Aggregation Hallucinations in Long Video Understanding

The paper introduces ELV-Halluc, the first benchmark for evaluating Semantic Aggregation Hallucination (SAH) in long videos. SAH occurs when models correctly perceive frame-level semantics but misattribute them across events, a problem that intensifies with semantic complexity. The benchmark uses event-by-event videos (average 672.4 seconds) and…

Hao Lu, Jiahao Wang, Yaolun Zhang, Ruohui Wang, et al.
Published
Aug 2025
Citations
10
Code
11 stars
Read summary Original
29

arXiv.org

Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory

The paper introduces M3-Agent, a multimodal agent framework with long-term memory that processes real-time video and audio to build episodic and semantic memories, organized in an entity-centric multimodal graph. It uses reinforcement learning for multi-turn reasoning and iterative memory retrieval. The authors also present M3-Bench, a long-video question…

Lin Long, Yichen He, Wentao Ye, Yiyuan Pan, et al.
Published
Aug 2025
Citations
64
Code
1.4K stars
Read summary Original