Paper 2511.20785
LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Calling
- Published
- Nov 2025
- Research lab
- Independent
- Citations
- 51
- GitHub
- 259 stars
01 In brief
Summary
LongVT is an end-to-end agentic framework that enables large multimodal models (LMMs) to reason over long videos by interleaving multimodal Chain-of-Tool-Thought (iMCoTT) with native video cropping tool calls.
It mimics human global-to-local viewing: the model first skims the video, then invokes a crop_video tool to inspect specific temporal windows, and self-corrects based on retrieved evidence.
To support training and evaluation, the authors curate VideoSIAH, a data suite with 247.9K SFT samples, 1.6K RL samples, 15.4K RFT samples, and a 652-question benchmark (VideoSIAH-Eval) with human-in-the-loop validation.
LongVT uses a three-stage training pipeline: cold-start SFT, agentic reinforcement learning with a joint answer-temporal grounding reward, and reinforcement fine-tuning.
Experiments on VideoMME, VideoMMMU, LVBench, and VideoSIAH-Eval show LongVT-7B-RFT achieves state-of-the-art performance among open-source models, with an average score of 47.7, outperforming baselines by up to 6 points on VideoSIAH-Eval.
Ablations confirm the necessity of each stage and the superiority of IoU over Recall rewards.
The framework also reduces inference latency by avoiding hallucination-driven verbose generation.
Code, data, and model weights are publicly available.
02 From the paper
Abstract
Large multimodal models (LMMs) have shown great potential for video reasoning with textual Chain-of-Thought. However, they remain vulnerable to hallucinations, especially when processing long-form videos where evidence is sparse and temporally dispersed. Inspired by how humans comprehend long videos - by first skimming globally and then examining relevant clips for details - we introduce LongVT, an end-to-end agentic framework that enables "Thinking with Long Videos" via interleaved Multimodal Chain-of-Tool-Thought. Specifically, we exploit LMMs' inherent temporal grounding ability as a native video cropping tool to zoom in on a specific video clip and resample finer-grained video frames. This global-to-local reasoning loop continues until answers are grounded in retrieved visual evidence. Given the scarcity of fine-grained question-answering (QA) data for the long video reasoning task, we curate and will release a data suite named VideoSIAH to facilitate both training and evaluation. Specifically, our training dataset consists of 247.9K samples for tool-integrated cold-start supervised fine-tuning, 1.6K samples for agentic reinforcement learning, and 15.4K samples for agentic reinforcement fine-tuning, respectively. Our evaluation benchmark consists of 1,280 QA pairs that are carefully curated through a semi-automatic data pipeline with human-in-the-loop validation. With a meticulously designed three-stage training strategy and extensive empirical validation, LongVT consistently outperforms existing strong baselines across four challenging long-video understanding and reasoning benchmarks. Our codes, data, and model checkpoints are publicly available at https://github.com/EvolvingLMMs-Lab/LongVT .