Paper 2607.14935
VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding
- Published
- Jul 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- 377 stars
01 In brief
Summary
VideoChat3 is a fully open, efficient, and generalist video multimodal large language model (MLLM) with 4B parameters, designed to address limitations in existing open-source models: limited cross-domain generalization, high computational demands, and incomplete openness.
It introduces two key architectural innovations: the Inflated 3D Vision Transformer (I3D-ViT), which performs chunk-wise spatiotemporal self-attention and temporal pooling to reduce visual tokens by 16x, and Adaptive Frame Resolution for streaming, which adjusts frame resolution based on model state (Silence, Standby, Response) to save computation.
The model is trained on three new datasets: VideoChat3-Academic2M (2.27M instances), VideoChat3-LV116K (116.2K long-video instances), and VideoChat3-OL617K (617K online streaming instances), created via a scalable data synthesis pipeline.
Training uses a four-stage curriculum.
Evaluations show VideoChat3 outperforms comparable open-source models like Qwen3-VL-4B and Molmo2-4B on most benchmarks, including temporal perception, long-video understanding, and temporal grounding, while also excelling in streaming scenarios.
The project fully releases model weights, training code, datasets, and construction pipelines for reproducibility.
02 From the paper
Abstract
Recent advances in video understanding have spanned motion, long video, and streaming interaction, driving this field toward real-world applications. Despite this progress, current open-source models remain limited in several ways. They often struggle to generalize across diverse video types, making them effective only in specific domains. High computational demands further restrict their efficiency and scalability. Moreover, most models are only partially open, with key components such as training code, strategy, or datasets unavailable, which hinders reproducibility and slows community-driven development. To address these issues, we introduce VideoChat3, a fully open, efficient, and generalist video-centric MLLM. VideoChat3 advances video understanding through two complementary designs. For efficiency, we introduce Inflated 3D Vision Transformer (I3D-ViT) and Adaptive Frame Resolution for Streaming Video Perception, which enables efficient spatiotemporal representation and reduces the cost of processing video inputs during training and inference. For effectiveness, we develop a scalable video data synthesis pipeline that curates three diverse, high-quality training datasets: VideoChat3-Academic2M, VideoChat3-LV116K, and VideoChat3-OL617K, covering general, long-form, and streaming video scenarios, improving the model's generalization across domains. By integrating these designs, VideoChat3 achieves a rare balance of broad generalization and computational efficiency. Experiments across general, long-form, and streaming benchmarks demonstrate that VideoChat3 surpasses prior open-source models with equal or larger parameter counts with only 4B parameters and higher efficiency.