Paper 2510.09608
StreamingVLM: Real-Time Understanding for Infinite Video Streams
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 73
- GitHub
- 1.1K stars
01 In brief
Summary
StreamingVLM is a framework for real-time understanding of infinite video streams, addressing the limitations of full attention (quadratic cost, poor long-video performance) and sliding window methods (coherence loss or high latency).
It maintains a compact KV cache with attention sinks, a short vision window, and a long text window, using contiguous RoPE to keep positions in-distribution.
Training uses full attention on short, overlapped video chunks to mimic inference-time attention patterns.
The authors built Inf-Streams-Train (over 4,000 hours of sports commentary) and Inf-Streams-Eval (videos averaging 2.12 hours).
Fine-tuned from Qwen-2.5-VL-7B-Instruct, StreamingVLM achieves a 66.18% win rate against GPT-4o mini on Inf-Streams-Eval, maintains real-time performance at up to 8 FPS on a single NVIDIA H100, and improves VQA benchmarks (LongVideoBench +4.30, OVOBench Realtime +5.96) without VQA-specific fine-tuning.
Ablations show the importance of contiguous RoPE, text eviction, a 16-second vision window, and the overlapped SFT strategy.
02 From the paper
Abstract
Vision-language models (VLMs) could power real-time assistants and autonomous agents, but they face a critical challenge: understanding near-infinite video streams without escalating latency and memory usage. Processing entire videos with full attention leads to quadratic computational costs and poor performance on long videos. Meanwhile, simple sliding window methods are also flawed, as they either break coherence or suffer from high latency due to redundant recomputation. In this paper, we introduce StreamingVLM, a model designed for real-time, stable understanding of infinite visual input. Our approach is a unified framework that aligns training with streaming inference. During inference, we maintain a compact KV cache by reusing states of attention sinks, a short window of recent vision tokens, and a long window of recent text tokens. This streaming ability is instilled via a simple supervised fine-tuning (SFT) strategy that applies full attention on short, overlapped video chunks, which effectively mimics the inference-time attention pattern without training on prohibitively long contexts. For evaluation, we build Inf-Streams-Eval, a new benchmark with videos averaging over two hours that requires dense, per-second alignment between frames and text. On Inf-Streams-Eval, StreamingVLM achieves a 66.18% win rate against GPT-4O mini and maintains stable, real-time performance at up to 8 FPS on a single NVIDIA H100. Notably, our SFT strategy also enhances general VQA abilities without any VQA-specific fine-tuning, improving performance on LongVideoBench by +4.30 and OVOBench Realtime by +5.96. Code is available at https://github.com/mit-han-lab/streaming-vlm.