Paper 2604.02317
A Simple Baseline for Streaming Video Understanding
- Published
- Apr 2026
- Research lab
- Independent
- Citations
- 13
- GitHub
- 157 stars
01 In brief
Summary
The paper introduces SIMPLESTREAM, a minimal baseline for streaming video understanding that feeds only the most recent N frames to an off-the-shelf VLM, without any memory, retrieval, or compression mechanisms.
Evaluated on OVO-Bench and StreamingBench, SIMPLESTREAM with Qwen3-VL-8B and 4 frames achieves 67.7% average accuracy on OVO-Bench and 80.59% on StreamingBench, surpassing all 13 compared offline and online baselines, including HERMES (59.2% on OVO-Bench).
It also maintains the lowest peak GPU memory and competitive latency.
Controlled ablations show that longer context is not uniformly beneficial: optimal window size varies with model scale and backbone family, and adding historical context (e.g., via Visual-RAG) improves memory-oriented tasks (EPM, ASI) but degrades real-time perception (e.g., OJR, OCR, ACR), revealing a perception-memory trade-off.
The authors argue that future streaming benchmarks should separate perception from memory and require strong recency baselines before claiming gains from added complexity.
02 From the paper
Abstract
Recent streaming video understanding methods increasingly rely on complex memory mechanisms to handle long video streams. We challenge this trend with a simple finding: a sliding-window baseline that feeds only the most recent N frames to an off-the-shelf VLM already matches or surpasses published streaming models. We formalize this baseline as SimpleStream and evaluate it against 13 major offline and online video LLM baselines on OVO-Bench and StreamingBench. Despite its simplicity, SimpleStream delivers consistently strong performance. With only 4 recent frames, it reaches 67.7% average accuracy on OVO-Bench and 80.59% on StreamingBench. Controlled ablations further show that the value of longer context is backbone-dependent rather than uniformly increasing with model scale, and reveal a consistent perception-memory trade-off: adding more historical context can improve recall, but often weakens real-time perception. This suggests that stronger memory, retrieval, or compression modules should not be taken as evidence of progress unless they clearly outperform SimpleStream under the same protocol. We therefore argue that future streaming benchmarks should separate recent-scene perception from long-range memory, so that performance improvements from added complexity can be evaluated more clearly.