Paper 2601.14724
HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding
- Published
- Jan 2026
- Research lab
- Independent
- Citations
- 22
- GitHub
- 94 stars
01 In brief
Summary
HERMES is a training-free framework for efficient streaming video understanding in Multimodal Large Language Models (MLLMs).
It conceptualizes the KV cache as a hierarchical memory system based on a mechanistic attention analysis, where shallow layers act as sensory memory with recency bias, deep layers as long-term memory focusing on frame-level anchor tokens, and middle layers as working memory balancing both.
HERMES comprises three components: hierarchical KV cache management, cross-layer memory smoothing, and position re-indexing.
During inference, it reuses a compact KV cache without auxiliary computations at query time, achieving 10x faster time-to-first-token (TTFT) compared to prior state-of-the-art methods.
Experiments show that HERMES reduces video tokens by up to 68% while maintaining or improving accuracy, with gains up to 11.4% on streaming benchmarks.
It maintains constant GPU memory usage and low latency across varying input lengths.
Evaluations on StreamingBench, OVO-Bench, RVS, MVBench, Egoschema, and VideoMME demonstrate its effectiveness across multiple MLLMs including LLaVA-OV and Qwen2.5-VL.
02 From the paper
Abstract
Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated significant improvement in offline video understanding. However, extending these capabilities to streaming video inputs, remains challenging, as existing models struggle to simultaneously maintain stable understanding performance, real-time responses, and low GPU memory overhead. To address this challenge, we propose HERMES, a novel training-free architecture for real-time and accurate understanding of video streams. Based on a mechanistic attention investigation, we conceptualize KV cache as a hierarchical memory framework that encapsulates video information across multiple granularities. During inference, HERMES reuses a compact KV cache, enabling efficient streaming understanding under resource constraints. Notably, HERMES requires no auxiliary computations upon the arrival of user queries, thereby guaranteeing real-time responses for continuous video stream interactions, which achieves 10$\times$ faster TTFT compared to prior SOTA. Even when reducing video tokens by up to 68% compared with uniform sampling, HERMES achieves superior or comparable accuracy across all benchmarks, with up to 11.4% gains on streaming datasets.