Paper 2605.15128
MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory
- Published
- May 2026
- Research lab
- Independent
- Citations
- 6
- GitHub
- 48 stars
01 In brief
Summary
MemEye is a visual-centric evaluation framework for multimodal agent memory, addressing the gap in benchmarks that test whether agents preserve visual evidence for later reasoning.
It introduces a two-axis taxonomy: visual evidence granularity (X-axis, from scene-level X1 to pixel-level X4) and memory reasoning depth (Y-axis, from atomic retrieval Y1 to evolutionary synthesis Y3).
The benchmark includes 371 mirrored multiple-choice and open-ended questions across 8 life-scenario tasks, with validation gates ensuring answerability, shortcut resistance, visual necessity, and reasoning structure.
Evaluation of 13 memory methods across 4 VLM backbones shows current systems struggle with fine-grained visual details and state changes over time.
Key findings reveal a trade-off: text-based memory manages state transitions but loses visual details, while image-based memory preserves details but struggles with temporal validity.
The framework highlights the need for evidence routing, temporal tracking, and detail extraction in future memory systems.
02 From the paper
Abstract
Long-term agent memory is increasingly multimodal, yet existing evaluations rarely test whether agents preserve the visual evidence needed for later reasoning. In prior work, many visually grounded questions can be answered using only captions or textual traces, allowing answers to be inferred without preserving the fine-grained visual evidence. Meanwhile, harder cases that require reasoning over changing visual states are largely absent. Therefore, we introduce MemEye, a framework that evaluates memory capabilities from two dimensions: one measures the granularity of decisive visual evidence (from scene-level to pixel-level evidence), and the other measures how retrieved evidence must be used (from single evidence to evolutionary synthesis). Under this framework, we construct a new benchmark across 8 life-scenario tasks, with ablation-driven validation gates for assessing answerability, shortcut resistance, visual necessity, and reasoning structure. By evaluating 13 memory methods across 4 VLM backbones, we show that current architectures still struggle to preserve fine-grained visual details and reason about state changes over time. Our findings show that long-term multimodal memory depends on evidence routing, temporal tracking, and detail extraction.