Paper 2508.09736
Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 64
- GitHub
- 1.4K stars
01 In brief
Summary
The paper introduces M3-Agent, a multimodal agent framework with long-term memory that processes real-time video and audio to build episodic and semantic memories, organized in an entity-centric multimodal graph.
It uses reinforcement learning for multi-turn reasoning and iterative memory retrieval.
The authors also present M3-Bench, a long-video question answering benchmark with 100 robot-perspective videos (M3-Bench-robot) and 920 web videos (M3-Bench-web), containing 1,276 and 3,214 QA pairs respectively, covering five question types.
Experiments show M3-Agent outperforms baselines, including a prompting agent using Gemini-1.5-Pro and GPT-4o, by 6.7%, 7.7%, and 5.3% on M3-Bench-robot, M3-Bench-web, and VideoMME-long.
Ablations show semantic memory is critical (removing it drops accuracy by 17.1%, 19.2%, 13.1%), and RL training improves accuracy by 10.0%, 8.0%, 9.3% on the respective benchmarks.
The memorization model is trained via imitation learning on synthetic data, and the control model via DAPO.
The work advances multimodal agents toward human-like long-term memory.
02 From the paper
Abstract
We introduce M3-Agent, a novel multimodal agent framework equipped with long-term memory. Like humans, M3-Agent can process real-time visual and auditory inputs to build and update episodic and semantic memories, gradually accumulating world knowledge. Its memory is organized in an entity-centric, multimodal manner, enabling deeper and more consistent understanding of the environment. Given an instruction, M3-Agent autonomously performs multi-turn reasoning and retrieves relevant memories to complete tasks. To evaluate memory effectiveness and memory-based reasoning in multimodal agents, we develop M3-Bench, a long-video question answering benchmark comprising 100 newly recorded robot-perspective videos (M3-Bench-robot) and 920 diverse web-sourced videos (M3-Bench-web). We annotate QA pairs designed to test capabilities essential for agent applications, such as person understanding, general knowledge extraction, and cross-modal reasoning. Experimental results show that M3-Agent, trained via reinforcement learning, outperforms the strongest baseline, a prompting agent using Gemini-1.5-pro and GPT-4o, achieving 6.7%, 7.7%, and 5.3% higher accuracy on M3-Bench-robot, M3-Bench-web and VideoMME-long, respectively. Our work advances multimodal agents toward more human-like long-term memory and provides insights for their practical design. Model, code and data are available at https://github.com/bytedance-seed/m3-agent.