Paper 2607.26760
Metis: Memory Foundation Model
- Published
- Jul 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- 94 stars
01 In brief
Summary
The paper introduces memory foundation models, which internalize memory into the backbone of large foundation models, contrasting with external memory modules like RAG.
The authors formalize native memory via a persistent, dynamically evolving memory state within the model's parameters and native memory procedures that autonomously store and utilize information during forward computation.
They propose Metis, the first prototype, which uses a new architecture with Metis blocks containing local and hyper memory blocks.
Local blocks maintain a dense memory state, while hyper blocks update it via memory attention.
Metis is trained through mid-training on a large-scale synthesized dataset (357,137 primary samples and 609,443 auxiliary samples) using objectives for memory reconstruction, operation, and regularization.
Experiments show Metis outperforms baselines like Temp-LoRA and δ-Mem on memory operation and QA tasks under no-context settings, with Metis-27B achieving the best results.
Ablation studies highlight the importance of adaptive aggregation and query-key normalization.
However, performance degrades on long-term tasks due to information compression, and irrelevant memory can interfere with general capabilities.
The authors release the project and model checkpoints for future research.
02 From the paper
Abstract
Recent advances in AI agents have increasingly internalized native capabilities into their underlying foundation models, giving rise to multimodal foundation models and large reasoning models. However, agent memory is still primarily implemented through external modules, leaving the native memory capability largely unexplored. In this paper, we take a first step toward this direction by introducing memory foundation models, which empower foundation models with native memory capabilities. We formalize native memory from two perspectives: a persistent and dynamically evolving memory state within the backbone, and native memory procedures that autonomously store and utilize information through model computation. We show that native memory offers advantages in architecture, end-to-end optimization, and efficiency. Based on this formulation, we propose Metis, the first prototype of memory foundation models. Metis introduces a new architecture that equips a foundation model with a native memory state, allowing historical information to be compressed into the model and accessed through memory attention. We construct large-scale memory-specific training data and introduce multiple optimization objectives to acquire these native memory procedures through mid-training. The online memory maintenance of Metis is gradient-free, and the memory update requires only a forward pass. At inference time, all learned model weights remain frozen, while the native memory states are autonomously transformed through standard forward computation. Through extensive experiments, we show that Metis exhibits native memory capabilities and further provide a detailed analysis of its strengths, limitations, and behaviors. To facilitate future research on memory foundation models, we release our project and model checkpoints.