Paper 2607.27919
Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory
- Published
- Jul 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- 13 stars
01 In brief
Summary
This paper introduces Memory Decoder at Scale, scaling parametric long-term memory models up to 6.9B parameters and pretraining them on 300B tokens.
To handle the computational bottleneck of constructing kNN distributions over 207B tokens, the authors develop a distributed Faiss pipeline using embedding compression, index sharding, and parallel search, along with sparse kNN distribution storage and distributed streaming.
Experiments show that scaling memory is more parameter-efficient than scaling the base model alone.
A 6.9B general memory paired with Pythia-410M achieves an average score of 37.34 across 17 benchmarks, surpassing Pythia-12B (37.24) with 39% fewer total parameters.
Domain memories (1.7B) improve average scores on biology, law, and finance benchmarks by over 9 points across Qwen3 backbones (0.6B to 14B).
Memory also transfers across vocabularies with only 20% of the standard training budget, improving OLMo-2-7B and OLMo-3-7B domain averages by 4.26 and 7.77 points.
Further analyses show memory remains effective with few-shot prompting, benefits knowledge-intensive tasks most, and improves with memory size and training budget.
02 From the paper
Abstract
Decoder-only language models entangle long-term memory and reasoning in a single parameter set, making it difficult to scale memory capacity independently. Memory Decoder introduces a parametric long-term memory module but only studies it at a relatively small scale. In this work, we present Memory Decoder at Scale, scaling memory models up to 6.9B parameters and pretraining them on 300B tokens. At this data scale, the combined cost of indexing and search makes a standard Faiss pipeline infeasible. We address this bottleneck with a distributed pipeline for Faiss indexing and retrieval, together with sparse, batch-wise loading of kNN distributions. Across model scales, we find that allocating more parameters to memory yields a better parameter-performance tradeoff than scaling the base model alone. On 17 benchmarks, pairing a 6.9B general memory with Pythia-410M raises its average score from 29.86 to 37.34, surpassing Pythia-12B (37.24) with 39% fewer total parameters. For Qwen3 Base models ranging from 0.6B to 14B, 1.7B domain memories improve the average score across the three domains by more than 9 points at every scale. Overall, our results demonstrate that independently scaling pretrained memory offers a more parameter efficient path to improving language model performance.