Paper 2601.07372
Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
- Published
- Jan 2026
- Research lab
- DeepSeek
- Citations
- 63
- GitHub
- 4.6K stars
01 In brief
Summary
The paper introduces Engram, a conditional memory module that augments Mixture-of-Experts (MoE) large language models with static N-gram lookup tables, providing a complementary sparsity axis to conditional computation.
Engram uses hashed N-gram embeddings with tokenizer compression, multi-head hashing, and context-aware gating to retrieve static knowledge in O(1) time.
The authors formulate a Sparsity Allocation problem and discover a U-shaped scaling law showing optimal performance when 20-25% of the sparse parameter budget is allocated to Engram memory.
Scaling Engram to 27B parameters (Engram-27B) outperforms an iso-parameter and iso-FLOPs MoE-27B baseline across knowledge (MMLU +3.0, CMMLU +4.0), reasoning (BBH +5.0, ARC-Challenge +3.7), and code/math (HumanEval +3.0, MATH +2.4) benchmarks.
Mechanistic analyses (LogitLens, CKA) show Engram relieves early layers from static reconstruction, effectively deepening the network.
Engram also improves long-context retrieval (Multi-Query NIAH: 97.0 vs.
84.2) by freeing attention capacity.
System-wise, deterministic addressing enables prefetching from host memory, with negligible overhead (<3%) when offloading a 100B-parameter table.
02 From the paper
Abstract
While Mixture-of-Experts (MoE) scales capacity via conditional computation, Transformers lack a native primitive for knowledge lookup, forcing them to inefficiently simulate retrieval through computation. To address this, we introduce conditional memory as a complementary sparsity axis, instantiated via Engram, a module that modernizes classic $N$-gram embedding for O(1) lookup. By formulating the Sparsity Allocation problem, we uncover a U-shaped scaling law that optimizes the trade-off between neural computation (MoE) and static memory (Engram). Guided by this law, we scale Engram to 27B parameters, achieving superior performance over a strictly iso-parameter and iso-FLOPs MoE baseline. Most notably, while the memory module is expected to aid knowledge retrieval (e.g., MMLU +3.4; CMMLU +4.0), we observe even larger gains in general reasoning (e.g., BBH +5.0; ARC-Challenge +3.7) and code/math domains~(HumanEval +3.0; MATH +2.4). Mechanistic analyses reveal that Engram relieves the backbone's early layers from static reconstruction, effectively deepening the network for complex reasoning. Furthermore, by delegating local dependencies to lookups, it frees up attention capacity for global context, substantially boosting long-context retrieval (e.g., Multi-Query NIAH: 84.2 to 97.0). Finally, Engram establishes infrastructure-aware efficiency: its deterministic addressing enables runtime prefetching from host memory, incurring negligible overhead. We envision conditional memory as an indispensable modeling primitive for next-generation sparse models.