Paper 2606.09079
FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention
- Published
- Jun 2026
- Research lab
- Independent
- Citations
- 2
- GitHub
- 101 stars
01 In brief
Summary
This report introduces FlashMemory-DeepSeek-V4 (FM-DS-V4), which uses Lookahead Sparse Attention (LSA) to reduce GPU memory for ultra-long context LLM serving.
LSA adds a Neural Memory Indexer to DeepSeek-V4-Flash, predicting and fetching only critical KV chunks every 64 steps, instead of keeping the full KV cache.
The indexer is trained independently as a dual-encoder, avoiding backbone loading.
On LongBench-v2, LongMemEval, and RULER, FM-DS-V4 uses only 13.5% of the baseline GPU KV cache (86.5% reduction) while improving average accuracy by +0.6%.
At 1M context, per-token compute drops to 0.30x, KV cache shrinks by 90% (3.73 to 0.37 GB), and throughput and concurrency increase 2.8x and 2.7x, respectively.
Limitations include context-independent overhead, a severe accuracy drop on MRCR (76% to 48%), and a length generalization ceiling of 2x training context.
The project is suspended due to organizational changes.
02 From the paper
Abstract
Conventional LLMs keep the full KV cache loaded during decoding, causing a severe GPU memory bottleneck for ultra-long context serving. In this report, we propose \textbf{Lookahead Sparse Attention (LSA)}, a novel inference paradigm powered by a Neural Memory Indexer built upon the DeepSeek-V4 architecture. Rather than passively attending to all historical tokens, LSA proactively predicts future context demands and preserves only the query-critical KV chunks in the GPU memory. Crucially, we instantiate this architecture via a \textbf{backbone-free decoupled training} strategy. By formulating the indexer as a standard dual-encoder architecture, we train it independently using standard retrieval training frameworks without ever loading the massive backbone model into GPU memory. We demonstrate that this ``less is more'' paradigm significantly maximizes serving efficiency while acting as an effective attention denoiser in tasks that rely on long-term global memory. Across primary long-context evaluation suites (e.g., LongBench-v2, LongMemEval, and RULER), \texttt{FM-DS-V4} compresses the average physical KV cache footprint down to merely 13.5\% of the full-context baseline, while consistently preserving or slightly elevating downstream accuracy (+0.6\% absolute margin on average). At 1M context, per-decode-token compute drops to 0.30$\times$ of the baseline and GPU KV cache shrinks by 90\% (3.73$\to$0.37 GB), translating into \textbf{2.8$\times$ aggregate throughput and 2.7$\times$ concurrency gains} in PD-disaggregated serving on 8$\times$H20 GPUs.