Paper 2510.26692
Kimi Linear: An Expressive, Efficient Attention Architecture
- Published
- Oct 2025
- Research lab
- Moonshot AI
- Citations
- 116
- GitHub
- 1.6K stars
01 In brief
Summary
Kimi Linear is a hybrid linear attention architecture that outperforms full attention under fair comparisons across short-context, long-context, and reinforcement learning (RL) scaling regimes.
Its core, Kimi Delta Attention (KDA), extends Gated DeltaNet with a finer-grained, channel-wise gating mechanism, improving memory control and enabling efficient chunkwise parallelization via a specialized Diagonal-Plus-Low-Rank (DPLR) transition matrix.
The architecture interleaves KDA layers with full attention (MLA) layers in a 3:1 ratio, using NoPE for MLA layers.
Pretrained with 3B activated and 48B total parameters on 1.4T tokens, Kimi Linear outperforms full MLA and hybrid GDN baselines on most benchmarks, including MMLU-Pro (51.0 vs.
47.2) and RULER (84.3 vs.
81.3), while reducing KV cache usage by up to 75% and achieving up to 6× decoding throughput at 1M context.
Scaling law experiments show ~1.16× computational efficiency over MLA.
The final checkpoint, trained on 5.7T tokens, achieves 94.8 on RULER at 1M context.
The authors release KDA kernels, vLLM integration, and model checkpoints.
02 From the paper
Abstract
We introduce Kimi Linear, a hybrid linear attention architecture that, for the first time, outperforms full attention under fair comparisons across various scenarios -- including short-context, long-context, and reinforcement learning (RL) scaling regimes. At its core lies Kimi Delta Attention (KDA), an expressive linear attention module that extends Gated DeltaNet with a finer-grained gating mechanism, enabling more effective use of limited finite-state RNN memory. Our bespoke chunkwise algorithm achieves high hardware efficiency through a specialized variant of the Diagonal-Plus-Low-Rank (DPLR) transition matrices, which substantially reduces computation compared to the general DPLR formulation while remaining more consistent with the classical delta rule. We pretrain a Kimi Linear model with 3B activated parameters and 48B total parameters, based on a layerwise hybrid of KDA and Multi-Head Latent Attention (MLA). Our experiments show that with an identical training recipe, Kimi Linear outperforms full MLA with a sizeable margin across all evaluated tasks, while reducing KV cache usage by up to 75% and achieving up to 6 times decoding throughput for a 1M context. These results demonstrate that Kimi Linear can be a drop-in replacement for full attention architectures with superior performance and efficiency, including tasks with longer input and output lengths. To support further research, we open-source the KDA kernel and vLLM implementations, and release the pre-trained and instruction-tuned model checkpoints.