Paper 2508.09834
Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 17
- GitHub
- 406 stars
01 In brief
Summary
This survey systematically reviews efficient architectures for Large Language Models (LLMs), addressing the computational bottlenecks of the standard Transformer, particularly the quadratic complexity of self-attention and the high costs of large Feed-Forward Networks.
It categorizes recent innovations into seven main areas: linear sequence modeling (e.g., linear attention, linear RNNs, state space models, test-time-training RNNs), sparse sequence modeling (static, dynamic, and training-free sparse attention), efficient full attention (IO-aware, grouped, mixture, and quantized attention), sparse mixture-of-experts (routing mechanisms, expert architectures, and conversion from dense models), hybrid architectures (inter-layer and intra-layer), diffusion LLMs (non-autoregressive, bridging with autoregressive, and multimodal extensions), and applications to other modalities (vision, audio, and multimodality).
The survey highlights key design principles, performance benefits, and limitations of each approach, and discusses future directions such as algorithm-system-hardware co-design, adaptive attention, enhanced MoE routing, and efficient models for edge devices and long contexts.
02 From the paper
Abstract
Large Language Models (LLMs) have delivered impressive results in language understanding, generation, reasoning, and pushes the ability boundary of multimodal models. Transformer models, as the foundation of modern LLMs, offer a strong baseline with excellent scaling properties. However, the traditional transformer architecture requires substantial computations and poses significant obstacles for large-scale training and practical deployment. In this survey, we offer a systematic examination of innovative LLM architectures that address the inherent limitations of transformers and boost the efficiency. Starting from language modeling, this survey covers the background and technical details of linear and sparse sequence modeling methods, efficient full attention variants, sparse mixture-of-experts, hybrid model architectures incorporating the above techniques, and emerging diffusion LLMs. Additionally, we discuss applications of these techniques to other modalities and consider their wider implications for developing scalable, resource-aware foundation models. By grouping recent studies into the above category, this survey presents a blueprint of modern efficient LLM architectures, and we hope this could help motivate future research toward more efficient, versatile AI systems.