Paper 2604.10098
Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation
- Published
- Apr 2026
- Research lab
- Independent
- Citations
- 3
- GitHub
- 138 stars
01 In brief
Summary
This survey is the first comprehensive review of Attention Sink (AS) in Transformers, a phenomenon where disproportionate attention is focused on a small set of uninformative tokens.
The authors synthesize over 210 studies, organizing the field into three key dimensions: Fundamental Utilization (e.g., Sink Token Preservation, Attention Redistribution, Learnable Prefix Tokens, Sink Token Repurposing), Mechanistic Interpretation (e.g., Softmax Limitations & No-Op Theory, Outlier Circuits, Implicit Attention Bias, Geometric Anchoring), and Strategic Mitigation (e.g., Gated Attention Mechanisms, Modified Softmax Functions, Learnable Attention Bias, Pre-training Interventions).
The survey covers AS across various architectures, including classical language models, LLMs, MoE LLMs, MLLMs, ViTs, and Diffusion Transformers.
It highlights applications in efficient inference, hallucination reduction, and robustness, and discusses challenges such as computational overhead and training from scratch.
The authors provide a taxonomy, practical guidelines, and future directions, including efficient AS handling and standardized benchmarks.
A GitHub repository is maintained for ongoing updates.
02 From the paper
Abstract
As the foundational architecture of modern machine learning, Transformers have driven remarkable progress across diverse AI domains. Despite their transformative impact, a persistent challenge across various Transformers is Attention Sink (AS), in which a disproportionate amount of attention is focused on a small subset of specific yet uninformative tokens. AS complicates interpretability, significantly affecting the training and inference dynamics, and exacerbates issues such as hallucinations. In recent years, substantial research has been dedicated to understanding and harnessing AS. However, a comprehensive survey that systematically consolidates AS-related research and offers guidance for future advancements remains lacking. To address this gap, we present the first survey on AS, structured around three key dimensions that define the current research landscape: Fundamental Utilization, Mechanistic Interpretation, and Strategic Mitigation. Our work makes a pivotal contribution by highlighting the key concepts and main trends in the field, guiding researchers through the evolution of AS-related studies. We envision this survey as a valuable resource, empowering researchers to effectively manage AS within the current Transformer paradigm, while simultaneously inspiring innovative advancements for the next generation of Transformers. The paper list of this work is available at https://github.com/ZunhaiSu/Awesome-Attention-Sink.