Paper 2607.02980
Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
- Published
- Jul 2026
- Research lab
- Independent
- Citations
- 1
- GitHub
- 138 stars
01 In brief
Summary
The paper introduces Hierarchical Landmark Sparse (HiLS) Attention, a chunk-wise sparse attention mechanism for large language models that learns chunk selection end-to-end under the language-modeling loss.
HiLS factorizes attention hierarchically: each query attends independently to retrieved chunks and fuses outputs using retrieval scores, which are optimized directly with the LM loss.
This enables native sparse training and avoids the quadratic cost of full attention.
Experiments show HiLS achieves performance comparable to full attention at in-domain lengths, extrapolates over 64x the training context length with 90% retrieval accuracy, and can convert existing full-attention models via lightweight continued pretraining.
At 7B scale, after 50B tokens of continued training, HiLS outperforms full-attention baselines on LongBench and exceeds YaRN-extended models.
Inference is faster: at 512K context, HiLS is 13.5x faster in prefill and 15.7x faster in decode.
The method uses landmark tokens for chunk summaries, a low-rank query calibration module, and HoPE positional encoding.
Ablations show these components are crucial for extrapolation.
HiLS also supports GQA and uses a hardware-efficient kernel that batches adjacent queries to improve Tensor Core utilization.
02 From the paper
Abstract
Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention. Chunk-wise sparse attention offers a promising alternative, but all existing methods fall short of full attention because of their inaccurate chunk selection. We propose Hierarchical Landmark Sparse (HiLS) Attention, a chunk-wise sparse attention mechanism that learns chunk selection end-to-end under the language-modeling (LM) loss. HiLS factorizes attention hierarchically: each query performs attention independently with each retrieved chunk to extract chunk-specific information, and the resulting outputs are fused according to chunk retrieval scores. By incorporating retrieval scores into the forward attention computation, HiLS optimizes them directly with the LM loss, enabling end-to-end retrieval learning and native sparse training. Experimental results show that HiLS-Attention achieves performance comparable to, and in some cases better than, full attention at in-domain context lengths. Meanwhile, HiLS-Attention extrapolates more than $64\times$ the training context length with 90% retrieval accuracy, far beyond full attention. Moreover, existing full-attention models can be converted to HiLS-Attention with lightweight continued pretraining, preserving in-domain performance while acquiring ultra-long-context extrapolation. Together with its sparse KV access and computation, HiLS-Attention breaks the usual efficiency-performance trade-off, enabling long-context LLMs that are both more efficient and more effective on general long-context tasks than their full-attention counterparts.