The Year in AI Papers
The year/Topics/Long context and attention

Research collection

Long context and attention

Extending or serving long context: sparse and linear attention architectures, KV-cache compression, memory-augmented generation, and long-context training.

Papers
26
Research labs
4
Official code
16

1–26 of 26 papers in this collection

01

Independent research

Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory

This paper introduces Memory Decoder at Scale, scaling parametric long-term memory models up to 6.9B parameters and pretraining them on 300B tokens. To handle the computational bottleneck of constructing kNN distributions over 207B tokens, the authors develop a distributed Faiss pipeline using embedding compression, index sharding, and parallel search,…

Rubin Wei, Jiaqi Cao, Jiarui Wang, Junming Zhang, et al.
Published
Jul 2026
Citations
0
Code
13 stars
Read summary Original
02

Independent research

Metis: Memory Foundation Model

The paper introduces memory foundation models, which internalize memory into the backbone of large foundation models, contrasting with external memory modules like RAG. The authors formalize native memory via a persistent, dynamically evolving memory state within the model's parameters and native memory procedures that autonomously store and utilize…

Zeyu Zhang, Ziliang Guo, Yihang Sun, Xichong Zhang, et al.
Published
Jul 2026
Citations
0
Code
94 stars
Read summary Original
03

Independent research

Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling

The paper introduces Hierarchical Landmark Sparse (HiLS) Attention, a chunk-wise sparse attention mechanism for large language models that learns chunk selection end-to-end under the language-modeling loss. HiLS factorizes attention hierarchically: each query attends independently to retrieved chunks and fuses outputs using retrieval scores, which are…

Xiang Hu, Xinyu Wei, Hao Gu, Minshen Zhang, et al.
Published
Jul 2026
Citations
1
Code
138 stars
Read summary Original
04

MiniMax

MiniMax Sparse Attention

MiniMax Sparse Attention (MSA) is a blockwise sparse attention mechanism built on Grouped Query Attention (GQA) to address the quadratic cost of softmax attention in ultra-long contexts. A lightweight Index Branch scores key-value blocks and selects a Top-k subset per GQA group, while the Main Branch computes exact block-sparse attention over selected…

Xunhao Lai, Weiqi Xu, Yufeng Yang, Qiaorui Chen, et al.
Published
Jun 2026
Citations
7
Code
442 stars
Read summary Original
05

Independent research

FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention

This report introduces FlashMemory-DeepSeek-V4 (FM-DS-V4), which uses Lookahead Sparse Attention (LSA) to reduce GPU memory for ultra-long context LLM serving. LSA adds a Neural Memory Indexer to DeepSeek-V4-Flash, predicting and fetching only critical KV chunks every 64 steps, instead of keeping the full KV cache. The indexer is trained independently as a…

Yan Wang, Qifan Zhang, Jiachen Yu, Tian Liang, et al.
Published
Jun 2026
Citations
2
Code
101 stars
Read summary Original
06

Independent research

KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks

The paper introduces KVarN, a calibration-free KV-cache quantization method that mitigates error accumulation during autoregressive decoding in long-horizon reasoning tasks. It identifies that quantization errors accumulate across timesteps, driven primarily by incorrect token scales, which cause outlier errors that disproportionately degrade end-to-end…

Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, et al.
Published
Jun 2026
Citations
3
Code
452 stars
Read summary Original
07

arXiv.org

ACC: Compiling Agent Trajectories for Long-Context Training

The paper introduces Agent Context Compilation (ACC), a method that converts multi-turn agent trajectories (from search, software engineering, and SQL agents) into long-context QA training pairs. Standard agent SFT masks tool responses, creating a supervision blind spot where evidence scattered across turns is unused. ACC compiles tool responses and…

Qisheng Su, Zhen Fang, Shiting Huang, Yu Zeng, et al.
Published
May 2026
Citations
0
Code
Not linked
Read summary Original
08

Together AI

OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization

OSCAR is an INT2 KV-cache quantization method that uses attention-aware covariance structures, estimated offline, to derive fixed rotations and clipping thresholds, aligning quantization with downstream attention. It composes rotations as R = U · H_Had · P_br, where U is the eigenbasis of query-aware (Q^T Q) or score-aware (V^T S^T S V) covariance, H_Had…

Zhongzhu Zhou, Donglin Zhuang, Jisen Li, Ziyan Chen, et al.
Published
May 2026
Citations
2
Code
555 stars
Read summary Original
09

arXiv.org

Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps

RTPurbo is a method that transforms full-attention large language models into highly sparse models with minimal adaptation, requiring only a few hundred training steps. It is based on three observations: only a small subset of attention heads (retrieval heads) truly need full long-context processing; long-range retrieval is governed by a low-dimensional…

Yanke Zhou, Yiduo Li, Hanlin Tang, Maohua Li, et al.
Published
May 2026
Citations
1
Code
Not linked
Read summary Original
10

arXiv.org

Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context

This paper presents a systematic study of long-context continued pre-training (LongPT) for large vision-language models (LVLMs), extending a 7B model from 32K to 128K context. The authors find that long-document VQA is substantially more effective than OCR transcription for training. Key findings include: balanced sequence-length distributions outperform…

Zhaowei Wang, Lishu Luo, Haodong Duan, Weiwei Liu, et al.
Published
May 2026
Citations
0
Code
Not linked
Read summary Original
11

arXiv.org

$δ$-mem: Efficient Online Memory for Large Language Models

δ-mem is a lightweight memory mechanism that augments a frozen full-attention LLM with a compact online state of associative memory, updated via delta-rule learning. It compresses past information into a fixed-size state matrix (e.g., 8×8) and uses its readout to generate low-rank corrections to the backbone's attention computation, avoiding full…

Jingdi Lei, Di Zhang, Junxian Li, Weida Wang, et al.
Published
May 2026
Citations
3
Code
253 stars
Read summary Original
12

NVIDIA

TriAttention: Efficient Long Reasoning with Trigonometric KV Compression

TriAttention is a KV cache compression method for long-context LLM reasoning. It exploits the observation that pre-RoPE Q/K vectors are highly concentrated around fixed non-zero centers, which remain stable across positions and contexts. This concentration causes attention to follow predictable distance preferences, which can be modeled as a trigonometric…

Weian Mao, Xi Lin, Wei Huang, Yuxin Xie, et al.
Published
Apr 2026
Citations
14
Code
838 stars
Read summary Original
13

Z.ai / GLM

IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse

IndexCache is a method to accelerate sparse attention in large language models by exploiting cross-layer redundancy in token selection. DeepSeek Sparse Attention (DSA) uses a lightning indexer at each layer to select top-k tokens, but this indexer still has O(L^2) complexity. IndexCache partitions layers into a small set of Full (F) layers that run their…

Yushi Bai, Qian Dong, Ting Jiang, Xin Lv, et al.
Published
Mar 2026
Citations
16
Code
131 stars
Read summary Original
14

arXiv.org

Query-focused and Memory-aware Reranker for Long Context Processing

The paper introduces QRRanker, a listwise reranking framework that trains LLMs to score passage-query relevance using attention scores from selected Query-focused Retrieval (QR) heads, avoiding generation-based scoring and Likert-scale supervision. It uses a contrastive loss on continuous attention-derived scores, enabling training on arbitrary retrieval…

Yuqing Li, Jiangnan Li, Mo Yu, Guoxuan Ding, et al.
Published
Feb 2026
Citations
5
Code
Not linked
Read summary Original
15

arXiv.org

FASA: Frequency-aware Sparse Attention

FASA is a training-free framework for query-aware token eviction in LLMs, addressing KV cache memory bottlenecks. It exploits functional sparsity in RoPE frequency chunks (FCs), where a small subset of 'dominant' FCs shows high contextual agreement with full attention heads. FASA uses a two-stage process: Token Importance Prediction (TIP) selects critical…

Yifei Wang, Yueqi Wang, Zhenrui Yue, Huimin Zeng, et al.
Published
Feb 2026
Citations
5
Code
Not linked
Read summary Original
16

arXiv.org

Recursive Language Models

The paper introduces Recursive Language Models (RLMs), an inference-time framework that treats arbitrarily long prompts as part of an external environment, allowing an LLM to programmatically examine, decompose, and recursively call itself over prompt snippets. RLMs use a REPL environment where the prompt is stored as a variable, and the model writes code…

Alex L. Zhang, Tim Kraska, Omar Khattab
Published
Dec 2025
Citations
64
Code
5.4K stars
Read summary Original
17

arXiv.org

QwenLong-L1.5: Post-Training Recipe for Long-Context Reasoning and Memory Management

QwenLong-L1.5 is a long-context reasoning model built on Qwen3-30B-A3B-Thinking, introduced by Alibaba's Tongyi Lab. It achieves performance comparable to GPT-5 and Gemini-2.5-Pro on long-context benchmarks, surpassing its baseline by 9.90 points on average. The model's post-training recipe includes three key innovations: (1) a data synthesis pipeline that…

Weizhou Shen, Ziyi Yang, Chenliang Li, Zhiyuan Lu, et al.
Published
Dec 2025
Citations
23
Code
550 stars
Read summary Original
18

arXiv.org

Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs

The paper introduces RoPE++, an extension of Rotary Position Embeddings (RoPE) for large language models. Standard RoPE discards the imaginary component of the complex-valued attention score, losing phase information. RoPE++ re-incorporates this imaginary part as a new group of attention heads, computed in parallel with real attention. Two configurations…

Xiaoran Liu, Yuerong Song, Zhigeng Liu, Zengfeng Huang, et al.
Published
Dec 2025
Citations
1
Code
33 stars
Read summary Original
19

arXiv.org

DoPE: Denoising Rotary Position Embedding

The paper introduces DOPE (Denoising Rotary Position Embedding), a training-free method to improve length extrapolation in LLMs by mitigating instabilities caused by RoPE. The authors show via spectral analysis that RoPE's low-frequency components cause spectral amplification, leading to low-rank, over-aligned attention patterns and attention sinks. DOPE…

Jing Xiong, Liyang Fan, Hui Shen, Zunhai Su, et al.
Published
Nov 2025
Citations
2
Code
Not linked
Read summary Original
20

arXiv.org

LightMem: Lightweight and Efficient Memory-Augmented Generation

LightMem is a lightweight and efficient memory-augmented generation system for Large Language Models (LLMs), presented at ICLR 2026. It addresses inefficiencies in existing memory systems by drawing inspiration from the Atkinson-Shiffrin human memory model, organizing memory into three stages: sensory memory (pre-compression and topic segmentation),…

Jizhan Fang, Xinle Deng, Haoming Xu, Ziyan Jiang, et al.
Published
Oct 2025
Citations
111
Code
1.1K stars
Read summary Original
21

arXiv.org

Efficient Long-context Language Model Training by Core Attention Disaggregation

This paper introduces core attention disaggregation (CAD), a technique to improve long-context LLM training by separating the parameter-free softmax(QK^T)V computation (core attention, CA) from other model components and scheduling it on a dedicated pool of resources. CAD leverages two key properties: statelessness (CA has no trainable parameters) and…

Yonghao Zhuang, Junda Chen, Bo Pang, Yi Gu, et al.
Published
Oct 2025
Citations
3
Code
Not linked
Read summary Original
22

Volume 1

Glyph: Scaling Context Windows via Visual-Text Compression

Glyph is a framework that scales context windows by rendering long texts into images and processing them with vision-language models (VLMs), achieving 3-4x token compression while maintaining accuracy comparable to leading LLMs like Qwen3-8B. It consists of three stages: continual pre-training on rendered long-text data, an LLM-driven genetic search to…

Jiale Cheng, Yusen Liu, Xinyu Zhang, Yulin Fei, et al.
Published
Oct 2025
Citations
42
Code
595 stars
Read summary Original
23

International Conference on Automated Software Engineering

LongCodeZip: Compress Long Context for Code Language Models

LongCodeZip is a training-free, model-agnostic, plug-and-play framework for compressing long code contexts for code LLMs. It uses a dual-stage strategy: coarse-grained compression selects function-level chunks ranked by conditional perplexity relative to the instruction, and fine-grained compression segments retained functions into blocks via…

Yuling Shi, Yichun Qian, Hongyu Zhang, Beijun Shen, et al.
Published
Oct 2025
Citations
39
Code
163 stars
Read summary Original
24

arXiv.org

MemMamba: Rethinking Memory Patterns in State Space Model

MemMamba is a novel architecture for long-sequence modeling that addresses the memory decay problem in state space models (SSMs) like Mamba. The paper first analyzes Mamba's memory mechanism, showing that its state update leads to exponential decay of early information both within layers (horizontal) and across layers (vertical), quantified by new metrics:…

Youjin Wang, Yangjingyi Chen, Jiahao Yan, Jiaxuan Lu, et al.
Published
Sep 2025
Citations
2
Code
Not linked
Read summary Original
25

arXiv.org

PRELUDE: A Benchmark Designed to Require Global Comprehension and Reasoning over Long Contexts

PRELUDE is a benchmark for evaluating long-context understanding in LLMs by asking whether a character's prequel story is consistent with the canonical narrative of a book. The task requires global comprehension and deep reasoning, as 88% of instances need evidence from multiple parts of the narrative. The dataset includes 795 instances across 13 books,…

Mo Yu, Tsz Ting Chung, Chulun Zhou, Tong Li, et al.
Published
Aug 2025
Citations
6
Code
Not linked
Read summary Original
26

Together AI

When Does Divide and Conquer Work for Long Context LLM? A Noise Decomposition Framework

This paper introduces a theoretical framework to analyze when divide-and-conquer (D&C) strategies, which split long inputs into chunks processed by multiple agents, are effective for long-context LLMs. The framework decomposes system fidelity loss into three components: task noise (cross-chunk dependencies), model noise (performance degradation with input…

Zhen Xu, Shang Zhu, Jue Wang, Junlin Wang, et al.
Published
Jun 2025
Citations
7
Code
Not linked
Read summary Original