The Year in AI Papers
The year/Independent research

Paper 2511.09146

DoPE: Denoising Rotary Position Embedding

Jing Xiong, Liyang Fan, Hui Shen, Zunhai Su, Min Yang, Lingpeng Kong, Ngai Wong

Published
Nov 2025
Research lab
Independent
Citations
2
GitHub
Not linked

01 In brief

Summary

The paper introduces DOPE (Denoising Rotary Position Embedding), a training-free method to improve length extrapolation in LLMs by mitigating instabilities caused by RoPE.

The authors show via spectral analysis that RoPE's low-frequency components cause spectral amplification, leading to low-rank, over-aligned attention patterns and attention sinks.

DOPE identifies noisy attention heads using truncated matrix entropy and suppresses their positional encoding, optionally reparameterizing attention maps with isotropic Gaussian noise.

Experiments on needle-in-a-haystack and many-shot in-context learning tasks show DOPE improves robustness and performance over baselines like Dynamic NTK, without fine-tuning.

The method selectively denoises heads, preserving extrapolative heads with low-rank structure, and demonstrates that controlling positional information injection per-head is key to robust extrapolation.

02 From the paper

Abstract

Positional encoding is essential for large language models (LLMs) to represent sequence order, yet recent studies show that Rotary Position Embedding (RoPE) can induce massive activation. We investigate the source of these instabilities via a spectral analysis of RoPE, and show that its low-frequency components concentrate structured energy, producing low-rank, over-aligned attention patterns. We theoretically reveal that this low-frequency alignment manifests as activation noise, degrading stability during long-context extrapolation. To mitigate this effect, we introduce Denoising Rotary Position Embedding (DoPE), a training-free method that identifies and suppresses noisy attention heads using truncated matrix entropy, then reparameterizes their attention maps with an isotropic Gaussian distribution. Across a range of settings, DoPE improves length extrapolation performance without fine-tuning, increases robustness to perturbations, and boosts both needle-in-a-haystack and many-shot in-context learning tasks. These results suggest that selective positional encoding is key to robust extrapolation. Our project page is Project: https://The-physical-picture-of-LLMs.github.io