The Year in AI Papers
The year/Independent research

Paper 2512.07525

Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs

Xiaoran Liu, Yuerong Song, Zhigeng Liu, Zengfeng Huang, Qipeng Guo, Zhaoxiang Liu, Shiguo Lian, Ziwei He, Xipeng Qiu

Published
Dec 2025
Research lab
Independent
Citations
1
GitHub
33 stars

01 In brief

Summary

The paper introduces RoPE++, an extension of Rotary Position Embeddings (RoPE) for large language models.

Standard RoPE discards the imaginary component of the complex-valued attention score, losing phase information.

RoPE++ re-incorporates this imaginary part as a new group of attention heads, computed in parallel with real attention.

Two configurations are proposed: RoPE++_EH (equal head number, halved KV cache and QKV parameters) and RoPE++_EC (equal cache size, doubled attention heads).

Theoretical analysis shows imaginary attention has a sine-integral characteristic curve, which decays more slowly than the real part's cosine-integral curve, enabling better capture of long-context dependencies.

Experiments at 376M, 776M, and 1.5B scales, pre-trained on 50B tokens, show RoPE++ outperforms vanilla RoPE and other position embeddings (FoPE, Pythia, ALiBi) on short-context tasks, and achieves significant gains on long-context benchmarks (RULER, BABILong) up to 64k context.

RoPE++_EH matches RoPE's performance with half the cache, while RoPE++_EC improves performance at the same cache cost.

Noise perturbation experiments confirm imaginary attention is more critical for long-context modeling.

RoPE++ also combines with interpolation methods like YaRN and Linear PI.

02 From the paper

Abstract

Rotary Position Embeddings (RoPE) have become a standard for encoding sequence order in Large Language Models (LLMs) by applying rotations to query and key vectors in the complex plane. Standard implementations, however, utilize only the real component of the complex-valued dot product for attention score calculation. This simplification discards the imaginary component, which contains valuable phase information, leading to a potential loss of relational details crucial for modeling long-context dependencies. In this paper, we propose an extension that re-incorporates this discarded imaginary component. Our method leverages the full complex-valued representation to create a dual-component attention score. We theoretically and empirically demonstrate that this approach enhances the modeling of long-context dependencies by preserving more positional information. Furthermore, evaluations on a suite of long-context language modeling benchmarks show that our method consistently improves performance over the standard RoPE, with the benefits becoming more significant as context length increases. The code is available at https://github.com/OpenMOSS/rope_pp.