Paper 2510.13554
Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 25
- GitHub
- Not linked
01 In brief
Summary
This paper investigates how attention dynamics in LLMs can reveal reasoning patterns and improve reinforcement learning (RL) credit assignment.
The authors analyze attention heads, classifying them as locally or globally focused.
Locally focused heads show a sawtooth pattern near the diagonal, indicating phrasal chunks, while globally focused heads highlight anchor tokens with broad downstream influence.
They introduce two metrics: Windowed Average Attention Distance (WAAD) and Future Attention Influence (FAI).
Their joint analysis reveals a 'preplan-and-anchor' rhythm: WAAD peaks (long-range consultation) precede or coincide with high-FAI anchor tokens that organize subsequent reasoning.
Based on these insights, they propose three RL strategies that amplify token-level advantages for preplan tokens, anchor tokens, or their coupling.
Experiments on Countdown, CrossThink-QA, and math benchmarks (AIME, AMC, MATH500, OlympiadBench) with Qwen3 and Llama models show consistent performance gains over GRPO and other baselines, with the coupled rhythm credit strategy performing best.
Ablations confirm the importance of targeting top-ranked tokens and using both positive and negative advantage scaling.
02 From the paper
Abstract
The reasoning pattern of Large language models (LLMs) remains opaque, and reinforcement learning (RL) typically applies uniform credit across an entire generation, blurring the distinction between pivotal and routine steps. This work positions attention as a privileged substrate that renders the internal logic of LLMs legible, not merely as a byproduct of computation, but as a mechanistic blueprint of reasoning itself. We first distinguish attention heads between locally and globally focused information processing and reveal that locally focused heads produce a sawtooth pattern near the diagonal indicating phrasal chunks, while globally focused heads expose tokens that exert broad downstream influence over future tokens. We formalize these with two metrics: 1) Windowed Average Attention Distance, which measures the extent of backward attention within a clipped window; 2) Future Attention Influence, which quantifies a token's global importance as the average attention it receives from subsequent tokens. Taken together, these signals reveal a recurring preplan-and-anchor mechanism, where the model first performs a long-range contextual reference to generate an introductory token, which is immediately followed by or coincides with a semantic anchor token that organizes subsequent reasoning. Leveraging these insights, we introduce three novel RL strategies that dynamically perform targeted credit assignment to critical nodes (preplan tokens, anchor tokens, and their temporal coupling) and show consistent performance gains across various reasoning tasks. By aligning optimization with the model's intrinsic reasoning rhythm, we aim to transform opaque optimization into an actionable structure-aware process, hoping to offer a potential step toward more transparent and effective optimization of LLM reasoning.