Paper 2511.08577
Think-at-Hard: Selective Latent Iterations to Improve Reasoning Language Models
- Published
- Nov 2025
- Research lab
- Independent
- Citations
- 18
- GitHub
- 78 stars
01 In brief
Summary
The paper introduces Think-at-Hard (TaH), a looped transformer that selectively applies latent iterations to improve reasoning in small language models.
The authors identify a 'latent overthinking' problem where always iterating can flip correct predictions into errors.
An oracle policy that iterates only on mispredicted tokens boosts accuracy by up to 7.3%, motivating TaH.
TaH uses a neural decider to trigger iterations only on hard tokens, duo-causal attention for cross-depth information flow with full parallelism, and depth-aware LoRA adapters for deeper iterations.
A two-stage training scheme aligns the backbone and decider with a static oracle policy.
Experiments on nine benchmarks show TaH outperforms always-iterate baselines by 3.8-4.4% while skipping iterations on 93% of tokens, and exceeds single-iteration Qwen3 baselines by 3.0-3.8%.
With less than 3% extra parameters, gains increase to 5.3-6.2% and 6.1-6.8%, respectively.
TaH also reduces FLOPs and memory access compared to always-iterate methods, achieving higher accuracy with lower overhead.
02 From the paper
Abstract
Improving the reasoning abilities of Large Language Models (LLMs), especially under parameter constraints, is crucial for real-world applications. Looped transformers address this by performing multiple latent iterations to refine each token beyond a single forward pass. However, we identify a latent overthinking phenomenon: most token predictions are already correct after the first pass, but are sometimes revised into errors in later iterations. We ask whether selectively skipping latent iterations can improve accuracy, and reveal significant potential with an oracle iteration policy that boosts performance by up to 7.3%. Motivated by this, we propose Think-at-Hard (TaH), a looped transformer optimized for selective iteration. TaH employs a lightweight neural decider to trigger latent iteration, only at tokens likely to be incorrect after the standard forward pass. During latent iterations, depth-aware Low-Rank Adaptation (LoRA) modules shift the objective from general next-token prediction to focused hard-token refinement. A duo-causal attention mechanism extends attention from the token sequence dimension to an additional iteration depth dimension, enabling cross-iteration information flow with full sequential parallelism. Experiments on nine benchmarks show consistent gains across math, QA, and coding tasks. With identical parameter counts, TaH outperforms always-iterate baselines by 3.8-4.4% while skipping iterations on 93% of tokens, and exceeds single-iteration Qwen3 baselines by 3.0-3.8%. When allowing <3% more parameters from LoRA and decider, the gains further increase to 5.3-6.2% and 6.1-6.8%, respectively. Our code is available at https://github.com/thu-nics/TaH.