The Year in AI Papers
The year/Independent research

Paper 2510.19338

Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning

Ling Team, Bin Han, Caizhi Tang, Chen Liang, Donghao Zhang, Fan Yuan, Feng Zhu, Jie Gao, Jingyu Hu, Longfei Li, Meng Li, Mingyang Zhang, et al.

Published
Oct 2025
Research lab
Independent
Citations
9
GitHub
Not linked

01 In brief

Summary

This technical report introduces the Ring-linear model series, comprising Ring-mini-linear-2.0 (16B total, 1.6B active parameters) and Ring-flash-linear-2.0 (104B total, 7.4B active).

Both use a hybrid architecture that combines linear and softmax attention to reduce I/O and computational costs in long-context scenarios, cutting inference cost to 1/10 of a 32B dense model and over 50% versus the original Ring series.

The authors systematically explored the optimal ratio of attention types and developed the LingHe FP8 operator library, improving training efficiency by 50%.

They also achieved stable long-horizon reinforcement learning by systematically aligning training and inference engines, addressing issues like KV cache precision and MoE routing discrepancies.

The models maintain SOTA performance on reasoning benchmarks, with Ring-flash-linear-2.0 achieving 86.51% on AIME'25 and 90.24 on Codeforces Elo.

Continued pre-training from Ling-base-2.0 models restored over 98% of original performance.

Inference optimizations, including fused kernels and speculative decoding, yield over 2.5x prefill throughput and 2x decode throughput versus Ring-2.0 at long contexts.

02 From the paper

Abstract

In this technical report, we present the Ring-linear model series, specifically including Ring-mini-linear-2.0 and Ring-flash-linear-2.0. Ring-mini-linear-2.0 comprises 16B parameters and 957M activations, while Ring-flash-linear-2.0 contains 104B parameters and 6.1B activations. Both models adopt a hybrid architecture that effectively integrates linear attention and softmax attention, significantly reducing I/O and computational overhead in long-context inference scenarios. Compared to a 32 billion parameter dense model, this series reduces inference cost to 1/10, and compared to the original Ring series, the cost is also reduced by over 50%. Furthermore, through systematic exploration of the ratio between different attention mechanisms in the hybrid architecture, we have identified the currently optimal model structure. Additionally, by leveraging our self-developed high-performance FP8 operator library-linghe, overall training efficiency has been improved by 50%. Benefiting from the high alignment between the training and inference engine operators, the models can undergo long-term, stable, and highly efficient optimization during the reinforcement learning phase, consistently maintaining SOTA performance across multiple challenging complex reasoning benchmarks.