Paper 2510.22115
Every Activation Boosted: Scaling General Reasoner to 1 Trillion Open Language Foundation
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 30
- GitHub
- Not linked
01 In brief
Summary
Ling 2.0 is a series of reasoning-oriented language foundation models from Inclusion AI, scaling from 16B to 1T parameters under a unified high-sparsity Mixture-of-Experts (MoE) paradigm.
The series includes three non-thinking instruct models: Ling-mini-2.0 (16B total, 1.4B activated), Ling-flash-2.0 (103B total, 6.1B activated), and Ling-1T (1T total, 51B activated).
All models use 256 routed experts with 8 active plus 1 shared expert per token (~3.5% activation), achieving up to 7× active-compute efficiency over dense counterparts.
Key innovations include Ling Scaling Laws for hyperparameter and architecture design, a high-sparsity MoE with Multi-Token Prediction (MTP), reasoning-oriented data composition with progressive increase of reasoning data from 32% to 46% during pre-training, mid-training CoT pre-activation, the Warmup-Stable-Merge (WSM) scheduler replacing LR decay with checkpoint merging, Decoupled Fine-Tuning (DFT) with Evolutionary Chain-of-Thought (Evo-CoT) reinforcement learning, Sentence-Level Policy Optimization (LPO), Group Arena Reward (GAR) for human preference alignment, and full-scale FP8 training with fine-grained heterogeneous pipelines.
Ling-1T establishes a new Pareto frontier of reasoning accuracy versus computational efficiency, demonstrating that sparse activation aligned with reasoning objectives enables scalable and efficient intelligence.
02 From the paper
Abstract
We introduce Ling 2.0, a series reasoning-oriented language foundation built upon the principle that every activation boosts reasoning capability. Designed to scale from tens of billions to one trillion parameters under a unified Mixture-of-Experts (MoE) paradigm, Ling 2.0 emphasizes high sparsity, cross-scale consistency, and efficiency guided by empirical scaling laws. The series includes three non-thinking (instruct) models - Ling-mini-2.0, Ling-flash-2.0, and Ling-1T - ranging from 16B to 1T total parameters and achieving up to 7-fold active-compute efficiency compared with dense counterparts. Ling 2.0 integrates coordinated innovations across model architecture, pre-training, post-training, and infrastructure: a high-sparsity MoE with MTP for efficient reasoning, reasoning-oriented data and mid-training CoT activation, reinforcement-based fine-tuning (DFT, Evo-CoT), and full-scale FP8 training with fine-grained heterogeneous pipelines. At the trillion scale, Ling-1T establishes a new Pareto frontier of reasoning accuracy versus computational efficiency, demonstrating that sparse activation, when properly aligned with reasoning objectives, enables scalable and efficient intelligence. Collectively, Ling 2.0 provides a coherent, open, and efficient foundation for advancing future reasoning and thinking models, including the Ring series built upon the same base.