Paper 2607.16051
Loop the Loopies!
- Published
- Jul 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- Not linked
01 In brief
Summary
The paper introduces the Loopie series, two Mixture-of-Experts (MoE) language models: Loopie-20B-A2B and Loopie-6B-A0.6B, which use a novel 'layer-loop' recurrence pattern where each layer is applied twice before moving to the next, unlike prior 'model-loop' approaches.
The key contribution is the Loopie Recipe, a compute-matched scaling method that halves stored layers, doubles the microbatch size, and reinvests efficiency gains into model capacity, allowing looped models to outperform vanilla Transformers under the same pre-training compute budget.
Extensive ablations and a scaling ladder (0.15B to 1B baselines) show consistent advantages.
Post-training includes a novel Supervised Pre-Training (SPT) stage on 2T tokens, which improves both reasoning and general metrics without catastrophic forgetting, followed by reinforcement learning using GSPO with DAPO techniques.
Loopie-20B-A2B achieves frontier-level reasoning, scoring 92.09% on AIME 2024 and 94.21% on AMC, while using only 3.5T pre-training tokens compared to 25T for comparable models.
The models are trained in two pre-training stages: 2.28T tokens on Nemotron-CC-v2-HQ, then 1.26T tokens of high-quality annealing data.
02 From the paper
Abstract
We present the Loopie series, consisting of two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a 6B-parameter model with 0.6B active parameters. Looped Transformers have long faced a challenge: given an N times increase in pre-training compute, increasing the parameter count by a factor of N usually outperforms looping a model N times. Loopie addresses this challenge. Extensive ablation studies, including comparisons with a vanilla 30B-A3B model, show that Loopie substantially outperforms vanilla Transformer baselines trained with the same compute budget. With a novel post-training method, Loopie develops strong reasoning abilities and achieves frontier-level reasoning performance.