Scaling Latent Reasoning via Looped Language Models

The paper introduces Ouro, a family of pre-trained Looped Language Models (LoopLM) that integrate iterative computation in latent space during pre-training, using an entropy-regularized objective for adaptive depth allocation.

Trained on 7.7T tokens, Ouro 1.4B and 2.6B models match or exceed the performance of 4B and 8B standard transformers, achieving 2-3x parameter efficiency.

Controlled experiments show this advantage stems from superior knowledge manipulation, not increased knowledge capacity.

LoopLM also improves safety and faithfulness compared to explicit chain-of-thought.

The paper establishes loop depth as a third scaling axis beyond model size and data, releasing the Ouro model family publicly.

The architecture uses a shared-weight transformer block applied recurrently, with an exit gate for adaptive computation.

Training includes a two-stage process: entropy-regularized pre-training and focused gate training.

The models show strong performance on reasoning benchmarks, with performance peaking at the trained depth of 4 recurrent steps.

KV cache sharing strategies reduce memory overhead by 4x during decoding.

The paper also provides theoretical analysis showing LoopLM can solve graph reachability in O(log D) steps, and scaling laws for total and step-wise loss.

RLVR attempts did not yield gains over SFT checkpoints due to infrastructure limitations with dynamic early-exit mechanisms.

The work demonstrates that iterative latent computation is a promising direction for parameter-efficient scaling in the reasoning era.

Future work will focus on improving performance extrapolation at greater depths and exploring more complex recurrent mechanisms.

The paper includes extensive appendices on prior choice validation, physics of LoopLMs, evaluation settings, and scaling law analysis.

The…