The Year in AI Papers
The year/Independent research

Paper 2605.20613

HRM-Text: Efficient Pretraining Beyond Scaling

Guan Wang, Changling Liu, Chenyu Wang, Cai Zhou, Yuhao Sun, Yifei Wu, Shuai Zhen, Luca Scimeca, Yasin Abbasi Yadkori

Published
May 2026
Research lab
Independent
Citations
3
GitHub
1.8K stars

01 In brief

Summary

HRM-Text introduces a Hierarchical Recurrent Model (HRM) that decouples computation into slow strategic and fast execution layers, inspired by biological multi-timescale processing.

Trained from scratch on only 40B unique tokens (60B total with repetition) and a $1,500 budget, a 1B-parameter model achieves 60.7% on MMLU, 81.9% on ARC-C, 82.2% on DROP, 84.5% on GSM8K, and 56.2% on MATH.

This performance is competitive with 2–7B parameter open models while using 96–432x less compute and 100–900x fewer tokens.

Key innovations include MagicNorm for stable recurrence, warmup deep credit assignment, and a task-completion objective with PrefixLM masking that trains only on instruction-response pairs.

Ablations show each component contributes to gains.

The model also demonstrates greater effective depth than standard transformers.

Contamination analysis shows no significant benchmark inflation.

Training took 46 hours on 16 H100 GPUs, using Adam-atan2, EMA, and FSDP2.

The work aims to democratize pretraining by drastically reducing compute requirements.

02 From the paper

Abstract

The current pretraining paradigm for large language models relies on massive compute and internet-scale raw text, creating a significant barrier to foundational research. In contrast, biological systems demonstrate highly sample-efficient learning through multi-timescale processing, such as the functional organization of the frontoparietal loop. Taking this as inspiration, we introduce HRM-Text, which replaces standard Transformers with a Hierarchical Recurrent Model (HRM) that decouples computation into slow-evolving strategic and fast-evolving execution layers. To stabilize this deep recurrence for language modeling, we introduce MagicNorm and warmup deep credit assignment. Furthermore, instead of standard raw-text pretraining, we train exclusively on instruction-response pairs using a task-completion objective and PrefixLM masking. Serving as an empirical existence proof of efficient pretraining, a 1B-parameter HRM-Text model trained from scratch on only 40 billion unique tokens and $1,500 budget achieves 60.7% on MMLU, 81.9% on ARC-C, 82.2% on DROP, 84.5% on GSM8K, and 56.2% on MATH. Despite utilizing roughly 100-900x fewer training tokens and 96-432x less estimated compute than standard baselines, HRM-Text performs competitively with 2-7B parameter open models. These results demonstrate that co-designing architectures and objectives can radically reduce the compute-to-performance ratio, making pretraining from scratch accessible to the broader research community.