Paper 2510.27688
Continuous Autoregressive Language Models
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 10
- GitHub
- 814 stars
01 In brief
Summary
This paper introduces Continuous Autoregressive Language Models (CALM), a paradigm shift from discrete next-token prediction to continuous next-vector prediction.
CALM uses a high-fidelity autoencoder to compress K tokens into a single continuous vector, reducing the number of autoregressive steps by a factor of K.
The authors develop a likelihood-free framework including an energy-based generative head for training, a BrierLM metric for evaluation, and algorithms for temperature sampling.
Experiments on the Pile dataset show that CALM achieves comparable performance to strong discrete baselines at significantly lower computational cost, establishing next-vector prediction as a scalable pathway for efficient language models.
The paper also discusses future work on autoencoder design, model architecture, sampling methods, and scaling laws.
02 From the paper
Abstract
The efficiency of large language models (LLMs) is fundamentally limited by their sequential, token-by-token generation process. We argue that overcoming this bottleneck requires a new design axis for LLM scaling: increasing the semantic bandwidth of each generative step. To this end, we introduce Continuous Autoregressive Language Models (CALM), a paradigm shift from discrete next-token prediction to continuous next-vector prediction. CALM uses a high-fidelity autoencoder to compress a chunk of K tokens into a single continuous vector, from which the original tokens can be reconstructed with over 99.9\% accuracy. This allows us to model language as a sequence of continuous vectors instead of discrete tokens, which reduces the number of generative steps by a factor of K. The paradigm shift necessitates a new modeling toolkit; therefore, we develop a comprehensive likelihood-free framework that enables robust training, evaluation, and controllable sampling in the continuous domain. Experiments show that CALM significantly improves the performance-compute trade-off, achieving the performance of strong discrete baselines at a significantly lower computational cost. More importantly, these findings establish next-vector prediction as a powerful and scalable pathway towards ultra-efficient language models. Code: https://github.com/shaochenze/calm. Project: https://shaochenze.github.io/blog/2025/CALM.