The Year in AI Papers
The year/Independent research

Paper 2512.24617

Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space

Xingwei Qu, Shaowen Wang, Zihao Huang, Kai Hua, Fan Yin, Rui-Jie Zhu, Jundong Zhou, Qiyang Min, Zihao Wang, Yizhi Li, Tianyu Zhang, He Xing, et al.

Published
Dec 2025
Research lab
Independent
Citations
12
GitHub
Not linked

01 In brief

Summary

The paper introduces Dynamic Large Concept Models (DLCM), a hierarchical language modeling framework that learns variable-length semantic concepts from latent representations and performs reasoning in a compressed concept space, shifting computation from tokens to concepts.

DLCM uses a four-stage pipeline: encoding, dynamic segmentation via learned boundary detection, concept-level reasoning with a high-capacity transformer, and token-level decoding via causal cross-attention.

It introduces a compression-aware scaling law and a decoupled μP parametrization for stable training.

At a compression ratio of 4, DLCM reallocates about one-third of inference compute to a larger reasoning backbone, achieving a +2.69% average improvement across 12 zero-shot benchmarks under matched FLOPs, with the largest gains on reasoning-dominant tasks.

The paper also details a concept replication strategy for efficient cross-attention, a global parser for adaptive compression, and ablation studies showing the stability of rule-based boundary detection over learned predictors.

02 From the paper

Abstract

Large Language Models (LLMs) apply uniform computation to all tokens, despite language exhibiting highly non-uniform information density. This token-uniform regime wastes capacity on locally predictable spans while under-allocating computation to semantically critical transitions. We propose $\textbf{Dynamic Large Concept Models (DLCM)}$, a hierarchical language modeling framework that learns semantic boundaries from latent representations and shifts computation from tokens to a compressed concept space where reasoning is more efficient. DLCM discovers variable-length concepts end-to-end without relying on predefined linguistic units. Hierarchical compression fundamentally changes scaling behavior. We introduce the first $\textbf{compression-aware scaling law}$, which disentangles token-level capacity, concept-level reasoning capacity, and compression ratio, enabling principled compute allocation under fixed FLOPs. To stably train this heterogeneous architecture, we further develop a $\textbf{decoupled $μ$P parametrization}$ that supports zero-shot hyperparameter transfer across widths and compression regimes. At a practical setting ($R=4$, corresponding to an average of four tokens per concept), DLCM reallocates roughly one-third of inference compute into a higher-capacity reasoning backbone, achieving a $\textbf{+2.69$\%$ average improvement}$ across 12 zero-shot benchmarks under matched inference FLOPs.