Paper 2512.24880
mHC: Manifold-Constrained Hyper-Connections
- Published
- Dec 2025
- Research lab
- DeepSeek
- Citations
- 65
- GitHub
- Not linked
01 In brief
Summary
The paper introduces Manifold-Constrained Hyper-Connections (mHC), a framework that addresses training instability and scalability issues in Hyper-Connections (HC) by projecting residual connection matrices onto the Birkhoff polytope (doubly stochastic matrices) using the Sinkhorn-Knopp algorithm.
This restores the identity mapping property, ensuring norm preservation and compositional closure, which prevents signal explosion or vanishing.
mHC also includes infrastructure optimizations: kernel fusion with mixed precision, selective recomputing, and overlapping communication in DualPipe.
Experiments on MoE language models (3B, 9B, 27B) show mHC improves stability and performance, achieving a 0.021 loss reduction over baseline and outperforming HC on most benchmarks, with only 6.7% training overhead at expansion rate n=4.
The method demonstrates superior scalability across compute and token budgets, and reduces the Amax gain magnitude from ~3000 in HC to ~1.6, confirming enhanced propagation stability.
mHC is presented as a general extension of HC, opening avenues for exploring diverse manifold constraints in macro-architecture design.
The work is from DeepSeek-AI and builds on DeepSeek-V3 architecture, using TileLang for kernel implementation and 20 Sinkhorn-Knopp iterations in practice.
The paper includes detailed ablation studies, stability analyses, and system-level benchmarks, highlighting mHC's effectiveness for large-scale training and its potential for future foundational model evolution.
The authors anticipate mHC will contribute to understanding topological architecture design and suggest promising directions for foundational model development.
The framework's theoretical properties (norm preservation, compositional closure, geometric interpretation) are rigorously analyzed, and the infrastructure optimizations are detailed to ensure practical efficiency.
The results validate mHC as a flexible and practical extension of HC…
02 From the paper
Abstract
Recently, studies exemplified by Hyper-Connections (HC) have extended the ubiquitous residual connection paradigm established over the past decade by expanding the residual stream width and diversifying connectivity patterns. While yielding substantial performance gains, this diversification fundamentally compromises the identity mapping property intrinsic to the residual connection, which causes severe training instability and restricted scalability, and additionally incurs notable memory access overhead. To address these challenges, we propose Manifold-Constrained Hyper-Connections (mHC), a general framework that projects the residual connection space of HC onto a specific manifold to restore the identity mapping property, while incorporating rigorous infrastructure optimization to ensure efficiency. Empirical experiments demonstrate that mHC is effective for training at scale, offering tangible performance improvements and superior scalability. We anticipate that mHC, as a flexible and practical extension of HC, will contribute to a deeper understanding of topological architecture design and suggest promising directions for the evolution of foundational models.