DeepSeek
mHC: Manifold-Constrained Hyper-Connections
The paper introduces Manifold-Constrained Hyper-Connections (mHC), a framework that addresses training instability and scalability issues in Hyper-Connections (HC) by projecting residual connection matrices onto the Birkhoff polytope (doubly stochastic matrices) using the Sinkhorn-Knopp algorithm. This restores the identity mapping property, ensuring norm…
Zhenda Xie, Yixuan Wei, Huanqi Cao, Chenggang Zhao, et al.- Published
- Dec 2025
- Upvotes
- 331
- Citations
- 65