Paper 2607.14530
xHC: Expanded Hyper-Connections
- Published
- Jul 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- 60 stars
01 In brief
Summary
The paper introduces xHC (Expanded Hyper-Connections), a method to scale residual-stream expansion in Transformer models beyond the previous limit of N=4 streams.
Existing Hyper-Connections (HC) methods, particularly Manifold-Constrained HC (mHC), show diminishing returns and rising costs when scaling N due to two bottlenecks: insufficient write-back information for many streams and cubic cost of residual-mixing generation.
xHC addresses these with temporal feature augmentation (multi-scale causal convolutions) to enrich write-back signals and a sparse residual-stream architecture that updates only k=4 of N=16 streams while keeping dense read access.
Experiments on 18B and 28B MoE models show xHC improves average downstream scores by 4.0 points over mHC at 18B (48.8 vs 44.8) with only 4.1% extra training FLOPs.
Scaling-law experiments show vanilla and mHC require 1.50x and 1.19x the compute of xHC to reach the same loss.
The paper also introduces xHC-Flash, which reduces per-sublayer memory traffic from 73.5C to 40C, comparable to mHC's 34C at N=4, while retaining most performance gains.
xHC remains effective with the Muon optimizer.
02 From the paper
Abstract
Hyper-Connections (HC) expand the residual stream of Transformers into $N$ parallel streams, providing a form of memory scaling beyond model width and depth. Manifold-Constrained HC (mHC) stabilizes this formulation at scale. The large gains from $N{=}1$ to $N{=}4$ suggest residual-stream expansion as a promising scaling axis. However, existing HC-family methods typically stop at $N{=}4$. Our experiments reveal why: scaling mHC beyond this point yields diminishing performance gains and rapidly increasing training cost. We attribute this limitation to two bottlenecks: insufficient write-back information for an expanding number of streams and residual-mixing generation whose cost scales cubically with $N$. To address both bottlenecks, we propose xHC (Expanded Hyper-Connections), the first HC-family method to achieve meaningful expansion beyond $N{=}4$. xHC combines temporal feature augmentation for richer write-back with a sparse residual-stream architecture that updates only $k=4$ of the $N=16$ streams while retaining dense access to the full residual state. Across 18B and 28B MoE models, xHC delivers strong and consistent downstream improvements. On an 18B MoE model, xHC improves the average downstream score by 4.0 points over mHC, while adding only modest training FLOPs over the vanilla baseline. Scaling-law experiments show that the vanilla and mHC require $1.50\times$ and $1.19\times$ the compute of xHC, respectively, to reach the same loss. Practical large-$N$ training also requires controlling memory traffic from the expanded residual state. We therefore introduce xHC-Flash, which reduces the per-sublayer memory traffic from $73.5C$ to $40C$, comparable to the $34C$ required by mHC at $N{=}4$, while retaining the gains of full xHC. Together, xHC and xHC-Flash make large-$N$ residual-stream expansion effective and practical for LLM pre-training.