Paper 2605.06169
Mean Mode Screaming: Mean--Variance Split Residuals for 1000-Layer Diffusion Transformers
- Published
- May 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- 9 stars
01 In brief
Summary
This paper identifies a failure mode in ultra-deep Diffusion Transformers (DiTs) called Mean Mode Screaming (MMS), where token representations homogenize and centered variation is suppressed, leading to training collapse.
The mechanism involves a mean-coherent gradient shock that opens residual branches, causing a mean-dominated state.
The authors propose Mean–Variance Split (MV-Split) Residuals, which apply separate gains to centered and mean components, to stabilize training.
Experiments show that MV-Split prevents collapse in a 400-layer DiT and enables stable training of a 1000-layer DiT, outperforming LayerScale in convergence speed and final quality.
The paper also provides theoretical analysis of gradient decomposition and the Softmax Jacobian null space, and includes extensive appendices on diagnostics, derivations, and additional results.
The 1000-layer model achieves strong FID/IS scores and demonstrates text-to-image generation capability.
The work contributes a mechanistic understanding of deep DiT instability and a practical solution for scaling to extreme depths.
The authors also discuss limitations, including the challenge of predicting exact onset times and the applicability of the mechanism to other architectures.
Overall, the paper advances the scaling of diffusion transformers by addressing a specific structural vulnerability and offering a robust residual design.
The findings are supported by detailed experiments and theoretical insights, making a significant contribution to the field of generative modeling.
The code and model weights are publicly available.
The paper is a preprint and has not been peer-reviewed.
The authors are based in Beijing, China.
The work is relevant to researchers working on deep transformer architectures and diffusion models.
The proposed method…
02 From the paper
Abstract
Scaling Diffusion Transformers (DiTs) to hundreds of layers introduces a structural vulnerability: networks can enter a silent, mean-dominated collapse state that homogenizes token representations and suppresses centered variation. Through mechanistic auditing, we isolate the trigger event of this collapse as Mean Mode Screaming (MMS). MMS can occur even when training appears stable, with a mean-coherent backward shock on residual writers that opens deep residual branches and drives the network into a mean-dominated state. We show this behavior is driven by an exact decomposition of these gradients into mean-coherent and centered components, compounded by the structural suppression of attention-logit gradients through the null space of the Softmax Jacobian once values homogenize. To address this, we propose Mean-Variance Split (MV-Split) Residuals, which combine a separately gained centered residual update with a leaky trunk-mean replacement. On a 400-layer single-stream DiT, MV-Split prevents the divergent collapse that crashes the un-stabilized baseline; it tracks close to the baseline's pre-crash trajectory while remaining substantially better than token-isotropic gating methods such as LayerScale across the full schedule. Finally, we present a 1000-layer DiT as a scale-validation run at boundary scales, establishing that the architecture remains stably trainable at extreme depth.