Unified Latents (UL): How to train your latents

Unified Latents (UL) is a framework for learning latent representations regularized by a diffusion prior and decoded by a diffusion model.

The encoder outputs a deterministic latent, which is noised to a fixed minimum noise level (log-SNR of 5), linking the encoder's output noise to the prior's precision.

The training objective combines a diffusion prior loss (unweighted ELBO) and a diffusion decoder loss (sigmoid-weighted), with a loss factor controlling the latent bitrate.

On ImageNet-512, UL achieves a competitive FID of 1.4 with high reconstruction quality (PSNR) while requiring fewer training FLOPs than models trained on Stable Diffusion latents.

On Kinetics-600, it sets a new state-of-the-art FVD of 1.3.

The framework provides interpretable control over the reconstruction-modelling trade-off via hyper-parameters like loss factor and sigmoid bias.

Ablations show that the diffusion prior is crucial, fixed encoder variance is more stable than learned variance, and the method is insensitive to latent channel count.

The paper also discusses scaling studies on text-to-image and video datasets, demonstrating improved training efficiency and generation quality compared to baselines.