Paper 2510.11690
Diffusion Transformers with Representation Autoencoders
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 221
- GitHub
- 2K stars
01 In brief
Summary
The paper introduces Representation Autoencoders (RAEs), which replace the traditional VAE in latent diffusion models with a frozen pretrained representation encoder (e.g., DINOv2, SigLIP2, MAE) paired with a trained decoder.
RAEs provide both high-quality reconstructions and semantically rich latent spaces, outperforming SD-VAE in reconstruction FID (e.g., 0.49 vs 0.62) and linear probing accuracy (84.5% vs 8.0%).
To enable diffusion transformers to work in these high-dimensional latent spaces, the authors propose three key adjustments: matching DiT width to token dimensionality, applying a dimension-dependent noise schedule shift, and noise-augmented decoder training.
They also introduce DiT^DH, a shallow-but-wide head that improves scalability.
On ImageNet, their RAE-based DiT^DH-XL achieves state-of-the-art FIDs of 1.51 at 256×256 without guidance and 1.13 at both 256×256 and 512×512 with AutoGuidance, demonstrating faster convergence and better generation quality compared to VAE-based methods.
The work challenges assumptions about the incompatibility of semantic encoders with reconstruction and generation, showing that higher-dimensional latent spaces can be advantageous with proper architectural adjustments.
The authors also note an inconsistency in FID evaluation protocols and re-evaluate baselines with class-balanced sampling for fair comparison.
Overall, RAEs offer a new default for diffusion transformer training, linking semantic and generative modeling through a shared latent representation.
The paper includes theoretical proofs, extensive ablations, and comparisons with prior methods like REPA, DDT, and SiT, showing significant training speedups (up to 47×) and improved FID scores across model scales.
The approach also extends to high-resolution synthesis via decoder upsampling, achieving competitive FID at 512×512 without retraining the diffusion model.
The…
02 From the paper
Abstract
Latent generative modeling, where a pretrained autoencoder maps pixels into a latent space for the diffusion process, has become the standard strategy for Diffusion Transformers (DiT); however, the autoencoder component has barely evolved. Most DiTs continue to rely on the original VAE encoder, which introduces several limitations: outdated backbones that compromise architectural simplicity, low-dimensional latent spaces that restrict information capacity, and weak representations that result from purely reconstruction-based training and ultimately limit generative quality. In this work, we explore replacing the VAE with pretrained representation encoders (e.g., DINO, SigLIP, MAE) paired with trained decoders, forming what we term Representation Autoencoders (RAEs). These models provide both high-quality reconstructions and semantically rich latent spaces, while allowing for a scalable transformer-based architecture. Since these latent spaces are typically high-dimensional, a key challenge is enabling diffusion transformers to operate effectively within them. We analyze the sources of this difficulty, propose theoretically motivated solutions, and validate them empirically. Our approach achieves faster convergence without auxiliary representation alignment losses. Using a DiT variant equipped with a lightweight, wide DDT head, we achieve strong image generation results on ImageNet: 1.51 FID at 256x256 (no guidance) and 1.13 at both 256x256 and 512x512 (with guidance). RAE offers clear advantages and should be the new default for diffusion transformer training.