The Year in AI Papers
The year/Independent research

Paper 2601.16208

Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders

Shengbang Tong, Boyang Zheng, Ziteng Wang, Bingda Tang, Nanye Ma, Ellis Brown, Jihan Yang, Rob Fergus, Yann LeCun, Saining Xie

Published
Jan 2026
Research lab
Independent
Citations
40
GitHub
255 stars

01 In brief

Summary

This paper investigates scaling Representation Autoencoders (RAEs) for large-scale text-to-image (T2I) generation.

The authors train RAE decoders on a frozen SigLIP-2 encoder using web, synthetic, and text-rendering data, finding that data composition is crucial for text reconstruction.

They show that dimension-dependent noise scheduling remains essential, while architectural complexities like wide diffusion heads and noise-augmented decoding offer negligible benefits at scale.

In controlled comparisons against the FLUX VAE across DiT scales from 0.5B to 9.8B parameters, RAE-based models consistently converge faster (4.0x on GenEval, 4.6x on DPG-Bench) and achieve better performance during pretraining.

During finetuning, VAE models catastrophically overfit after 64 epochs, while RAE models remain stable through 256 epochs.

The shared representation space enables latent-space test-time scaling, where the LLM verifies generated latents directly, improving GenEval scores.

The authors conclude that RAEs provide a simpler and stronger foundation than VAEs for large-scale T2I generation, with implications for unified multimodal models.

02 From the paper

Abstract

Representation Autoencoders (RAEs) have shown distinct advantages in diffusion modeling on ImageNet by training in high-dimensional semantic latent spaces. In this work, we investigate whether this framework can scale to large-scale, freeform text-to-image (T2I) generation. We first scale RAE decoders on the frozen representation encoder (SigLIP-2) beyond ImageNet by training on web, synthetic, and text-rendering data, finding that while scale improves general fidelity, targeted data composition is essential for specific domains like text. We then rigorously stress-test the RAE design choices originally proposed for ImageNet. Our analysis reveals that scaling simplifies the framework: while dimension-dependent noise scheduling remains critical, architectural complexities such as wide diffusion heads and noise-augmented decoding offer negligible benefits at scale Building on this simplified framework, we conduct a controlled comparison of RAE against the state-of-the-art FLUX VAE across diffusion transformer scales from 0.5B to 9.8B parameters. RAEs consistently outperform VAEs during pretraining across all model scales. Further, during finetuning on high-quality datasets, VAE-based models catastrophically overfit after 64 epochs, while RAE models remain stable through 256 epochs and achieve consistently better performance. Across all experiments, RAE-based diffusion models demonstrate faster convergence and better generation quality, establishing RAEs as a simpler and stronger foundation than VAEs for large-scale T2I generation. Additionally, because both visual understanding and generation can operate in a shared representation space, the multimodal model can directly reason over generated latents, opening new possibilities for unified models.