Paper 2510.12586
There is No VAE: End-to-End Pixel-Space Generative Modeling via Self-Supervised Pre-training
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 15
- GitHub
- 153 stars
01 In brief
Summary
The paper introduces EPG, a two-stage training framework for pixel-space diffusion and consistency models that closes the performance gap with latent-space methods.
Stage one pre-trains an encoder using a contrastive loss and a representation consistency loss, aligning features of noisy images along deterministic sampling trajectories.
Stage two fine-tunes the encoder with a randomly initialized decoder end-to-end for generative tasks.
On ImageNet, the diffusion model achieves FID 1.58 (256) and 2.35 (512) with 75 NFE, surpassing prior pixel-space and VAE-based methods, and outperforms DiT with about 30% of its training compute.
The consistency model achieves FID 8.82 on ImageNet-256 in one step, the first successful pixel-space consistency model without pre-trained VAEs or diffusion models.
Ablations show the pre-training improves downstream performance and scales with compute.
The framework avoids VAE bottlenecks and offers efficient training across resolutions.
02 From the paper
Abstract
Pixel-space generative models are often more difficult to train and generally underperform compared to their latent-space counterparts, leaving a persistent performance and efficiency gap. In this paper, we introduce a novel two-stage training framework that closes this gap for pixel-space diffusion and consistency models. In the first stage, we pre-train encoders to capture meaningful semantics from clean images while aligning them with points along the same deterministic sampling trajectory, which evolves points from the prior to the data distribution. In the second stage, we integrate the encoder with a randomly initialized decoder and fine-tune the complete model end-to-end for both diffusion and consistency models. Our framework achieves state-of-the-art (SOTA) performance on ImageNet. Specifically, our diffusion model reaches an FID of 1.58 on ImageNet-256 and 2.35 on ImageNet-512 with 75 number of function evaluations (NFE) surpassing prior pixel-space methods and VAE-based counterparts by a large margin in both generation quality and training efficiency. In a direct comparison, our model significantly outperforms DiT while using only around 30\% of its training compute. Furthermore, our consistency model achieves an impressive FID of 8.82 on ImageNet-256, significantly outperforming its latent-space counterparts. This marks the first successful training of a consistency model directly on high-resolution images without relying on pre-trained VAEs or diffusion models. Our codes are available at: \href{https://github.com/AMAP-ML/EPG}{https://github.com/AMAP-ML/EPG}