Paper 2512.13687
Towards Scalable Pre-training of Visual Tokenizers for Generation
01 In brief
Summary
The paper introduces VTP, a visual tokenizer pre-training framework that integrates image-text contrastive learning, self-supervised learning (MIM and self-distillation), and reconstruction losses to address the 'pre-training scaling problem' in latent diffusion models.
The authors argue that reconstruction-only training biases the latent space toward low-level information, causing generative performance to stagnate or degrade with increased compute.
VTP, built on a ViT-based autoencoder, jointly optimizes these objectives to create a semantically rich latent space.
Experiments show that VTP unlocks a new scaling law: generative performance improves consistently with increased pre-training compute, model parameters, and data, unlike conventional autoencoders.
On ImageNet 256x256, VTP achieves 0.36 rFID, 78.2% zero-shot accuracy, 85.7% linear probing accuracy, and a 1.11 gFID with guidance, while also enabling fast convergence (2.03 gFID in 80 epochs without guidance).
The scaling properties generalize to text-to-image generation on LAION, where the CLIP loss improves text rendering.
VTP outperforms prior unified tokenizers like VILA-U and UniTok and shows better scalability than fixed-representation methods like RAE.
02 From the paper
Abstract
The quality of the latent space in visual tokenizers (e.g., VAEs) is crucial for modern generative models. However, the standard reconstruction-based training paradigm produces a latent space that is biased towards low-level information, leading to a foundation flaw: better pixel-level accuracy does not lead to higher-quality generation. This implies that pouring extensive compute into visual tokenizer pre-training translates poorly to improved performance in generation. We identify this as the ``pre-training scaling problem`` and suggest a necessary shift: to be effective for generation, a latent space must concisely represent high-level semantics. We present VTP, a unified visual tokenizer pre-training framework, pioneering the joint optimization of image-text contrastive, self-supervised, and reconstruction losses. Our large-scale study reveals two principal findings: (1) understanding is a key driver of generation, and (2) much better scaling properties, where generative performance scales effectively with compute, parameters, and data allocated to the pretraining of the visual tokenizer. After large-scale pre-training, our tokenizer delivers a competitive profile (78.2 zero-shot accuracy and 0.36 rFID on ImageNet) and 4.1 times faster convergence on generation compared to advanced distillation methods. More importantly, it scales effectively: without modifying standard DiT training specs, solely investing more FLOPS in pretraining VTP achieves 65.8\% FID improvement in downstream generation, while conventional autoencoder stagnates very early at 1/10 FLOPS. Our pre-trained models are available at https://github.com/MiniMax-AI/VTP.