Paper 2511.13720
Back to Basics: Let Denoising Generative Models Denoise
- Published
- Nov 2025
- Research lab
- Independent
- Citations
- 4
- GitHub
- 2.5K stars
01 In brief
Summary
The paper argues that denoising diffusion models should directly predict clean images (x-prediction) rather than noise (epsilon-prediction) or velocity (v-prediction), as natural data lies on a low-dimensional manifold while noised quantities do not.
The authors propose 'Just image Transformers' (JiT), a plain Vision Transformer applied to large pixel patches, using x-prediction with a v-loss.
They show that JiT with patch sizes 16 and 32 achieves competitive FID scores on ImageNet at 256 and 512 resolutions, where epsilon- and v-prediction fail catastrophically.
They also find that bottleneck designs can be beneficial and that increasing hidden units is not necessary.
The method is self-contained, requiring no tokenizer, pre-training, or extra losses.
Results include FID 1.82 for JiT-G/16 on ImageNet 256 and 1.78 for JiT-G/32 on 512, with scalability across model sizes and resolutions up to 1024.
02 From the paper
Abstract
Today's denoising diffusion models do not "denoise" in the classical sense, i.e., they do not directly predict clean images. Rather, the neural networks predict noise or a noised quantity. In this paper, we suggest that predicting clean data and predicting noised quantities are fundamentally different. According to the manifold assumption, natural data should lie on a low-dimensional manifold, whereas noised quantities do not. With this assumption, we advocate for models that directly predict clean data, which allows apparently under-capacity networks to operate effectively in very high-dimensional spaces. We show that simple, large-patch Transformers on pixels can be strong generative models: using no tokenizer, no pre-training, and no extra loss. Our approach is conceptually nothing more than "Just image Transformers", or JiT, as we call it. We report competitive results using JiT with large patch sizes of 16 and 32 on ImageNet at resolutions of 256 and 512, where predicting high-dimensional noised quantities can fail catastrophically. With our networks mapping back to the basics of the manifold, our research goes back to basics and pursues a self-contained paradigm for Transformer-based diffusion on raw natural data.