Paper 2511.10629
One Small Step in Latent, One Giant Leap for Pixels: Fast Latent Upscale Adapter for Your Diffusion Models
- Published
- Nov 2025
- Research lab
- Independent
- Citations
- 1
- GitHub
- 34 stars
01 In brief
Summary
The paper introduces the Latent Upscaler Adapter (LUA), a lightweight module that performs super-resolution directly on a diffusion model's latent code before VAE decoding, enabling high-resolution image synthesis without retraining the generator or adding diffusion stages.
LUA uses a shared SwinIR-style backbone with scale-specific pixel-shuffle heads for ×2 and ×4 upscaling, trained via a three-stage curriculum combining latent and pixel-domain losses.
It achieves comparable perceptual quality to pixel-space SR with nearly 3× lower decoding and upscaling time, adding only +0.42 s for 1024px generation from 512px.
LUA generalizes across different VAEs (FLUX, SD3, SDXL) with minimal fine-tuning.
On OpenImages, SDXL+LUA achieves the best fidelity at 2048² and 4096² (FID 180.80 and 176.90) with the lowest latency (3.52 s and 6.87 s), outperforming pixel-space SR and multi-stage re-diffusion pipelines.
At 1024², it trails direct sampling on FID but remains competitive on patch metrics.
The multi-head design saves about half the training cost compared to separate per-scale models.
Code is available at github.com/vaskers5/LUA.
02 From the paper
Abstract
Diffusion models struggle to scale beyond their training resolutions, as direct high-resolution sampling is slow and costly, while post-hoc image super-resolution (ISR) introduces artifacts and additional latency by operating after decoding. We present the Latent Upscaler Adapter (LUA), a lightweight module that performs super-resolution directly on the generator's latent code before the final VAE decoding step. LUA integrates as a drop-in component, requiring no modifications to the base model or additional diffusion stages, and enables high-resolution synthesis through a single feed-forward pass in latent space. A shared Swin-style backbone with scale-specific pixel-shuffle heads supports 2x and 4x factors and remains compatible with image-space SR baselines, achieving comparable perceptual quality with nearly 3x lower decoding and upscaling time (adding only +0.42 s for 1024 px generation from 512 px, compared to 1.87 s for pixel-space SR using the same SwinIR architecture). Furthermore, LUA shows strong generalization across the latent spaces of different VAEs, making it easy to deploy without retraining from scratch for each new decoder. Extensive experiments demonstrate that LUA closely matches the fidelity of native high-resolution generation while offering a practical and efficient path to scalable, high-fidelity image synthesis in modern diffusion pipelines.