Paper 2512.19693
The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencoding
- Published
- Dec 2025
- Research lab
- Independent
- Citations
- 15
- GitHub
- 209 stars
01 In brief
Summary
The paper introduces the Prism Hypothesis, which posits that multimodal data can be understood through a shared frequency spectrum: semantic encoders capture low-frequency components (abstract meaning), while pixel encoders retain high-frequency details (fine texture).
This is supported by experiments showing that text-image retrieval relies on low frequencies and that semantic encoders concentrate energy in low bands.
Building on this, the authors propose Unified Autoencoding (UAE), a tokenizer that harmonizes semantic and pixel representations in a single latent space via a frequency-band modulator.
UAE decomposes features into low-frequency semantic and high-frequency residual bands, applies a semantic-wise loss only to low bands, and uses frequency-masked prediction.
Experiments show UAE achieves state-of-the-art reconstruction (e.g., PSNR 32.17 on ImageNet with DINOv2-B) and generative performance (gFID 1.52), outperforming baselines like RAE, SVG, and UniFlow.
UAE also improves pixel-space generation when integrated with JIT, reducing FID from 31.81 to 25.80 at 40 epochs.
The work demonstrates that unified training shifts energy to higher frequencies, enhancing detail while preserving semantics, and offers a practical route toward unified tokenizers for understanding and generation.
02 From the paper
Abstract
Deep representations across modalities are inherently intertwined. In this paper, we systematically analyze the spectral characteristics of various semantic and pixel encoders. Interestingly, our study uncovers a highly inspiring and rarely explored correspondence between an encoder's feature spectrum and its functional role: semantic encoders primarily capture low-frequency components that encode abstract meaning, whereas pixel encoders additionally retain high-frequency information that conveys fine-grained detail. This heuristic finding offers a unifying perspective that ties encoder behavior to its underlying spectral structure. We define it as the Prism Hypothesis, where each data modality can be viewed as a projection of the natural world onto a shared feature spectrum, just like the prism. Building on this insight, we propose Unified Autoencoding (UAE), a model that harmonizes semantic structure and pixel details via an innovative frequency-band modulator, enabling their seamless coexistence. Extensive experiments demonstrate that UAE effectively unifies semantic abstraction and pixel-level fidelity within a single latent space, achieving state-of-the-art performance. Moreover, we show that UAE can be directly applied to pixel-space modeling, significantly improving both FID and IS over the vanilla JIT baseline. Our code is avaliable at: https://github.com/WeichenFan/UAE.