Paper 2512.16922
Next-Embedding Prediction Makes Strong Vision Learners
- Published
- Dec 2025
- Research lab
- Independent
- Citations
- 13
- GitHub
- 340 stars
01 In brief
Summary
The paper introduces Next-Embedding Predictive Autoregression (NEPA), a self-supervised visual pretraining method that trains a Vision Transformer to predict future patch embeddings from past ones, using causal masking and stop-gradient, without pixel reconstruction, discrete tokens, or contrastive losses.
NEPA achieves 83.8% and 85.3% top-1 accuracy on ImageNet-1K with ViT-B and ViT-L backbones after fine-tuning, and 48.3% and 54.0% mIoU on ADE20K semantic segmentation.
Ablations show that causal masking, autoregressive shifting, and stop-gradient are essential; random masking hurts performance.
Architectural components like RoPE, QK-Norm, and LayerScale improve stability and accuracy.
The method scales with model size and training steps, and attention analysis shows object-centric, long-range dependencies.
NEPA is simpler than JEPA and MAE, requiring no decoder or auxiliary heads, and offers a modality-agnostic pretraining paradigm.
02 From the paper
Abstract
Inspired by the success of generative pretraining in natural language, we ask whether the same principles can yield strong self-supervised visual learners. Instead of training models to output features for downstream use, we train them to generate embeddings to perform predictive tasks directly. This work explores such a shift from learning representations to learning models. Specifically, models learn to predict future patch embeddings conditioned on past ones, using causal masking and stop gradient, which we refer to as Next-Embedding Predictive Autoregression (NEPA). We demonstrate that a simple Transformer pretrained on ImageNet-1k with next embedding prediction as its sole learning objective is effective - no pixel reconstruction, discrete tokens, contrastive loss, or task-specific heads. This formulation retains architectural simplicity and scalability, without requiring additional design complexity. NEPA achieves strong results across tasks, attaining 83.8% and 85.3% top-1 accuracy on ImageNet-1K with ViT-B and ViT-L backbones after fine-tuning, and transferring effectively to semantic segmentation on ADE20K. We believe generative pretraining from embeddings provides a simple, scalable, and potentially modality-agnostic alternative to visual self-supervised learning.