The Year in AI Papers
The year/Meta AI

Paper 2603.03276

Beyond Language Modeling: An Exploration of Multimodal Pretraining

Shengbang Tong, David Fan, John Nguyen, Ellis Brown, Gaoyue Zhou, Shengyi Qian, Boyang Zheng, Théophane Vallaeys, Junlin Han, Rob Fergus, Naila Murray, Marjan Ghazvininejad, et al.

Published
Mar 2026
Research lab
Meta AI
Citations
21
GitHub
Not linked

01 In brief

Summary

This paper presents controlled, from-scratch experiments to clarify the design space of unified multimodal pretraining, using the Transfusion framework (next-token prediction for language, diffusion for vision) on text, video, image-text pairs, and action-conditioned video.

Key findings: (1) Representation Autoencoders (RAE), e.g., SigLIP 2, provide a single optimal visual representation for both understanding and generation, outperforming VAEs; (2) visual and language data are complementary, with multimodal co-training yielding synergy for downstream tasks like VQA and world modeling, while minimally impacting text performance; (3) world modeling capabilities (e.g., navigation) emerge from general multimodal pretraining, requiring minimal domain-specific data; (4) Mixture-of-Experts (MoE) architectures enable efficient scaling, naturally inducing modality specialization and outperforming hand-crafted separation strategies.

IsoFLOP analysis reveals a scaling asymmetry: vision is more data-hungry than language, and MoE harmonizes this gap by shifting language scaling toward a more data-hungry regime.

The paper concludes that modality competition is solvable via data and architecture choices, and that unified multimodal models can evolve into world models.

02 From the paper

Abstract

The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models remains opaque. We provide empirical clarity through controlled, from-scratch pretraining experiments, isolating the factors that govern multimodal pretraining without interference from language pretraining. We adopt the Transfusion framework, using next-token prediction for language and diffusion for vision, to train on diverse data including text, video, image-text pairs, and even action-conditioned video. Our experiments yield four key insights: (i) Representation Autoencoder (RAE) provides an optimal unified visual representation by excelling at both visual understanding and generation; (ii) visual and language data are complementary and yield synergy for downstream capabilities; (iii) unified multimodal pretraining leads naturally to world modeling, with capabilities emerging from general training; and (iv) Mixture-of-Experts (MoE) enables efficient and effective multimodal scaling while naturally inducing modality specialization. Through IsoFLOP analysis, we compute scaling laws for both modalities and uncover a scaling asymmetry: vision is significantly more data-hungry than language. We demonstrate that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language while accommodating the data-intensive nature of vision, paving the way for truly unified multimodal models.