The Year in AI Papers
The year/Independent research

Paper 2607.09657

Scalable Visual Pretraining for Language Intelligence

Yiming Zhang, Zhonghan Zhao, Wenwei Zhang, Haiteng Zhao, Tianyang Lin, Huanze Tang, Yunhua Zhou, Demin Song, Kuikun Liu, Haochen Ye, Haian Huang, Yuzhe Gu, et al.

Published
Jul 2026
Research lab
Independent
Citations
0
GitHub
Not linked

01 In brief

Summary

This paper introduces Visual Pretraining (VP), a framework that trains foundation models directly on raw document images without text extraction or image-text pairing, using a next-visual-latent prediction objective.

VP consistently outperforms text-only pretraining (TP) on scientific reasoning benchmarks across multiple backbones (Qwen3.5, Qwen3, Llama3.2 Vision, Llama3.1), using only 25% of the token budget (20B vs 80B tokens).

Gains scale with training data and are largest on visually dense pages.

VP also improves cross-modal alignment and multimodal benchmark performance without paired supervision.

The method is efficient, with a decoder-free design, and shows that visual pretraining complements text pretraining for language intelligence.

02 From the paper

Abstract

The rapid progress of large foundation models has been driven predominantly by pretraining on large-scale text corpora. However, many forms of knowledge are conveyed through visual representations, where figures, typeset equations, and page layouts carry rich information that cannot be faithfully or completely captured by text alone. Yet current pretraining approaches discard these visual cues by converting visually rich sources, such as documents and web pages, into plain text for learning language intelligence. This paper challenges the default assumption that language models must be trained on text-only representations and shows that Visual Pretraining is a scalable learner for foundation model intelligence. To this end, we conduct a systematic study of unsupervised visual pretraining paradigms that directly leverage visual documents without text extraction. Across multiple backbones and benchmarks, visual pretraining on the same underlying corpora consistently outperforms text-only pretraining, offering an efficient pathway to scalable language intelligence.