The Year in AI Papers
The year/Independent research

Paper 2510.14979

From Pixels to Words -- Towards Native Vision-Language Primitives at Scale

Haiwen Diao, Mingxuan Li, Silei Wu, Linjun Dai, Xiaohua Wang, Hanming Deng, Lewei Lu, Dahua Lin, Ziwei Liu

Published
Oct 2025
Research lab
Independent
Citations
12
GitHub
881 stars

01 In brief

Summary

This paper introduces NEO, a family of native Vision-Language Models (VLMs) that unify vision and language processing in a single monolithic architecture, contrasting with modular VLMs that combine separate visual encoders and language models.

NEO is built from first principles using three key primitives: flexible position encoding, Multi-Head Native Attention (MHNA) for joint visual-textual processing, and Native Rotary Position Embedding (Native-RoPE) that decouples temporal, height, and width dimensions.

The architecture includes a randomly initialized pre-Buffer for visual learning and a post-LLM initialized from a pretrained LLM, which are merged after pre-training.

NEO is trained end-to-end on 390M image-text samples across three stages: pre-training (345M samples), mid-training (40M samples), and supervised fine-tuning (4M instructions).

Experiments show NEO-2.2B and NEO-9B achieve competitive performance, narrowing the gap with top modular VLMs and outperforming existing native VLMs on benchmarks like MMMU, MMBench, and ChartQA.

Ablations confirm the benefits of mixed attention, Native-RoPE, and the pre-Buffer design.

The authors position NEO as a scalable, reusable foundation for future native VLM research.

02 From the paper

Abstract

The edifice of native Vision-Language Models (VLMs) has emerged as a rising contender to typical modular VLMs, shaped by evolving model architectures and training paradigms. Yet, two lingering clouds cast shadows over its widespread exploration and promotion: (-) What fundamental constraints set native VLMs apart from modular ones, and to what extent can these barriers be overcome? (-) How to make research in native VLMs more accessible and democratized, thereby accelerating progress in the field. In this paper, we clarify these challenges and outline guiding principles for constructing native VLMs. Specifically, one native VLM primitive should: (i) effectively align pixel and word representations within a shared semantic space; (ii) seamlessly integrate the strengths of formerly separate vision and language modules; (iii) inherently embody various cross-modal properties that support unified vision-language encoding, aligning, and reasoning. Hence, we launch NEO, a novel family of native VLMs built from first principles, greatly narrowing the gap with top-tier modular counterparts across diverse real-world scenarios. With 390M image-text examples, NEO efficiently develops visual perception from scratch while mitigating vision-language conflicts inside a dense and monolithic model crafted from our elaborate primitives. We position NEO as a cornerstone for scalable and powerful native VLM development, paired with a rich set of reusable components that foster a cost-effective and extensible ecosystem. Our code and models are publicly available at: https://github.com/EvolvingLMMs-Lab/NEO.