The Year in AI Papers
The year/Independent research

Paper 2509.07979

Visual Representation Alignment for Multimodal Large Language Models

Heeji Yoon, Jaewoo Jung, Junwan Kim, Hyungyu Choi, Heeseong Shin, Sangbeom Lim, Honggyu An, Chaehyun Kim, Jisang Han, Donghyun Kim, Chanho Eom, Sunghwan Hong, et al.

Published
Sep 2025
Research lab
Independent
Citations
35
GitHub
162 stars

01 In brief

Summary

Multimodal large language models (MLLMs) trained with visual instruction tuning underperform on vision-centric tasks like object counting and spatial reasoning.

The authors attribute this to text-only supervision, which provides indirect guidance for the visual pathway, causing MLLMs to discard fine-grained visual details from the vision encoder.

They propose VIsual Representation ALignment (VIRAL), a regularization strategy that aligns the internal visual representations of MLLMs with those of pre-trained vision foundation models (VFMs) using a cosine similarity loss.

This alignment preserves critical visual details and incorporates additional visual knowledge from VFMs, enhancing reasoning over complex visual inputs.

Experiments on LLaVA-1.5 with various vision encoders (CLIP, SigLIPv2) and language backbones (Vicuna-7B/13B, Qwen2.5-7B) show consistent improvements across benchmarks, with gains up to 17.3% and an average improvement of 9.4% over the baseline.

Ablation studies identify DINOv2 as the most effective VFM and the 16th layer of a 32-layer model as the optimal alignment target.

VIRAL also improves attention focus, increases sensitivity to spatial token order, and accelerates convergence with only about 3% training overhead.

02 From the paper

Abstract

Multimodal large language models (MLLMs) trained with visual instruction tuning have achieved strong performance across diverse tasks, yet they remain limited in vision-centric tasks such as object counting or spatial reasoning. We attribute this gap to the prevailing text-only supervision paradigm, which provides only indirect guidance for the visual pathway and often leads MLLMs to discard fine-grained visual details during training. In this paper, we present VIsual Representation ALignment (VIRAL), a simple yet effective regularization strategy that aligns the internal visual representations of MLLMs with those of pre-trained vision foundation models (VFMs). By explicitly enforcing this alignment, VIRAL enables the model not only to retain critical visual details from the input vision encoder but also to complement additional visual knowledge from VFMs, thereby enhancing its ability to reason over complex visual inputs. Our experiments demonstrate consistent improvements across all tasks on widely adopted multimodal benchmarks. Furthermore, we conduct comprehensive ablation studies to validate the key design choices underlying our framework. We believe this simple finding opens up an important direction for the effective integration of visual information in training MLLMs.