Paper 2510.11693
Scaling Language-Centric Omnimodal Representation Learning
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 15
- GitHub
- 48 stars
01 In brief
Summary
This paper introduces LCO-EMB, a language-centric framework for multimodal representation learning that leverages the implicit cross-modal alignment in multimodal large language models (MLLMs).
The authors show that text-only contrastive fine-tuning of the language decoder reduces anisotropy and improves kernel-level similarity across image, audio, and video modalities, indicating latent alignment from generative pretraining.
LCO-EMB uses LoRA to minimally perturb the model, preserving this alignment.
It achieves state-of-the-art results on MIEB-Lite with only ~370k training pairs, outperforming models trained on much larger datasets.
The paper also identifies a Generation-Representation Scaling Law (GRSL), showing that representation quality after contrastive learning scales positively with the MLLM's generative capability.
A PAC-Bayesian bound theoretically links generative loss to the upper bound on representation performance.
Experiments on SeaDoc, a new low-resource visual document retrieval benchmark, confirm that continual generative pretraining before contrastive learning improves retrieval performance.
02 From the paper
Abstract
Recent multimodal embedding approaches leveraging multimodal large language models (MLLMs) fine-tuned with contrastive learning (CL) have shown promising results, yet the underlying reasons behind their superiority remain underexplored. This work argues that a crucial advantage of MLLM-based approaches stems from implicit cross-modal alignment achieved during generative pretraining, where the language decoder learns to exploit multimodal signals within a shared representation space for generating unimodal outputs. Through analysis of anisotropy and kernel similarity structure, we empirically confirm that latent alignment emerges within MLLM representations, allowing CL to serve as a lightweight refinement stage. Leveraging this insight, we propose a Language-Centric Omnimodal Embedding framework, termed LCO-Emb. Extensive experiments across diverse backbones and benchmarks demonstrate its effectiveness, achieving state-of-the-art performance across modalities. Furthermore, we identify a Generation-Representation Scaling Law (GRSL), showing that the representational capabilities gained through contrastive refinement scales positively with the MLLM's generative capabilities. This suggests that improving generative abilities evolves as an effective paradigm for enhancing representation quality. We provide a theoretical explanation of GRSL, which formally links the MLLM's generative quality to the upper bound on its representation performance, and validate it on a challenging, low-resource visual-document retrieval task, showing that continual generative pretraining before CL can further enhance the potential of a model's embedding capabilities. Codes, models, and resources are available at https://github.com/LCO-Embedding/LCO-Embedding.