Paper 2602.08099
VidVec: Unlocking Video MLLM Embeddings for Video-Text Retrieval
- Published
- Feb 2026
- Research lab
- Independent
- Citations
- 3
- GitHub
- Not linked
01 In brief
Summary
VidVec introduces a method to leverage generative Multimodal Large Language Models (MLLMs) for video–text retrieval without visual fine-tuning.
The authors show that intermediate layers of off-the-shelf video MLLMs (e.g., VideoLLaMA3-7B) encode substantial retrieval-relevant information, outperforming final-layer embeddings in zero-shot settings.
They propose a zero-shot two-stage approach (VidVec-ZS) that extracts embeddings from an intermediate layer and uses the MLLM head as a calibrated likelihood scorer for reranking, achieving strong results without training.
Additionally, they introduce an in-context optimization strategy (VidVec-O) that fine-tunes a lightweight LoRA on only 60K text-only pairs, mapping dense video captions to short summaries, using a Dual-Softmax Loss.
This text-only training yields state-of-the-art performance on MSR-VTT, MSVD, VATEX, and DiDeMo benchmarks, surpassing prior MLLM embedders and Video Foundation Models trained on orders of magnitude more data.
Key contributions include layer-wise analysis, calibrated reranking, and efficient text-only alignment, demonstrating that MLLMs can be effectively adapted for video retrieval without visual supervision.
02 From the paper
Abstract
Recent studies have adapted generative Multimodal Large Language Models (MLLMs) into embedding extractors for vision tasks, typically through fine-tuning to produce universal representations. However, their performance on video remains inferior to Video Foundation Models (VFMs). In this paper, we focus on leveraging MLLMs for video-text embedding and retrieval. We first conduct a systematic layer-wise analysis, showing that intermediate (pre-trained) MLLM layers already encode substantial task-relevant information. Leveraging this insight, we demonstrate that combining intermediate-layer embeddings with a calibrated MLLM head yields strong zero-shot retrieval performance without any training. Building on these findings, we introduce a lightweight text-based alignment strategy which maps dense video captions to short summaries and enables task-related video-text embedding learning without visual supervision. Remarkably, without any fine-tuning beyond text, our method outperforms current methods, often by a substantial margin, achieving state-of-the-art results across common video retrieval benchmarks.