Paper 2603.06569
Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders
- Published
- Mar 2026
- Research lab
- Independent
- Citations
- 4
- GitHub
- 206 stars
01 In brief
Summary
Penguin-VL, developed by Tencent AI Lab, introduces compact 2B and 8B vision-language models that challenge the reliance on contrastive pretraining (e.g., CLIP/SigLIP) for vision encoders.
The authors argue that contrastive learning suppresses fine-grained visual cues needed for reasoning.
Instead, they propose Penguin-Encoder, initialized from a text-only LLM (Qwen3-0.6B), with bidirectional attention and 2D-RoPE.
Training includes a reconstruction loss (amplitude, direction, relation) and a two-stage recipe (low-resolution pretraining, high-resolution fine-tuning).
The VLM uses a Temporal Redundancy-Aware (TRA) token compression for videos.
On benchmarks, Penguin-VL 2B outperforms Qwen3-VL 2B on ChartQA (86.6 vs 76.9), DocVQA (94.1 vs 93.3), MathVista (67.3 vs 61.3), and LongVideoBench (59.5 vs 52.1), while trailing on OCRBench and MMMU-Pro.
The 8B model leads on DocVQA (96.2), ChartQA (90.5), and NextQA (85.4), but lags on MathVerse and LogicVista.
Ablations show LLM initialization and relation loss improve average scores by +3.3 and +1.3, respectively, and Penguin-Encoder outperforms SigLIP2 and Qwen3-ViT under matched data.
The work demonstrates that improved visual representation, not model scaling, drives performance in resource-constrained settings.
02 From the paper
Abstract
Vision Language Model (VLM) development has largely relied on scaling model size, which hinders deployment on compute-constrained mobile and edge devices such as smartphones and robots. In this work, we explore the performance limits of compact (e.g., 2B and 8B) VLMs. We challenge the prevailing practice that state-of-the-art VLMs must rely on vision encoders initialized via massive contrastive pretraining (e.g., CLIP/SigLIP). We identify an objective mismatch: contrastive learning, optimized for discrimination, enforces coarse and category-level invariances that suppress fine-grained visual cues needed for dense captioning and complex VLM reasoning. To address this issue, we present Penguin-VL, whose vision encoder is initialized from a text-only LLM. Our experiments reveal that Penguin-Encoder serves as a superior alternative to traditional contrastive pretraining, unlocking a higher degree of visual fidelity and data efficiency for multimodal understanding. Across various image and video benchmarks, Penguin-VL achieves performance comparable to leading VLMs (e.g., Qwen3-VL) in mathematical reasoning and surpasses them in tasks such as document understanding, visual knowledge, and multi-perspective video understanding. Notably, these gains are achieved with a lightweight architecture, demonstrating that improved visual representation rather than model scaling is the primary driver of performance. Our ablations show that Penguin-Encoder consistently outperforms contrastive-pretrained encoders, preserving fine-grained spatial and temporal cues that are critical for dense perception and complex reasoning. This makes it a strong drop-in alternative for compute-efficient VLMs and enables high performance in resource-constrained settings. Code: https://github.com/tencent-ailab/Penguin-VL