Paper 2510.25616
Don't Blind Your VLA: Aligning Visual Representations for OOD Generalization
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 28
- GitHub
- 69 stars
01 In brief
Summary
The paper investigates how fine-tuning Vision-Language-Action (VLA) models on robotic tasks degrades the visual representations inherited from their base Vision-Language Models (VLMs).
The authors show that naive action fine-tuning causes representation collapse and attention sink, leading to domain-specific forgetting of vision-language knowledge.
They introduce the VL-Think task suite to diagnose this degradation and propose a lightweight Visual Representation Alignment method that anchors the VLA's internal visual features to a frozen pretrained vision teacher (e.g., C-RADIOv3) using a cosine similarity loss.
This method, applied during supervised fine-tuning with LoRA, consistently improves out-of-distribution generalization across semantic, vision, and execution axes on the Simpler benchmark, yielding up to a 10% relative gain over standard SFT.
Ablations show that aligning middle transformer layers with a frozen MLP projector and a small alignment coefficient (λ=0.2) works best.
The method also improves linear probing accuracy on ImageNet-100 and partially recovers VL capabilities in color and shape domains, though other domains remain limited by data and LoRA expressivity.
02 From the paper
Abstract
The growing success of Vision-Language-Action (VLA) models stems from the promise that pretrained Vision-Language Models (VLMs) can endow agents with transferable world knowledge and vision-language (VL) grounding, laying a foundation for action models with broader generalization. Yet when these VLMs are adapted to the action modality, it remains unclear to what extent their original VL representations and knowledge are preserved. In this work, we conduct a systematic study of representation retention during VLA fine-tuning, showing that naive action fine-tuning leads to degradation of visual representations. To characterize and measure these effects, we probe VLA's hidden representations and analyze attention maps, further, we design a set of targeted tasks and methods that contrast VLA models with their counterpart VLMs, isolating changes in VL capabilities induced by action fine-tuning. We further evaluate a range of strategies for aligning visual representations and introduce a simple yet effective method that mitigates degradation and yields improved generalization to out-of-distribution (OOD) scenarios. Taken together, our analysis clarifies the trade-off between action fine-tuning and the degradation of VL representations and highlights practical approaches to recover inherited VL capabilities. Code is publicly available: https://blind-vla-paper.github.io