Paper 2604.02327
Steerable Visual Representations
- Published
- Apr 2026
- Research lab
- Independent
- Citations
- 2
- GitHub
- 119 stars
01 In brief
Summary
The paper introduces Steerable Visual Representations (SteerViT), a method to make pretrained Vision Transformers (ViTs) steerable by natural language.
SteerViT injects text into the visual encoder via lightweight gated cross-attention layers (early fusion), unlike late-fusion models like CLIP.
It is trained on a referential segmentation pretext task using 162k images and 2.28M image-text pairs.
SteerViT achieves a Pareto improvement: it can be steered by text (e.g., 96% retrieval accuracy on the new CORE benchmark vs.
43.7% for DINOv2) while preserving representation quality (98.8% of DINOv2's average performance on classification, segmentation, and action recognition).
It also matches or outperforms dedicated methods on anomaly detection (82.1 PRO on MVTec AD) and personalized object discrimination (58.1% PR-AUC on PODS) without task-specific training.
The method adds only 21M trainable parameters and generalizes across DINOv2, SigLIP, and MAE backbones.
A gating factor allows continuous interpolation between vanilla and steered features, with an optimal scaling factor of 0.6.
02 From the paper
Abstract
Pretrained Vision Transformers (ViTs) such as DINOv2 and MAE provide generic image features that can be applied to a variety of downstream tasks such as retrieval, classification, and segmentation. However, such representations tend to focus on the most salient visual cues in the image, with no way to direct them toward less prominent concepts of interest. In contrast, Multimodal LLMs can be guided with textual prompts, but the resulting representations tend to be language-centric and lose their effectiveness for generic visual tasks. To address this, we introduce Steerable Visual Representations, a new class of visual representations, whose global and local features can be steered with natural language. While most vision-language models (e.g., CLIP) fuse text with visual features after encoding (late fusion), we inject text directly into the layers of the visual encoder (early fusion) via lightweight cross-attention. We introduce benchmarks for measuring representational steerability, and demonstrate that our steerable visual features can focus on any desired objects in an image while preserving the underlying representation quality. Our method also matches or outperforms dedicated approaches on anomaly detection and personalized object discrimination, exhibiting zero-shot generalization to out-of-distribution tasks.