Paper 2603.16932
Look Where It Matters: High-Resolution Crops Retrieval for Efficient VLMs
- Published
- Mar 2026
- Research lab
- Independent
- Citations
- 1
- GitHub
- 11 stars
01 In brief
Summary
The paper presents AwaRes, a framework for efficient vision-language model (VLM) inference that processes a low-resolution global image and uses tool-calling to retrieve only the high-resolution crops needed for a query.
AwaRes trains a coupled-decision policy (CDP) that jointly decides whether to escalate resolution and which crops to request.
Supervision is generated automatically: an LLM judge labels whether low resolution is sufficient, and an oracle grounding model localizes evidence to create crop targets.
Training uses cold-start supervised fine-tuning (SFT) followed by multi-turn GRPO with a composite reward combining semantic answer correctness and crop-cost penalties.
On six benchmarks (ChartQA, DocVQA, OCRBench, POPE, RealWorldQA, V*Bench), AwaRes achieves an average accuracy of 80.30, nearly matching the full-resolution baseline (80.46), while using only 36% of the visual tokens.
It outperforms fixed-budget token pruning methods (e.g., VisionZip) and the adaptive baseline VisionThink (accuracy 80.30 vs.
79.23, RTR 0.36 vs.
0.61).
AwaRes also reduces wall-clock latency (e.g., 0.6s vs.
4.3s on ChartQA) by using short tool calls and KV-cache reuse.
Ablations show the importance of trajectory-level SFT, tool-turn weighting, and the GRPO reward components.
02 From the paper
Abstract
Vision-language models (VLMs) typically process images at a native high-resolution, forcing a trade-off between accuracy and computational efficiency: high-resolution inputs capture fine details but incur significant computational costs, while low-resolution inputs advocate for efficiency, they potentially miss critical visual information, like small text. We present AwaRes, a spatial-on-demand framework that resolves this accuracy-efficiency trade-off by operating on a low-resolution global view and using tool-calling to retrieve only high-resolution segments needed for a given query. We construct supervised data automatically: a judge compares low- vs.\ high-resolution answers to label whether cropping is needed, and an oracle grounding model localizes the evidence for the correct answer, which we map to a discrete crop set to form multi-turn tool-use trajectories. We train our framework with cold-start SFT followed by multi-turn GRPO with a composite reward that combines semantic answer correctness with explicit crop-cost penalties. Project page: https://nimrodshabtay.github.io/AwaRes