Paper 2508.19652
Self-Rewarding Vision-Language Model via Reasoning Decomposition
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 91
- GitHub
- 180 stars
01 In brief
Summary
Vision-SR1 is a three-stage self-rewarding reinforcement learning method for vision-language models (VLMs) that improves visual reasoning without external visual supervision.
It decomposes VLM reasoning into visual perception and language reasoning, prompting the model to first generate a self-contained visual description, then re-prompting it to answer using only that description to compute a visual reward.
A multi-reward policy optimization with separate advantages, log probabilities, and KL terms for each component prevents reward entanglement.
Experiments on Qwen2.5-VL (3B, 7B) and Mimo-VL-7B show Vision-SR1 outperforms baselines like Vision-R1, Perception-R1, and Visionary-R1 on benchmarks including MMMU, MMMU-Pro, MathVerse, and HallusionBench, while reducing language shortcut rates and improving spatial reasoning.
The method adds only ~20% training overhead over standard GRPO and no extra GPU cost, unlike external reward models.
A new metric, Language Shortcut Rate (LSR), quantifies reliance on language priors.
Ablations confirm the self-reward component is essential.
The training data, Vision-SR1-47K, spans math, science knowledge, and general visual reasoning.
02 From the paper
Abstract
Vision-Language Models (VLMs) often suffer from visual hallucinations: generating things that are not consistent with visual inputs and language shortcuts, where they skip the visual part and just rely on text priors. These issues arise because most post training methods for VLMs rely on simple verifiable answer matching and supervise only final outputs, leaving intermediate visual reasoning without explicit guidance. As a result, VLMs receive sparse visual signals and often learn to prioritize language based reasoning over visual perception. We introduce Vision SR1, a three stage self rewarding reinforcement learning method that improves visual reasoning without relying on external visual supervision. Vision SR1 decomposes VLM reasoning into two components: visual reasoning and language reasoning, where the model is first prompted to produce self-contained visual descriptions sufficient to answer the question without referring back to the input image, before jointly optimizing both visual and language reasoning through our multi reward loss objective. To validate this self containment, the same VLM model is reprompted to perform language reasoning using only the generated visual reasoning as input to compute visual reward. The final reward is computed through a decoupled reward-advantage framework, where visual reward and language reasoning reward each have their advantages calculated separately. Our experiments show that Vision SR1 improves visual reasoning, mitigates visual hallucinations, and reduces reliance on language shortcuts across diverse vision language tasks, while being more efficient than methods that rely on external visual reward models, which require additional GPUs to host. In contrast, Vision SR1 introduces no extra GPU overhead beyond that of standard training.