Paper 2509.00676
LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 30
- GitHub
- 4.7K stars
01 In brief
Summary
The paper introduces LLaVA-Critic-R1, a multimodal critic model trained via reinforcement learning (RL) on preference-labeled critic data, which surprisingly also becomes a strong policy model.
By reformulating critic data into verifiable tasks and using GRPO with preference and format rewards, the model improves over its base (Qwen-2.5-VL-7B) by +5.7% on average across 26 benchmarks, matching or surpassing specialized reasoning VLMs.
Extending this to ThinkLite-VL-7B yields LLaVA-Critic-R1+, achieving state-of-the-art 71.9 on MMMU at 7B scale.
Test-time self-critique with Best-of-128 gives an average +13.8% improvement on five reasoning tasks.
Ablations show that critic training enhances visual perception and structured reasoning, and that policy-then-critic training best balances both abilities.
SFT-based distillation degrades policy performance, while cold-start RL is optimal.
The approach generalizes to other base models like Mimo-VL and LLaMA-3.2-Vision, consistently improving both critic and policy capabilities.
02 From the paper
Abstract
In vision-language modeling, critic models are typically trained to evaluate outputs -- assigning scalar scores or pairwise preferences -- rather than to generate responses. This separation from policy models, which produce the responses, is so entrenched that critics are rarely considered for direct policy use. In this work, we challenge this convention. We propose to reorganize preference-labeled critic datasets into verifiable training signals and perform reinforcement learning directly on a base generative model, producing LLaVA-Critic-R1, a multimodal critic trained to optimize preference judgments while retaining full generation ability. Surprisingly, LLaVA-Critic-R1 emerges not only as a top-performing critic but also as a competitive policy model -- matching or surpassing specialized reasoning VLMs trained with in-domain data across 26 visual reasoning and understanding benchmarks, with an average gain of +5.7% over its base model (Qwen-2.5-VL-7B). Extending this approach to existing strong reasoning VLMs yields LLaVA-Critic-R1+, which further advances policy performance without sacrificing critic quality, achieving a SoTA performance of 71.9 on MMMU at the 7B scale. Finally, we show that the enhanced critic ability benefits inference: applying self-critique at test time yields an average +13.8% improvement on five representative reasoning tasks without additional training. Our results reveal that RL training on critic data can produce a unified model excelling at both evaluation and generation, offering a simple path toward scalable, self-improving multimodal systems.