Paper 2604.11626
RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time
- Published
- Apr 2026
- Research lab
- Independent
- Citations
- 7
- GitHub
- 56 stars
01 In brief
Summary
The paper introduces RationalRewards, a reasoning-based reward model for visual generation that produces structured, multi-dimensional critiques before assigning scores, unlike traditional scalar reward models.
It is trained using Preference-Anchored Rationalization (PARROT), a variational framework that recovers rationales from preference data via anchored generation, consistency filtering, and distillation.
RationalRewards (8B) achieves state-of-the-art preference prediction among open-source models, competitive with Gemini-2.5-Pro, using 10-20x less training data.
As an RL reward, it improves text-to-image and image-editing generators beyond scalar baselines.
Notably, its test-time Generate-Critique-Refine loop, which refines prompts without parameter updates, matches or exceeds RL-based fine-tuning on several benchmarks, suggesting structured reasoning can unlock latent generator capabilities.
The model also enables data filtering, critique visualization, and dense visual rewards.
02 From the paper
Abstract
Most reward models for visual generation reduce rich human judgments to a single unexplained score, discarding the reasoning that underlies preference. We show that teaching reward models to produce explicit, multi-dimensional critiques before scoring transforms them from passive evaluators into active optimization tools, improving generators in two complementary ways: at training time, structured rationales provide interpretable, fine-grained rewards for reinforcement learning; at test time, a Generate-Critique-Refine loop turns critiques into targeted prompt revisions that improve outputs without any parameter updates. To train such a reward model without costly rationale annotations, we introduce Preference-Anchored Rationalization (PARROT), a principled framework that recovers high-quality rationales from readily available preference data through anchored generation, consistency filtering, and distillation. The resulting model, RationalRewards (8B), achieves state-of-the-art preference prediction among open-source reward models, competitive with Gemini-2.5-Pro, while using 10-20x less training data than comparable baselines. As an RL reward, it consistently improves text-to-image and image-editing generators beyond scalar alternatives. Most strikingly, its test-time critique-and-refine loop matches or exceeds RL-based fine-tuning on several benchmarks, suggesting that structured reasoning can unlock latent capabilities in existing generators that suboptimal prompts fail to elicit.