Paper 2508.20751
Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 90
- GitHub
- 276 stars
01 In brief
Summary
The paper introduces Pref-GRPO, a reinforcement learning method for text-to-image (T2I) generation that replaces pointwise reward score maximization with pairwise preference fitting to address reward hacking.
It identifies the root cause of reward hacking as an 'illusory advantage' arising from normalizing tightly clustered pointwise scores by a small group standard deviation, which amplifies minor score gaps.
Pref-GRPO uses a Pairwise Preference Reward Model (PPRM) to compare image pairs within a group, using each image's win rate as the reward, which yields larger within-group variance and robustness to reward noise.
Experiments on FLUX.1-dev show Pref-GRPO improves semantic alignment on UniGenBench (+5.84% overall, +12.69% on Text, +12.04% on Logical Reasoning) and on GenEval and T2I-CompBench, without sacrificing perceptual quality.
The paper also introduces UniGenBench, a fine-grained T2I evaluation benchmark with 600 prompts across 5 primary themes and 20 sub-themes, evaluating 10 primary dimensions and 27 sub-dimensions with 1-5 testpoints per prompt, using an MLLM (Gemini-2.5-Pro) for automated construction and evaluation.
Benchmarking reveals that Style and World Knowledge are saturated across strong models, while fine-grained compositional capabilities like Logical Reasoning, Grammar, and Compound remain bottlenecks.
Ablations show that converting point scores to win rates already improves stability, and Pref-GRPO is robust to 10% preference noise.
02 From the paper
Abstract
Recent advancements highlight the importance of GRPO-based reinforcement learning methods and benchmarking in enhancing text-to-image (T2I) generation. However, current methods using pointwise reward models (RM) for scoring generated images are susceptible to reward hacking. We reveal that this happens when minimal score differences between images are amplified after normalization, creating illusory advantages that drive the model to over-optimize for trivial gains, ultimately destabilizing the image generation process. To address this, we propose Pref-GRPO, a pairwise preference reward-based GRPO method that shifts the optimization objective from score maximization to preference fitting, ensuring more stable training. In Pref-GRPO, images are pairwise compared within each group using preference RM, and the win rate is used as the reward signal. Extensive experiments demonstrate that PREF-GRPO differentiates subtle image quality differences, providing more stable advantages and mitigating reward hacking. Additionally, existing T2I benchmarks are limited by coarse evaluation criteria, hindering comprehensive model assessment. To solve this, we introduce UniGenBench, a unified T2I benchmark comprising 600 prompts across 5 main themes and 20 subthemes. It evaluates semantic consistency through 10 primary and 27 sub-criteria, leveraging MLLM for benchmark construction and evaluation. Our benchmarks uncover the strengths and weaknesses of both open and closed-source T2I models and validate the effectiveness of Pref-GRPO.