The Year in AI Papers
The year/Independent research

Paper 2606.09076

Z-Reward: Beyond Scalar Rewards by Internalizing Reasoning into Score Distributions

Xin Jin, Huanqia Cai, Zhen Li, Zechao Zhan, Dengyang Jiang, Aiming Hao, Yuming Jiang, Xiangpeng Yang, Chunle Guo, Peng Gao, Ming-Ming Cheng, Steven C. H. Hoi

Published
Jun 2026
Research lab
Independent
Citations
0
GitHub
Not linked

01 In brief

Summary

Z-Reward is a teacher-student framework for text-to-image reward modeling that represents visual preference as a reasoning-conditioned score distribution rather than a scalar.

The teacher, a 27B VLM, is trained with Group-wise Direct Score Optimization (GDSO), which combines GRPO-style policy-gradient rewards with direct supervision on score distributions and score gaps.

The student, a 9B VLM, is trained with Reasoning-Internalized Score Distillation (RISD), which distills the teacher's reasoning-conditioned distribution into direct scoring without generating reasoning chains.

On an internally annotated test set, the 27B GDSO teacher achieves 89.6% human preference accuracy, outperforming SFT, RewardDance, and GRPO.

The 9B RISD student reaches 88.6%, outperforming the OPD baseline and closely matching the teacher.

Z-Reward also serves as a differentiable reward signal for text-to-image optimization, yielding a 41.3% net human-preference improvement over the SFT baseline.

The framework decouples reasoning-heavy judgment from efficient deployment, addressing the tension between high-quality scoring and scalable, differentiable reward signals.

02 From the paper

Abstract

Reward models are central to text-to-image post-training, but visual preference is subjective and better represented as a distribution over rubric scores than as a deterministic scalar. Existing scalar, score-token, and pairwise reward models over-compress uncertainty and fine-grained score differences, while reasoning-based generative rewards provide stronger judgments but are costly to deploy and difficult to use as direct optimization signals. We propose Z-Reward, a teacher-student reward modeling framework that decouples reasoning-heavy judgment from efficient reward deployment. The teacher is a large VLM that uses reasoning to infer rubric-aligned score distributions, and is trained with Group-wise Direct Score Optimization (GDSO), which combines policy-gradient rewards from distribution expectations with direct pointwise and pairwise supervision on score distributions and score gaps. The student is trained with Reasoning-Internalized Score Distillation (RISD), which transfers the teacher's reasoning-conditioned score distribution into a compact VLM without requiring explicit reasoning chains at inference time. On our internally annotated evaluation set, the 27B GDSO teacher reaches 89.6% human preference accuracy, outperforming SFT, RewardDance, and GRPO, while the 9B RISD student reaches 88.6%, outperforming the OPD baseline and closely matching the larger teacher. We further show that Z-Reward can serve as a differentiable reward signal for text-to-image optimization, yielding a 41.3% net human-preference improvement over the SFT baseline.