The Year in AI Papers
The year/Independent research

Paper 2607.11886

Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation

Runhui Huang, Qihui Zhang, Zhe Liu, Yu Gao, Jie Wu, Hengshuang Zhao

Published
Jul 2026
Research lab
Independent
Citations
0
GitHub
Not linked

01 In brief

Summary

The paper introduces SpectraReward, a training-free reward function for text-to-image reinforcement learning (RL) that converts pretrained multimodal large language models (MLLMs) into reward models.

Instead of asking the MLLM to judge images or answer questions, SpectraReward measures how well the original prompt can be recovered from a generated image via a single image-conditioned, teacher-forced forward pass, using the average prompt-token log-likelihood as the reward.

This reuses the MLLM's pretrained image-text alignment without preference labels or fine-tuning.

The authors also propose Self-SpectraReward, a special case for unified multimodal models where the policy's own understanding branch serves as the reward model for its generation branch, creating a closed-loop self-improving framework.

Experiments cover two diffusion models, three RL algorithms, nine reward MLLM backbones (4B to 235B parameters), and five out-of-distribution benchmarks.

Results show significant and consistent improvements over baselines and prior MLLM-derived reward methods.

Notably, larger reward MLLMs are not always better; Self-SpectraReward matches or surpasses much larger external models, suggesting reward-policy alignment is key.

The paper also analyzes the semantic spectrum's sensitivity to token-level errors and discusses limitations, including dependence on MLLM visual reasoning and potential biases.

02 From the paper

Abstract

In this paper, we propose SpectraReward, a training-free reward function that turns pretrained MLLMs into off-the-shelf reward models for image-generation reinforcement learning. Instead of asking the MLLM to judge a generated image or answer decomposed verification questions, SpectraReward measures how well the original prompt can be recovered from the generated image through a single image-conditioned, teacher-forced forward pass. We use the average image-conditioned prompt log-likelihood as the reward, directly reusing the MLLM's pretrained image-text alignment ability without preference labels, reward-model fine-tuning. We further introduce Self-SpectraReward, a special case for unified multimodal models where the policy's own understanding branch serves as the reward model for its generation branch, forming a closed-loop self-improving framework without external reward models or external knowledge. Extensive experiments validate SpectraReward through a broad image-generation RL study covering two diffusion models, three RL algorithms, nine reward MLLM backbones from four MLLM families spanning 4B to 235B parameters, and five out-of-distribution text-to-image benchmarks. Results show that both SpectraReward and Self-SpectraReward significantly and consistently improve generation performance and outperform prior MLLM-derived reward training methods. Further analysis reveals that larger reward MLLMs are not always better, while Self-SpectraReward can match or surpass much larger external reward models, suggesting that reward-policy alignment is a key factor for effective image-generation RL. Project Page: https://huangrh99.github.io/SpectraReward/