The Year in AI Papers
The year/Independent research

Paper 2605.08063

Flow-OPD: On-Policy Distillation for Flow Matching Models

Zhen Fang, Wenxuan Huang, Yu Zeng, Yiming Zhao, Shuang Chen, Kaituo Feng, Yunlong Lin, Lin Chen, Zehui Chen, Shaosheng Cao, Feng Zhao

Published
May 2026
Research lab
Independent
Citations
11
GitHub
274 stars

01 In brief

Summary

Flow-OPD is a novel post-training framework for Flow Matching text-to-image models, addressing reward sparsity and gradient interference in multi-task alignment.

It uses a two-stage strategy: first, domain-specialized teachers are trained via single-reward GRPO; then, a student model is initialized via a Flow-based Cold-Start (SFT or model merging) and trained through on-policy sampling, task-routing labeling, and dense trajectory-level supervision.

Manifold Anchor Regularization (MAR) anchors generation to a high-quality manifold using a task-agnostic teacher, preventing aesthetic degradation.

Built on Stable Diffusion 3.5 Medium, Flow-OPD improves GenEval from 63 to 92 and OCR accuracy from 59 to 94, achieving about 10 points over vanilla GRPO, with an emergent 'teacher-surpassing' effect.

It outperforms GRPO baselines on T2I-CompBench and general quality metrics, demonstrating superior generalization and human preference alignment.

02 From the paper

Abstract

Existing Flow Matching (FM) text-to-image models suffer from two critical bottlenecks under multi-task alignment: the reward sparsity induced by scalar-valued rewards, and the gradient interference arising from jointly optimizing heterogeneous objectives, which together give rise to a 'seesaw effect' of competing metrics and pervasive reward hacking. Inspired by the success of On-Policy Distillation (OPD) in the large language model community, we propose Flow-OPD, the first unified post-training framework that integrates on-policy distillation into Flow Matching models. Flow-OPD adopts a two-stage alignment strategy: it first cultivates domain-specialized teacher models via single-reward GRPO fine-tuning, allowing each expert to reach its performance ceiling in isolation; it then establishes a robust initial policy through a Flow-based Cold-Start scheme and seamlessly consolidates heterogeneous expertise into a single student via a three-step orchestration of on-policy sampling, task-routing labeling, and dense trajectory-level supervision. We further introduce Manifold Anchor Regularization (MAR), which leverages a task-agnostic teacher to provide full-data supervision that anchors generation to a high-quality manifold, effectively mitigating the aesthetic degradation commonly observed in purely RL-driven alignment. Built upon Stable Diffusion 3.5 Medium, Flow-OPD raises the GenEval score from 63 to 92 and the OCR accuracy from 59 to 94, yielding an overall improvement of roughly 10 points over vanilla GRPO, while preserving image fidelity and human-preference alignment and exhibiting an emergent 'teacher-surpassing' effect. These results establish Flow-OPD as a scalable alignment paradigm for building generalist text-to-image models. The codes and weights will be released in: https://github.com/CostaliyA/Flow-OPD .