The Year in AI Papers
The year/Independent research

Paper 2509.15207

FlowRL: Matching Reward Distributions for LLM Reasoning

Xuekai Zhu, Daixuan Cheng, Dinghuai Zhang, Hengli Li, Kaiyan Zhang, Che Jiang, Youbang Sun, Ermo Hua, Yuxin Zuo, Xingtai Lv, Qizheng Zhang, Lin Chen, et al.

Published
Sep 2025
Research lab
Independent
Citations
37
GitHub
Not linked

01 In brief

Summary

FlowRL is a policy optimization algorithm for large language model (LLM) reasoning that shifts from reward maximization to reward distribution matching.

It uses a learnable partition function to normalize scalar rewards into a target distribution and minimizes the reverse KL divergence between the policy and this distribution, which is equivalent to a trajectory balance objective from GFlowNets.

To handle long chain-of-thought (CoT) reasoning, FlowRL incorporates length normalization to prevent gradient explosion and importance sampling to correct for off-policy sampling mismatch.

Experiments on math and code benchmarks show FlowRL outperforms GRPO and PPO by 10.0% and 5.1% respectively on math tasks, and achieves the best results on code benchmarks (e.g., LiveCodeBench Avg@16 37.43%, Codeforces rating 1549.47, HumanEval+ 83.28%).

Ablation studies confirm the importance of importance sampling and the optimal beta hyperparameter (beta=15).

Diversity analysis shows FlowRL generates more diverse solutions than baselines, and a case study illustrates FlowRL avoids repetitive patterns that GRPO exhibits.

The method is theoretically shown to jointly maximize reward and policy entropy, promoting diverse exploration and generalization.

02 From the paper

Abstract

We propose FlowRL: matching the full reward distribution via flow balancing instead of maximizing rewards in large language model (LLM) reinforcement learning (RL). Recent advanced reasoning models adopt reward-maximizing methods (\eg, PPO and GRPO), which tend to over-optimize dominant reward signals while neglecting less frequent but valid reasoning paths, thus reducing diversity. In contrast, we transform scalar rewards into a normalized target distribution using a learnable partition function, and then minimize the reverse KL divergence between the policy and the target distribution. We implement this idea as a flow-balanced optimization method that promotes diverse exploration and generalizable reasoning trajectories. We conduct experiments on math and code reasoning tasks: FlowRL achieves a significant average improvement of $10.0\%$ over GRPO and $5.1\%$ over PPO on math benchmarks, and performs consistently better on code reasoning tasks. These results highlight reward distribution-matching as a key step toward efficient exploration and diverse reasoning in LLM reinforcement learning.