The Year in AI Papers
The year/Independent research

Paper 2605.06139

Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex

Yun Qu, Qi Wang, Yixiu Mao, Heming Zou, Yuhang Jiang, Yingyue Li, Wutong Xu, Lizhou Cai, Weijie Liu, Clive Bai, Kai Yang, Yangkun Chen, et al.

Published
May 2026
Research lab
Independent
Citations
1
GitHub
Not linked

01 In brief

Summary

This paper introduces Listwise Policy Optimization (LPO), a framework for reinforcement learning with verifiable rewards (RLVR) in large language models.

The authors show that existing group-based policy gradient methods (e.g., GRPO, Dr.GRPO, MaxRL) implicitly perform approximate target-projection on a response simplex, where the target is a reward-weighted softmax distribution.

LPO makes this projection explicit by decoupling target construction from divergence minimization, using forward or reverse KL divergence.

The method provides theoretical guarantees of monotonic improvement and gradient properties (bounded, zero-sum, self-correcting).

Experiments across logic, math, programming, and multi-modal geometry tasks with various LLM backbones (1.5B–14B) show LPO consistently outperforms baselines in Pass@1 and Pass@k accuracy, maintains higher response entropy, and exhibits more stable gradient norms.

The forward KL variant shows particular strength in preserving diversity.

LPO also demonstrates scalability on larger datasets and generalization across model families.

02 From the paper

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a standard approach for large language models (LLMs) post-training to incentivize reasoning capacity. Among existing recipes, group-based policy gradient is prevalent, which samples a group of responses per prompt and updates the policy via group-relative advantage signals. This work reveals that these optimization strategies share a common geometric structure: each implicitly defines a target distribution on the response simplex and projects toward it via first-order approximation. Building on this insight, we propose Listwise Policy Optimization (LPO) to explicitly conduct the target-projection, which demystifies the implicit target by restricting the proximal RL objective to the response simplex, and then projects the policy via exact divergence minimization. This framework provides (i) monotonic improvement on the listwise objective with bounded, zero-sum, and self-correcting projection gradients, and (ii) flexibility in divergence selection with distinct structural properties through the decoupled projection step. On diverse reasoning tasks and LLM backbones, LPO consistently improves training performance over typical policy gradient baselines under matched targets, while intrinsically preserving optimization stability and response diversity.