Paper 2508.14460
DuPO: Enabling Reliable LLM Self-Verification via Dual Preference Optimization
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 5
- GitHub
- Not linked
01 In brief
Summary
DuPO is a dual learning-based preference optimization framework that generates annotation-free feedback for LLM optimization.
It addresses limitations of RLVR (costly labels, restricted to verifiable tasks) and traditional dual learning (strictly dual task pairs).
DuPO decomposes a primal task's input into known and unknown components, then constructs a dual task to reconstruct the unknown part using the primal output and known information, broadening applicability to non-invertible tasks.
The reconstruction quality serves as a self-supervised reward.
Empirically, DuPO improves average translation quality by 2.13 COMET over 756 directions, boosts mathematical reasoning accuracy by an average of 6.4 points on three benchmarks, and enhances performance by 9.3 points as an inference-time reranker.
It also scales to various backbones and can elicit reasoning from base models without SFT.
02 From the paper
Abstract
We present DuPO, a dual learning-based preference optimization framework that generates annotation-free feedback via a generalized duality. DuPO addresses two key limitations: Reinforcement Learning with Verifiable Rewards (RLVR)'s reliance on costly labels and applicability restricted to verifiable tasks, and traditional dual learning's restriction to strictly dual task pairs (e.g., translation and back-translation). Specifically, DuPO decomposes a primal task's input into known and unknown components, then constructs its dual task to reconstruct the unknown part using the primal output and known information (e.g., reversing math solutions to recover hidden variables), broadening applicability to non-invertible tasks. The quality of this reconstruction serves as a self-supervised reward to optimize the primal task, synergizing with LLMs' ability to instantiate both tasks via a single model. Empirically, DuPO achieves substantial gains across diverse tasks: it enhances the average translation quality by 2.13 COMET over 756 directions, boosts the mathematical reasoning accuracy by an average of 6.4 points on three challenge benchmarks, and enhances performance by 9.3 points as an inference-time reranker (trading computation for accuracy). These results position DuPO as a scalable, general, and annotation-free paradigm for LLM optimization.