Paper 2509.23102
Multiplayer Nash Preference Optimization
- Published
- Sep 2025
- Research lab
- Independent
- Citations
- 6
- GitHub
- 35 stars
01 In brief
Summary
This paper introduces Multiplayer Nash Preference Optimization (MNPO), a framework that generalizes Nash learning from human feedback (NLHF) to n-player games for aligning large language models with human preferences.
MNPO formulates alignment as a game where each policy competes against a population of opponents while being regularized toward a reference model.
The authors propose two variants: TD-MNPO, which uses time-dependent historical policies as opponents and has provable convergence guarantees, and HT-MNPO, which handles heterogeneous preference oracles (e.g., multiple reward models) but lacks formal guarantees.
The framework unifies existing preference optimization methods (e.g., DPO, SimPO, INPO) as special cases.
Experiments with Gemma-2-9B-it show MNPO consistently outperforms baselines on instruction-following benchmarks (AlpacaEval 2.0, Arena-Hard, MT-Bench) and maintains or improves performance on reasoning, math, and coding benchmarks.
For instance, TD-MNPO achieves 57.27 on AlpacaEval 2.0 and 52.26 on Arena-Hard, surpassing INPO by 1.18 and 4.23 points respectively.
HT-MNPO with Athene-RM-8B reaches 59.64 on AlpacaEval 2.0.
The authors conclude that multiplayer formulations better capture heterogeneous, non-transitive preferences and provide more robust alignment.
02 From the paper
Abstract
Reinforcement learning from human feedback (RLHF) has emerged as the standard paradigm for aligning large language models with human preferences. However, reward-based methods grounded in the Bradley-Terry assumption struggle to capture the nontransitivity and heterogeneity of real-world preferences. To address this, recent studies have reframed alignment as a two-player Nash game, giving rise to Nash learning from human feedback (NLHF). While this perspective has inspired algorithms such as INPO, ONPO, and EGPO that offer strong theoretical and empirical guarantees, they remain fundamentally restricted to two-player interactions, introducing a single-opponent bias that fails to capture the full complexity of realistic preference structures. This work introduces Multiplayer Nash Preference Optimization (MNPO), a novel framework that generalizes NLHF to the multiplayer regime. It formulates alignment as an n-player game, where each policy competes against a population of opponents while being regularized toward a reference model. We demonstrate that MNPO inherits the equilibrium guarantees of two-player methods while enabling richer competitive dynamics and improved coverage of diverse preference structures. Comprehensive empirical evaluation shows that MNPO consistently outperforms existing NLHF baselines on instruction-following benchmarks, achieving superior alignment quality under heterogeneous annotator conditions and mixed-policy evaluation scenarios. Together, these results establish MNPO as a principled and scalable framework for aligning LLMs with complex, non-transitive human preferences. Code is available at: https://github.com/smiles724/MNPO