The Year in AI Papers
The year/Qwen

Paper 2601.06487

ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Ranking

Qiang Zhang, Boli Chen, Fanrui Zhang, Ruixue Ding, Shihang Wang, Qiuchen Wang, Yinfeng Huang, Haonan Zhang, Rongxiang Zhu, Pengyong Wang, Ailin Ren, Xin Li, et al.

Published
Jan 2026
Research lab
Qwen
Citations
9
GitHub
274 stars

01 In brief

Summary

ArenaRL is a reinforcement learning framework for open-ended LLM agents that replaces pointwise scalar rewards with tournament-based relative ranking to overcome discriminative collapse, where reward models fail to distinguish subtle differences among high-quality trajectories.

It introduces a process-aware pairwise evaluation mechanism and a seeded single-elimination tournament that achieves O(N) complexity while matching the accuracy of exhaustive O(N^2) comparisons.

The paper also presents two new benchmarks, Open-Travel and Open-DeepResearch, with full SFT-RL-evaluation pipelines.

Experiments show ArenaRL substantially outperforms GRPO and GSPO baselines on these benchmarks and on open-ended writing tasks, with win rates of 41.8% on Open-Travel and 64.3% on Open-DeepResearch, and a 99% valid generation rate.

The method also improves performance on real-world business tasks from the Amap ecosystem.

02 From the paper

Abstract

Reinforcement learning has substantially improved the performance of LLM agents on tasks with verifiable outcomes, but it still struggles on open-ended agent tasks with vast solution spaces (e.g., complex travel planning). Due to the absence of objective ground-truth for these tasks, current RL algorithms largely rely on reward models that assign scalar scores to individual responses. We contend that such pointwise scoring suffers from an inherent discrimination collapse: the reward model struggles to distinguish subtle advantages among different trajectories, resulting in scores within a group being compressed into a narrow range. Consequently, the effective reward signal becomes dominated by noise from the reward model, leading to optimization stagnation. To address this, we propose ArenaRL, a reinforcement learning paradigm that shifts from pointwise scalar scoring to intra-group relative ranking. ArenaRL introduces a process-aware pairwise evaluation mechanism, employing multi-level rubrics to assign fine-grained relative scores to trajectories. Additionally, we construct an intra-group adversarial arena and devise a tournament-based ranking scheme to obtain stable advantage signals. Empirical results confirm that the built seeded single-elimination scheme achieves nearly equivalent advantage estimation accuracy to full pairwise comparisons with O(N^2) complexity, while operating with only O(N) complexity, striking an optimal balance between efficiency and precision. Furthermore, to address the lack of full-cycle benchmarks for open-ended agents, we build Open-Travel and Open-DeepResearch, two high-quality benchmarks featuring a comprehensive pipeline covering SFT, RL training, and multi-dimensional evaluation. Extensive experiments show that ArenaRL substantially outperforms standard RL baselines, enabling LLM agents to generate more robust solutions for complex real-world tasks.