The Year in AI Papers
The year/Independent research

Paper 2508.07050

ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability

Wenhan Liu, Xinyu Ma, Weiwei Sun, Yutao Zhu, Yuchen Li, Dawei Yin, Zhicheng Dou

Published
Aug 2025
Research lab
Independent
Citations
47
GitHub
Not linked

01 In brief

Summary

ReasonRank is a reasoning-intensive listwise passage reranker that outperforms existing baselines on reasoning-heavy IR benchmarks.

The authors address the scarcity of reasoning-intensive training data by proposing an automated synthesis framework that collects queries from diverse domains (complex QA, coding, math, web search) and uses DeepSeek-R1 to generate pointwise and listwise labels, followed by a self-consistency filtering mechanism, yielding 13K high-quality samples.

They then train the model in two stages: cold-start supervised fine-tuning (SFT) to learn reasoning patterns, and reinforcement learning (RL) with a novel multi-view ranking reward combining NDCG@10, Recall@10, and Rank-Biased Overlap (RBO) to account for the sliding-window ranking strategy.

Experiments on BRIGHT and R2MED show ReasonRank (7B and 32B) significantly outperforms prior reasoning rerankers like Rank1, Rank-R1, and Rank-K, with the 7B model surpassing 32B baselines.

It also achieves 2-2.7x lower latency than the pointwise Rank1.

Ablations confirm the importance of diverse training data, self-consistency filtering, SFT, RL, and the multi-view reward.

On BEIR, ReasonRank (32B) also achieves the best average NDCG@10, demonstrating generalization to traditional IR tasks.

The code is available at https://github.com/8421BCD/ReasonRank.

02 From the paper

Abstract

Large Language Model (LLM) based listwise ranking has shown superior performance in many passage ranking tasks. With the development of Large Reasoning Models (LRMs), many studies have demonstrated that step-by-step reasoning during test-time helps improve listwise ranking performance. However, due to the scarcity of reasoning-intensive training data, existing rerankers perform poorly in many complex ranking scenarios, and the ranking ability of reasoning-intensive rerankers remains largely underdeveloped. In this paper, we first propose an automated reasoning-intensive training data synthesis framework, which sources training queries and passages from diverse domains and applies DeepSeek-R1 to generate high-quality training labels. To empower the listwise reranker with strong reasoning ability, we further propose a two-stage training approach, which includes a cold-start supervised fine-tuning (SFT) stage and a reinforcement learning (RL) stage. During the RL stage, we design a novel multi-view ranking reward tailored to the multi-turn nature of listwise ranking. Extensive experiments demonstrate that our trained reasoning-intensive reranker \textbf{ReasonRank} outperforms existing baselines significantly and also achieves much lower latency than the pointwise reranker. Our codes are available at https://github.com/8421BCD/ReasonRank.