Paper 2605.14236
Active Learners as Efficient PRP Rerankers
- Published
- May 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- 7 stars
01 In brief
Summary
The paper reframes Pairwise Ranking Prompting (PRP) reranking as active learning from noisy pairwise comparisons, arguing that classical sorting algorithms are mismatched with noisy, order-sensitive LLM judgments.
The authors propose active rankers, particularly the Mohajer algorithm, as drop-in replacements for sorting in call-constrained settings.
They also introduce a randomized-direction oracle that uses one LLM call per pair instead of two, converting systematic position bias into zero-mean noise.
On TREC DL2019/2020 with Flan-T5-XL, Mohajer outperforms the best sorting baseline by +9.7 NDCG@10 at 300 calls (66.1 vs.
56.4) under the bidirectional oracle.
The randomized-direction oracle improves both strategies: BubbleSort gains +5.5 NDCG@10 at 300 calls, and Mohajer reaches a higher quality ceiling (68.0 vs.
66.96) with 44% fewer calls (250 vs.
450).
Across BEIR-style tasks, active rankers achieve comparable NDCG@10 to QuickSort with up to 7x fewer calls.
The authors conclude that active ranking is preferable when the budget exceeds a warm-up threshold, while sorting is better for very small or very large budgets.
02 From the paper
Abstract
Pairwise Ranking Prompting (PRP) elicits pairwise preference judgments from an LLM, which are then aggregated into a ranking, usually via classical sorting algorithms. However, judgments are noisy, order-sensitive, and sometimes intransitive, so sorting assumptions do not match the setting. Because sorting aims to recover a full permutation, truncating it to meet a call budget does not produce a dependable top-K. We thus reframe PRP reranking as active learning from noisy pairwise comparisons and show that active rankers are drop-in replacements that improve NDCG@10 per call in the call-constrained regime. Our noise-robust framework also introduces a randomized-direction oracle that uses a single LLM call per pair. This approach converts systematic position bias into zero-mean noise, enabling unbiased aggregate ranking without the cost of bidirectional calls.