The Year in AI Papers
The year/Independent research

Paper 2601.08763

Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs

Zhiyuan Hu, Yucheng Wang, Yufei He, Jiaying Wu, Yilun Zhao, See-Kiong Ng, Cynthia Breazeal, Anh Tuan Luu, Hae Won Park, Bryan Hooi

Published
Jan 2026
Research lab
Independent
Citations
9
GitHub
Not linked

01 In brief

Summary

The paper introduces Uniqueness-Aware Reinforcement Learning (UARL) to address exploration collapse in RL-trained LLMs, where policies prematurely converge to a few dominant reasoning patterns, improving pass@1 but limiting pass@k.

UARL operates at the rollout level, using an LLM judge to cluster solutions by high-level strategy and reweighting GRPO advantages inversely with cluster size, rewarding correct but rare strategies.

Experiments on Qwen2.5-7B, Qwen3-8B, and OLMo-3-7B across math (AIME, HLE), physics (OlympiadBench), and medicine (MedCaseReasoning) show consistent improvements in pass@k and AUC@K up to k=256 without sacrificing pass@1, with higher entropy during training and better coverage of human-annotated solution strategies.

The method outperforms baselines like SimpleRL, DAPO, and Forking Token, especially on hard benchmarks, and demonstrates sustained exploration and strategy diversity at scale.

Limitations include reliance on an LLM judge and measuring rarity only within a single problem's rollout set.

02 From the paper

Abstract

Reinforcement learning (RL) has become a central paradigm for post-training large language models (LLMs), particularly for complex reasoning tasks, yet it often suffers from exploration collapse: policies prematurely concentrate on a small set of dominant reasoning patterns, improving pass@1 while limiting rollout-level diversity and gains in pass@k. We argue that this failure stems from regularizing local token behavior rather than diversity over sets of solutions. To address this, we propose Uniqueness-Aware Reinforcement Learning, a rollout-level objective that explicitly rewards correct solutions that exhibit rare high-level strategies. Our method uses an LLM-based judge to cluster rollouts for the same problem according to their high-level solution strategies, ignoring superficial variations, and reweights policy advantages inversely with cluster size. As a result, correct but novel strategies receive higher rewards than redundant ones. Across mathematics, physics, and medical reasoning benchmarks, our approach consistently improves pass@$k$ across large sampling budgets and increases the area under the pass@$k$ curve (AUC@$K$) without sacrificing pass@1, while sustaining exploration and uncovering more diverse solution strategies at scale.