Paper 2511.21667
Escaping the Verifier: Learning to Reason via Demonstrations
- Published
- Nov 2025
- Research lab
- Together AI
- Citations
- 5
- GitHub
- Not linked
01 In brief
Summary
The paper introduces RARO (Relativistic Adversarial Reasoning Optimization), a method to train large language models (LLMs) to reason using only expert demonstrations, without task-specific verifiers or human preferences.
RARO frames the problem as inverse reinforcement learning, setting up an adversarial game between a policy and a relativistic critic that share the same LLM.
The critic compares expert and policy answer pairs, outputting which is better or a tie, while the policy learns to produce expert-like answers.
Both are trained jointly via GRPO, with key stabilizations including a replay buffer, tie option, and shared weights.
Experiments on Countdown, DeepMath, and Poetry Writing show RARO significantly outperforms verifier-free baselines (e.g., +13.7% accuracy on Countdown, +8.2% on DeepMath, +19.1% win-rate on Poetry) and nearly matches RL with verifiable rewards (RLVR) on Countdown.
RARO also exhibits scaling trends similar to RLVR with model size and test-time compute, and enables test-time scaling via a critic-based tournament.
Ablations confirm each component is essential.
The method offers a practical pathway for reasoning learning in non-verifiable domains, with future work on stability, sample efficiency, and interpretability.
02 From the paper
Abstract
Training Large Language Models (LLMs) to reason often relies on Reinforcement Learning (RL) with task-specific verifiers. However, many real-world reasoning-intensive tasks lack verifiers, despite offering abundant expert demonstrations that remain under-utilized for reasoning-focused training. We introduce RARO (Relativistic Adversarial Reasoning Optimization), which learns strong reasoning capabilities from expert demonstrations alone via Inverse Reinforcement Learning. RARO sets up an adversarial game between a policy and a relativistic critic: the policy learns to mimic expert answers, while the critic aims to identify the experts among expert-policy answer pairs. Both the policy and the critic are trained jointly and continuously via RL, and we identify the key stabilization techniques required for robust learning. Empirically, RARO significantly outperforms strong verifier-free baselines across all evaluation tasks: +13.7% accuracy on Countdown (1.5B), +8.2% accuracy on DeepMath (7B), and +19.1% win-rate on Poetry Writing (7B) against expert poems. RARO also exhibits similar robust scaling trends as RL with verifiers. These results demonstrate that RARO effectively elicits strong reasoning performance from expert demonstrations alone, enabling robust reasoning learning even when task-specific verifiers are unavailable.