Paper 2508.05004
R-Zero: Self-Evolving Reasoning LLM from Zero Data
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 167
- GitHub
- 831 stars
01 In brief
Summary
R-Zero is a fully autonomous framework for training reasoning LLMs without any pre-existing tasks or labels.
It initializes two models from a single base LLM: a Challenger, trained via GRPO to generate questions at the edge of the Solver's capability (using an uncertainty reward based on self-consistency, plus repetition and format penalties), and a Solver, trained via GRPO on filtered challenging questions with pseudo-labels from majority voting.
This co-evolutionary loop iteratively improves reasoning.
Experiments on Qwen3 and OctoThinker models show significant gains on math benchmarks (e.g., +6.49 for Qwen3-4B) and general-domain reasoning (e.g., +7.54), with generalization to MMLU-Pro and SuperGPQA.
Ablations confirm the importance of repetition penalty and task filtering.
The framework also synergizes with supervised fine-tuning as a mid-training method.
However, performance eventually degrades after multiple iterations, with larger models delaying collapse, and pseudo-label accuracy decreases as questions become harder.
Limitations include reliance on verifiable domains and potential model collapse from self-generated data.
02 From the paper
Abstract
Self-evolving Large Language Models (LLMs) offer a scalable path toward super-intelligence by autonomously generating, refining, and learning from their own experiences. However, existing methods for training such models still rely heavily on vast human-curated tasks and labels, typically via fine-tuning or reinforcement learning, which poses a fundamental bottleneck to advancing AI systems toward capabilities beyond human intelligence. To overcome this limitation, we introduce R-Zero, a fully autonomous framework that generates its own training data from scratch. Starting from a single base LLM, R-Zero initializes two independent models with distinct roles, a Challenger and a Solver. These models are optimized separately and co-evolve through interaction: the Challenger is rewarded for proposing tasks near the edge of the Solver capability, and the Solver is rewarded for solving increasingly challenging tasks posed by the Challenger. This process yields a targeted, self-improving curriculum without any pre-existing tasks and labels. Empirically, R-Zero substantially improves reasoning capability across different backbone LLMs, e.g., boosting the Qwen3-4B-Base by +6.49 on math-reasoning benchmarks and +7.54 on general-domain reasoning benchmarks.