Paper 2512.02472
Guided Self-Evolving LLMs with Minimal Human Supervision
- Published
- Dec 2025
- Research lab
- Independent
- Citations
- 31
- GitHub
- Not linked
01 In brief
Summary
The paper introduces R-FEW, a guided self-evolving framework for large language models (LLMs) that uses minimal human supervision to achieve stable and controllable self-improvement.
R-FEW addresses issues like concept drift and diversity collapse in unguided self-play methods (e.g., R-Zero) by integrating a few-shot grounded Challenger and an online curriculum Solver.
The Challenger samples 1-5% of human-labeled anchor data from WebInstruct to guide synthetic question generation, while the Solver selects mid-difficulty samples (based on success rates) from both synthetic and human data for training.
Experiments on Qwen3-4B/8B-Base show consistent improvements: Qwen3-8B-Base improves by +3.0 points over R-Zero on math tasks and matches General-Reasoner, which uses 20x more human data.
Ablations confirm the importance of challenger training, warm-up, and curriculum learning.
Analysis shows R-FEW mitigates diversity collapse and reward hacking (e.g., length inflation) compared to R-Zero.
The framework achieves performance on par with or exceeding heavily supervised pipelines, demonstrating the power of lightweight human grounding.
02 From the paper
Abstract
AI self-evolution has long been envisioned as a path toward superintelligence, where models autonomously acquire, refine, and internalize knowledge from their own learning experiences. Yet in practice, unguided self-evolving systems often plateau quickly or even degrade as training progresses. These failures arise from issues such as concept drift, diversity collapse, and mis-evolution, as models reinforce their own biases and converge toward low-entropy behaviors. To enable models to self-evolve in a stable and controllable manner while minimizing reliance on human supervision, we introduce R-Few, a guided Self-Play Challenger-Solver framework that incorporates lightweight human oversight through in-context grounding and mixed training. At each iteration, the Challenger samples a small set of human-labeled examples to guide synthetic question generation, while the Solver jointly trains on human and synthetic examples under an online, difficulty-based curriculum. Across math and general reasoning benchmarks, R-Few achieves consistent and iterative improvements. For example, Qwen3-8B-Base improves by +3.0 points over R-Zero on math tasks and achieves performance on par with General-Reasoner, despite the latter being trained on 20 times more human data. Ablation studies confirm the complementary contributions of grounded challenger training and curriculum-based solver training, and further analysis shows that R-Few mitigates drift, yielding more stable and controllable co-evolutionary dynamics.