Paper 2510.03259
Verifying Meta-Awareness via Predictive Rewards in Reasoning Models
- Published
- Sep 2025
- Research lab
- Independent
- Citations
- 2
- GitHub
- 12 stars
01 In brief
Summary
The paper introduces MAPR (Meta-Awareness via Predictive Reward), a reinforcement learning framework that enhances reasoning models by training them to predict their own rollout statistics—specifically solution length, pass-rate, and concepts used—and rewarding alignment with actual statistics.
This self-generated meta-prediction is verified against empirical outcomes, eliminating the need for external supervision.
MAPR also includes MAPR-efficient, a variant that uses predictive gating to filter trivial or unsolvable prompts, early cutoff to terminate lengthy incorrect rollouts, and notion feed-in to provide hints.
Experiments on Qwen3-4B/8B/14B models show significant gains over GRPO baselines across six math benchmarks, with an average improvement of 13.04% on Qwen3-4B, 8.74% on Qwen3-8B, and 6.63% on Qwen3-14B.
MAPR also speeds up GRPO training by over 1.28× to reach the same performance and achieves an 83.18% accuracy gain on AIME25.
Ablations confirm the contribution of each reward component, and the method generalizes to DAPO and other model families (Llama 3.1, Gemma 2).
02 From the paper
Abstract
Recent research on reasoning models explores the meta-awareness of language models, including their ability to determine optimal thinking duration, recognize knowledge boundaries, and structure concept-level thinking. While current large reasoning models depend solely on answer-based verification, we show that adding meta-awareness objectives leads to significant performance gains over models without such meta-knowledge. MAPR (Meta-Awareness via Predictive Reward) utilizes a self-generated task of predicting rollout statistics - specifically length, pass-rate, and concepts used - allowing for verification against the actual statistics. Furthermore, by leveraging this self-predictive capability, the model can regulate its reasoning behavior by i) filtering out trivial or unsolvable prompts, ii) reducing lengthy generations that tend to be incorrect, and iii) generating hints relevant to the problem. The results are inspiring: MAPR yields significant improvements in both accuracy and training efficiency on various reasoning benchmarks. More specifically, our method can speed up GRPO training by over 1.28x to reach the same performance, and achieve 83.18% gain in accuracy on AIME25, and a 13.04% average gain over six mathematics benchmarks. The code is publicly available at https://github.com/akatigre/MAPR-RL.