Paper 2604.20733
Near-Future Policy Optimization
- Published
- Apr 2026
- Research lab
- Independent
- Citations
- 2
- GitHub
- Not linked
01 In brief
Summary
The paper introduces Near-Future Policy Optimization (NPO), a mixed-policy reinforcement learning with verifiable rewards (RLVR) method that improves training by using trajectories from a near-future checkpoint of the same training run.
The authors formalize a quality-variance trade-off, defining effective learning signal S = Q/V, where Q is the fraction of correct trajectories a source provides on prompts the current policy fails, and V is the variance cost from importance weighting.
They show that a near-future checkpoint balances Q and V better than external teachers (high Q, high V) or past replay (low Q, moderate V).
NPO replaces one rollout slot with a verified-correct trajectory from a near-future checkpoint when the current policy struggles on a prompt.
They validate NPO with manual early-stage and late-stage interventions, and propose AutoNPO, which automatically triggers interventions based on training signals and selects the checkpoint distance maximizing S.
On Qwen3-VL-8B-Instruct with GRPO, NPO improves average performance from 57.88 to 62.84, and AutoNPO to 63.15, while accelerating convergence by ~2.1x.
Ablations show importance-sampling correction is unnecessary for NPO due to the near-policy property.
The work is part of a Self-Taught RLVR research program.
02 From the paper
Abstract
Reinforcement learning with verifiable rewards (RLVR) has become a core post-training recipe. Introducing suitable off-policy trajectories into on-policy exploration accelerates RLVR convergence and raises the performance ceiling, yet finding a source of such trajectories remains the key challenge. Existing mixed-policy methods either import trajectories from external teachers (high-quality but distributionally far) or replay past training trajectories (close but capped in quality), and neither simultaneously satisfies the strong enough (higher $Q$ , more new knowledge to learn) and close enough (lower $V$ , more readily absorbed) conditions required to maximize the effective learning signal $\mathcal{S} = Q/V$. We propose \textbf{N}ear-Future \textbf{P}olicy \textbf{O}ptimization (\textbf{NPO}), a simple mixed-policy scheme that learns from a policy's own near-future self: a later checkpoint from the same training run is a natural source of auxiliary trajectories that is both stronger than the current policy and closer than any external source, directly balancing trajectory quality against variance cost. We validate NPO through two manual interventions, early-stage bootstrapping and late-stage plateau breakthrough, and further propose \textbf{AutoNPO},an adaptive variant that automatically triggers interventions from online training signals and selects the guide checkpoint that maximizes $S$. On Qwen3-VL-8B-Instruct with GRPO, NPO improves average performance from 57.88 to 62.84, and AutoNPO pushes it to 63.15, raising the final performance ceiling while accelerating convergence.