Paper 2602.10693
VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Training
- Published
- Feb 2026
- Research lab
- Independent
- Citations
- 7
- GitHub
- 35 stars
01 In brief
Summary
VESPO addresses high variance in off-policy reinforcement learning for large language models by deriving a principled importance weight reshaping kernel from a variational formulation.
It recasts weight reshaping as a measure change to an implicit proposal distribution, solving a KL-regularized objective with a variance constraint to obtain a closed-form kernel: φ(W) = W^c1 * exp(c2(1-W)).
This kernel operates directly on sequence-level importance weights without length normalization, preserving inter-token dependencies, and admits an explicit variance bound ensuring bounded gradient contributions under arbitrary staleness.
Experiments on math reasoning (DAPO-Math) and code generation (PRIME-RL) across dense (Llama-3.2-3B, Qwen3-8B) and MoE (Qwen3-30B-A3B) models show VESPO maintains stable training under staleness up to 64×, outperforming baselines like GRPO, GSPO, SAPO, TOPR, CISPO, and BAPO.
On Qwen3-30B-A3B-Base, VESPO achieves 70.0% average math accuracy vs.
59.1% for the best baseline, and 63.0% pass@10 on code vs.
61.8% for SAPO, with no retuning between domains.
The method is a drop-in replacement in standard RL pipelines, requiring no extra forward passes or memory.
02 From the paper
Abstract
Off-policy updates are inevitable in reinforcement learning (RL) for large language models (LLMs) due to rollout staleness from asynchronous training and mismatches between training and inference engines. Naive importance sampling gives an unbiased correction but suffers from high variance, which is amplified by unbounded ratios and autoregressive generation. Prior remedies either rely on scenario-specific engineering, or trade bias for variance via token-level clipping or sequence-level normalization, yet these approaches remain largely heuristic. We propose Variational sEquence-level Soft Policy Optimization (VESPO). By explicitly incorporating variance reduction into a variational formulation, we derive a principled closed-form reshaping kernel that operates directly on sequence-level importance weights, avoids token-level approximation and length normalization, and admits an explicit variance bound for the deployed kernel. Experiments on math reasoning and code generation show that VESPO maintains stable training under severe off-policy conditions (staleness up to 64x) and delivers consistent gains across both dense and Mixture-of-Experts (MoE) models, outperforming recent reshaping baselines under matched setup. Code is available at https://github.com/FloyedShen/VESPO.