The Year in AI Papers
The year/Independent research

Paper 2606.29526

The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning

Jing Liang, Hongyao Tang, Yi Ma, Yancheng He, Weixun Wang, Xiaoyang Li, Ju Huang, Wenbo Su, Jinyi Liu, Yan Zheng, Jianye Hao, Bo Zheng

Published
Jun 2026
Research lab
Independent
Citations
0
GitHub
Not linked

01 In brief

Summary

The paper addresses training-inference mismatch in LLM reinforcement learning, where separate training and inference engines produce inconsistent probabilities for the same trajectories, causing off-policyness and training instability.

The authors identify an objective misalignment: improving the training policy does not guarantee improvement of the inference policy used in deployment.

They propose Monotonic Inference Policy Improvement (MIPI), a principle that optimizes the inference policy's performance monotonically.

They introduce Monotonic Inference Policy Update (MIPU), a two-step framework: Step 1 constructs sampler-referenced candidate updates using a truncated importance weight (TIS-style), and Step 2 accepts or rejects synchronized candidates based on an inference-gap proxy (post-update gap).

Experiments under FP8-quantized rollout on Qwen3-4B and Qwen3-1.7B show MIPU achieves better average pass@1 accuracy (66.71% and 53.97%) and more stable training than baselines (GRPO, MIS, LR-decay).

Ablations confirm Step 1 improves candidate quality, while Step 2 filters unreliable updates; random rollback control shows Step 2's benefit is not merely from rejecting more updates.

The method reduces risk of accumulating updates whose gains are not realized by the inference policy, though without formal monotonic guarantees.

02 From the paper

Abstract

Reinforcement learning (RL) has gained growing attention in large language model (LLM) post-training, yet RL training remains fragile and can suffer from instability or collapse. One vital cause is training-inference mismatch: LLM adopts separate inference and training engines for generation efficiency and training precision, which in practice exhibits inconsistent probabilities for the same trajectories on training and inference sides, even with synchronized model parameters. This naturally induces a special type of off-policyness ever existing and poisoning the training. Prior works have made various efforts in addressing the off-policyness to stabilize the training policies under the mismatch. In this paper, we point out the objective misalignment neglected by existing works that an effective update to the policy in the training engine not necessarily ensures the improvement of the inference policy, i.e., the one used in deployment. To this end, we propose a new policy optimization objective for LLM RL, named Monotonic Inference Policy Improvement (MIPI). Following this principle, we introduce Monotonic Inference Policy Update (MIPU), a two-step LLM RL framework that constructs sampler-referenced candidate updates and selectively accepts synchronized candidates using an inference-side gap proxy. Experiments conducted on two model scales under high mismatch show that MIPU improves average reasoning performance and training stability.