Paper 2509.04419
Towards a Unified View of Large Language Model Post-Training
- Published
- Sep 2025
- Research lab
- Independent
- Citations
- 39
- GitHub
- 211 stars
01 In brief
Summary
This paper introduces a unified theoretical framework for large language model (LLM) post-training, showing that Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) are not contradictory but instances of a single optimization process.
The authors derive a Unified Policy Gradient Estimator (UPGE) that subsumes the gradients of various post-training algorithms (SFT, PPO, GRPO, REINFORCE, CISPO, GSPO, SRFT, LUFFY) into one expression with four interchangeable components: stabilization mask, reference policy denominator, advantage estimate, and likelihood gradient.
They demonstrate that SFT and RL optimize a common objective with different bias-variance tradeoffs.
Based on this, they propose Hybrid Post-Training (HPT), an algorithm that dynamically switches between SFT and RL losses based on real-time rollout performance (using a gate threshold γ).
Experiments on six mathematical reasoning benchmarks and two out-of-distribution suites show that HPT consistently outperforms strong baselines (SFT, GRPO, SFT→GRPO, LUFFY, SRFT) across models of varying scales (Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, LLaMA3.1-8B).
Key findings include that HPT achieves the highest Pass@1024, indicating enhanced exploration, and that dynamic integration of SFT and RL improves both exploitation and exploration, with γ=0 yielding the best average performance (41.9 on Qwen2.5-Math-1.5B).
02 From the paper
Abstract
Two major sources of training data exist for post-training modern language models: online (model-generated rollouts) data, and offline (human or other-model demonstrations) data. These two types of data are typically used by approaches like Reinforcement Learning (RL) and Supervised Fine-Tuning (SFT), respectively. In this paper, we show that these approaches are not in contradiction, but are instances of a single optimization process. We derive a Unified Policy Gradient Estimator, and present the calculations of a wide spectrum of post-training approaches as the gradient of a common objective under different data distribution assumptions and various bias-variance tradeoffs. The gradient estimator is constructed with four interchangeable parts: stabilization mask, reference policy denominator, advantage estimate, and likelihood gradient. Motivated by our theoretical findings, we propose Hybrid Post-Training (HPT), an algorithm that dynamically selects different training signals. HPT is designed to yield both effective exploitation of demonstration and stable exploration without sacrificing learned reasoning patterns. We provide extensive experiments and ablation studies to verify the effectiveness of our unified theoretical framework and HPT. Across six mathematical reasoning benchmarks and two out-of-distribution suites, HPT consistently surpasses strong baselines across models of varying scales and families.