The Year in AI Papers
The year/Independent research

Paper 2508.05629

On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification

Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, Xu Yang

Published
Aug 2025
Research lab
Independent
Citations
112
GitHub
1.1K stars

01 In brief

Summary

This paper analyzes the generalization gap between Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) for Large Language Models.

The authors show that the SFT gradient can be interpreted as a policy gradient with an implicit, sparse reward that is inversely proportional to the model's token probability, leading to unstable updates and poor generalization.

To address this, they propose Dynamic Fine-Tuning (DFT), which rescales the SFT loss by the token probability (using a stop-gradient), effectively neutralizing the inverse-probability weighting.

This one-line change stabilizes training and improves generalization.

Experiments on math reasoning (Qwen, LLaMA, DeepSeekMath), code generation, and multimodal tasks show DFT consistently outperforms standard SFT, often by large margins, and also surpasses offline RL methods like DPO and RFT, and even online methods like GRPO and PPO in some settings.

However, DFT underperforms SFT on factual knowledge tasks (e.g., Natural Questions), indicating it is best suited for reasoning tasks.

The method is simple, efficient, and requires no additional reward models or sampling, offering a practical alternative to RL-based fine-tuning.

02 From the paper

Abstract

In this work, we present a simple yet theoretically motivated improvement to Supervised Fine-Tuning (SFT) for the Large Language Model (LLM), addressing its limited generalization compared to reinforcement learning (RL). Through mathematical analysis, we reveal that standard SFT gradients implicitly encode a problematic reward structure that may severely restrict the generalization capabilities of model compared to RL. To rectify this, we propose Dynamic Fine-Tuning (\model), stabilizing gradient updates for each token by dynamically rescaling the objective function with the probability of this token. With just a single-line change, the method outperforms standard SFT on multiple difficult benchmarks and base models, from math reasoning to code generation and multi-modal tasks, demonstrating improved generalization. Additionally, \model~achieves competitive results in offline RL settings, providing an effective yet streamlined alternative. By bridging theoretical insights with practical solutions, this work advances the state of SFT. The source code will be available at https://github.com/yongliang-wu/DFT.