Paper 2602.03392
On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models
- Published
- Feb 2026
- Research lab
- Independent
- Citations
- 7
- GitHub
- Not linked
01 In brief
Summary
This paper establishes a theoretical framework for analyzing entropy dynamics during reinforcement fine-tuning (RFT) of large language models.
The authors derive a first-order expression for entropy change from a single logit update, showing that the direction of entropy change is determined by the update direction and a discriminator score S* = p_k(H + log p_k).
They extend this to Group Relative Policy Optimization (GRPO), revealing that entropy change depends on the deviation of S* from its policy-weighted expectation.
Based on this analysis, they propose two entropy-discriminator clipping methods (Clip_B and Clip_V) that filter out outlier tokens to stabilize entropy.
Experiments on Qwen2.5 models with DAPO-Math-17k show that these methods outperform vanilla GRPO on AIME24, AIME25, and DAPO500 benchmarks, improving both Avg@K and Pass@K metrics.
The framework also provides a unified interpretation of existing entropy-based methods, such as clipping mechanisms, entropy regularization, and probability-weighted updating.
The results demonstrate that controlling entropy preserves model exploration and enhances performance.
02 From the paper
Abstract
Entropy serves as a critical metric for measuring the diversity of outputs generated by large language models (LLMs), providing valuable insights into their exploration capabilities. While recent studies increasingly focus on monitoring and adjusting entropy to better balance exploration and exploitation in reinforcement fine-tuning (RFT), a principled understanding of entropy dynamics during this process is yet to be thoroughly investigated. In this paper, we establish a theoretical framework for analyzing the entropy dynamics during the RFT process, which begins with a discriminant expression that quantifies entropy change under a single logit update. This foundation enables the derivation of a first-order expression for entropy change, which can be further extended to the update formula of Group Relative Policy Optimization (GRPO). The corollaries and insights drawn from the theoretical analysis inspire the design of entropy control methods, and also offer a unified lens for interpreting various entropy-based methods in existing studies. We provide empirical evidence to support the main conclusions of our analysis and demonstrate the effectiveness of the derived entropy-discriminator clipping methods. This study yields novel insights into RFT training dynamics, providing theoretical support and practical strategies for optimizing the exploration-exploitation balance during LLM fine-tuning.