Paper 2603.19835
FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization
01 In brief
Summary
FIPO (Future-KL Influenced Policy Optimization) is a reinforcement learning algorithm that addresses the coarse-grained credit assignment problem in GRPO-style training for large language models.
By incorporating a discounted Future-KL divergence into the policy update, FIPO re-weights token advantages based on their influence on subsequent trajectory behavior, creating a dense advantage formulation.
Evaluated on Qwen2.5-32B-Base, FIPO extends average chain-of-thought length from ~4,000 to over 10,000 tokens and increases AIME 2024 Pass@1 accuracy from 50.0% to a peak of 58.0% (converging at ~56.0%), outperforming DeepSeek-R1-Zero-Math-32B (~47.0%) and o1-mini (~56.0%).
The method includes stability mechanisms like influence weight clipping and extreme value filtering, and uses a soft decay window to prioritize proximal signals.
FIPO achieves these gains without a critic model, demonstrating that dense supervision can be realized within the GRPO framework.
The training system is open-sourced, built on the verl framework, and trained on the DAPO-17K dataset with a mini-batch size of 64 for stability.
The paper also discusses limitations including computational cost, task generalization, and performance gaps versus distillation.
02 From the paper
Abstract
We present Future-KL Influenced Policy Optimization (FIPO), a reinforcement learning algorithm designed to overcome reasoning bottlenecks in large language models. While GRPO style training scales effectively, it typically relies on outcome-based rewards (ORM) that distribute a global advantage uniformly across every token in a trajectory. We argue that this coarse-grained credit assignment imposes a performance ceiling by failing to distinguish critical logical pivots from trivial tokens. FIPO addresses this by incorporating discounted future-KL divergence into the policy update, creating a dense advantage formulation that re-weights tokens based on their influence on subsequent trajectory behavior. Empirically, FIPO enables models to break through the length stagnation seen in standard baselines. Evaluated on Qwen2.5-32B, FIPO extends the average chain-of-thought length from roughly 4,000 to over 10,000 tokens and increases AIME 2024 Pass@1 accuracy from 50.0% to a peak of 58.0% (converging at approximately 56.0\%). This outperforms both DeepSeek-R1-Zero-Math-32B (around 47.0%) and o1-mini (approximately 56.0%). Our results suggest that establishing dense advantage formulations is a vital path for evolving ORM-based algorithms to unlock the full reasoning potential of base models. We open-source our training system, built on the verl framework.