Paper 2604.13902
DiPO: Disentangled Perplexity Policy Optimization for Fine-grained Exploration-Exploitation Trade-Off
- Published
- Apr 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- Not linked
01 In brief
Summary
The paper introduces DiPO (Disentangled Perplexity Policy Optimization), a method for fine-grained exploration-exploitation trade-off in Reinforcement Learning with Verifiable Rewards (RLVR) for LLMs.
It identifies two dilemmas in GRPO-based methods: extreme sample groups (all-correct or all-error) yield zero advantage, and perplexity (PPL) distribution shows ineffective exploration/exploitation.
DiPO proposes Perplexity Space Disentangling (PSD), which uses a PPL queue and statistical estimation to find an optimal threshold τ* that partitions samples into four quadrants (CH, CL, EH, EL) based on PPL and correctness.
It also introduces Bidirectional Reward Reallocation (BRR), which reallocates rewards only for extreme groups: for hard groups in the exploitation space, the maximum-PPL sample's reward is set to 1; for easy groups in the exploration space, the maximum-PPL sample's reward is set to 0, minimizing perturbation to verification rewards.
Experiments on mathematical reasoning (AIME24, AIME25, MATH, AMC, OLY, MIN) and function calling (BFCLv3) show DiPO outperforms baselines like GRPO, DAPO, and CDE across multiple models (Qwen3-4B, Qwen3-8B, Qwen2.5-7B, Qwen2.5-3B/7B-Instruct).
Hyperparameter analysis shows α=0.1 is optimal.
Ablations confirm PSD and BRR contributions.
Quantitative analysis shows DiPO improves PPL distribution separation and maintains higher upper bounds in later training.
02 From the paper
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has catalyzed significant advances in the reasoning capabilities of Large Language Models (LLMs). However, effectively managing the exploration and exploitation trade-off remains a critical challenge. In this paper, we fully analyze the exploration and exploitation dilemma of extremely hard and easy samples during the training and propose a new fine-grained trade-off mechanism. Concretely, we introduce a perplexity space disentangling strategy that divides the sample space into distinct exploration (high perplexity) and exploitation (low perplexity) subspaces, thereby mining fine-grained samples requiring exploration-exploitation trade-off. Subsequently, we propose a bidirectional reward allocation mechanism with a minimum impact on verification rewards to implement perplexity-guided exploration and exploitation, enabling more stable policy optimization. Finally, we have evaluated our method on two mainstream tasks: mathematical reasoning and function calling, and experimental results demonstrate the superiority of the proposed method, confirming its effectiveness in enhancing LLM performance by fine-grained exploration-exploitation trade-off.