Bottom-up Policy Optimization: Your Language Model Policy Secretly Contains Internal Policies

This paper introduces Bottom-up Policy Optimization (BuPO), a novel reinforcement learning (RL) paradigm for large language models (LLMs) that optimizes internal layer policies before the full language model policy.

The authors decompose the LLM policy into Internal Layer Policies and Internal Modular Policies via the Transformer's residual stream.

Entropy analysis reveals that internal policies universally transition from high-entropy exploration in early layers to deterministic convergence in top layers, but with distinct patterns: Qwen models show progressive reasoning, while Llama models converge abruptly.

They also discover that optimizing internal layers induces feature refinement, where lower layers capture high-level reasoning representations early.

BuPO aligns internal layer policies during early training stages, then applies standard GRPO.

Experiments on complex reasoning benchmarks (MATH500, AMC23, AIME24, AIME25) show consistent improvements over baselines like GRPO, PPO, Reinforce++, and RLOO across Qwen3 and Llama-OctoThinker models.

Ablations show that the optimal internal layer corresponds to the last layer with positive FFN entropy change (the 'Boundary Layer'), and that moderate internal alignment steps (e.g., 30) are best.

The method also generalizes to out-of-domain tasks, and the internal policy analysis is computationally cheap (about 0.44% of RL training time).