Paper 2604.06268
RAGEN-2: Reasoning Collapse in Agentic RL
- Published
- Apr 2026
- Research lab
- Independent
- Citations
- 10
- GitHub
- 2.8K stars
01 In brief
Summary
RAGEN-2 identifies a failure mode in multi-turn LLM agent reinforcement learning called template collapse, where reasoning appears diverse within inputs but becomes input-agnostic across inputs, invisible to entropy-based metrics.
The authors decompose reasoning quality into within-input diversity (conditional entropy) and cross-input distinguishability (mutual information, MI), proposing MI proxies like Retrieval-Acc and MI-ZScore-EMA for diagnosis.
They explain template collapse via a signal-to-noise ratio (SNR) mechanism: low within-input reward variance weakens task gradients, allowing input-agnostic regularization (KL, entropy) to dominate and erase cross-input reasoning differences.
To mitigate this, they introduce SNR-Aware Filtering, which selects high-signal prompts per iteration based on reward variance.
Experiments across planning, math reasoning, web navigation, and code execution show that MI correlates more strongly with task performance than entropy, and SNR-Aware Filtering consistently improves both input dependence and task performance across algorithms, model scales, and modalities.
02 From the paper
Abstract
RL training of multi-turn LLM agents is inherently unstable, and reasoning quality directly determines task performance. Entropy is widely used to track reasoning stability. However, entropy only measures diversity within the same input, and cannot tell whether reasoning actually responds to different inputs. In RAGEN-2, we find that even with stable entropy, models can rely on fixed templates that look diverse but are input-agnostic. We call this template collapse, a failure mode invisible to entropy and all existing metrics. To diagnose this failure, we decompose reasoning quality into within-input diversity (Entropy) and cross-input distinguishability (Mutual Information, MI), and introduce a family of mutual information proxies for online diagnosis. Across diverse tasks, mutual information correlates with final performance much more strongly than entropy, making it a more reliable proxy for reasoning quality. We further explain template collapse with a signal-to-noise ratio (SNR) mechanism. Low reward variance weakens task gradients, letting regularization terms dominate and erase cross-input reasoning differences. To address this, we propose SNR-Aware Filtering to select high-signal prompts per iteration using reward variance as a lightweight proxy. Across planning, math reasoning, web navigation, and code execution, the method consistently improves both input dependence and task performance.