The Year in AI Papers
The year/Google DeepMind

Paper 2605.10899

RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards

Gaotang Li, Bhavana Dalvi Mishra, Zifeng Wang, Jun Yan, Yanfei Chen, Chun-Liang Li, Long T. Le, Rujun Han, George Lee, Hanghang Tong, Chen-Yu Lee, Tomas Pfister

Published
May 2026
Research lab
Google DeepMind
Citations
3
GitHub
Not linked

01 In brief

Summary

RubricEM is a reinforcement learning framework for training deep research agents on open-ended tasks where verifiable rewards are unavailable.

It treats rubrics as a shared interface structuring policy execution, judge feedback, and agent memory.

The framework decomposes trajectories into four rubric-guided stages (Plan, Research, Review, Answer) and uses Stage-Structured GRPO (SS-GRPO) for denser, stagewise credit assignment.

A shared-backbone reflection meta-policy is jointly trained to distill judged trajectories into reusable rubric-grounded guidance, stored in a rubric bank for within-episode refinement and cross-episode transfer.

An asynchronous pipeline avoids training bottlenecks.

RubricEM-8B, trained with 1400 RL steps, achieves state-of-the-art performance among open models on four long-form benchmarks (HealthBench, ResearchQA, DeepResearchBench, ResearchRubrics), averaging 55.5, surpassing DR Tulu-8B-RL and approaching proprietary systems like OpenAI Deep Research.

Ablations confirm that both SS-GRPO and meta-policy training contribute gains, and the model transfers well to short-form benchmarks.

Theoretical analyses formalize the value of stage information and judge-aligned credit assignment.

02 From the paper

Abstract

Training deep research agents, namely systems that plan, search, evaluate evidence, and synthesize long-form reports, pushes reinforcement learning beyond the regime of verifiable rewards. Their outputs lack ground-truth answers, their trajectories span many tool-augmented decisions, and standard post-training offers little mechanism for turning past attempts into reusable experience. In this work, we argue that rubrics should serve not merely as final-answer evaluators, but as the shared interface that structures policy execution, judge feedback, and agent memory. Based on this view, we introduce RubricEM, a rubric-guided reinforcement learning framework that combines stagewise policy decomposition with reflection-based meta-policy evolution. RubricEM first makes research trajectories stage-aware by conditioning planning, evidence gathering, review, and synthesis on self-generated rubrics. It then assigns credit with Stage-Structured GRPO, which uses stagewise rubric judgments to provide denser semantic feedback for long-horizon optimization. In parallel, RubricEM trains a shared-backbone reflection meta-policy that distills judged trajectories into reusable rubric-grounded guidance for future attempts. The resulting RubricEM-8B achieves strong performance across four long-form research benchmarks, outperforming comparable open models and approaching proprietary deep-research systems. Beyond final performance, we perform thorough analyses to understand the key ingredients of RubricEM.