The Year in AI Papers
The year/Independent research

Paper 2607.25659

CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization

Bo-Wen Zhang, Junwei He, Wen Wang, Song-Lin Lv, Wentao Ma, Rongyi Lin, Shuhan Zhong, Lan-Zhe Guo

Published
Jul 2026
Research lab
Independent
Citations
0
GitHub
Not linked

01 In brief

Summary

CoRT (Counterfactual Replay for Token-level credit weighting) is a method for rubric-conditioned GRPO that addresses the uniform token-level credit assignment in standard GRPO.

It rescores the same sampled response under a criteria-free prompt, computes tokenwise log-likelihood contrasts, and maps these to bounded, response-normalized weights.

These weights redistribute the signed GRPO advantage across tokens, emphasizing rubric-dependent tokens without changing the response-level reward or training an auxiliary scorer.

Experiments on instruction-following benchmarks (IFBench, IFEval, MultiDimIF, AdvancedIF) with Qwen3-4B, Qwen2.5-7B, and Qwen3-14B models show CoRT improves over response-level GRPO in most comparisons, with an average gain of 4.4 percentage points, and remains competitive with learned token-level credit baselines like RTT.

CoRT also integrates with DAPO and GSPO, improving most metrics.

Training diagnostics show that response normalization and a SmoothStep ramp are crucial for stability, preventing scale drift and abrupt updates.

CoRT preserves general capabilities and offers a lightweight alternative to explicit token relevance learning.

02 From the paper

Abstract

Rubric-based reinforcement learning enriches language model training by evaluating model outputs against explicit criteria. Yet in GRPO-style pipelines, these structured judgments are reduced to a scalar response-level reward and converted into a response-level advantage, which is broadcast uniformly to all generated tokens. This leaves no explicit mechanism for allocating credit within a response, even when different criteria are grounded in different spans, formatting decisions, or semantic choices. We propose CoRT, a token-level credit weighting method for rubric-conditioned GRPO. Instead of training an auxiliary token scoring model, CoRT uses counterfactual replay to rescore the same sampled response under the original rubric-conditioned prompt and a matched criteria-free prompt. The resulting tokenwise log-likelihood contrasts serve as a proxy for dependence on the rubric context. CoRT maps these contrasts to bounded, response-normalized weights and uses them to redistribute the signed GRPO advantage across tokens, without introducing an auxiliary scorer or changing the response-level reward. Experiments across instruction-tuned models and reward granularities show that CoRT improves over matched response-level GRPO in the vast majority of comparisons, with an average gain of 4.4 percentage points. The method remains competitive with learned token-level credit baselines while avoiding a separate relevance-learning stage. These results suggest that policy-internal counterfactual likelihood contrasts provide an effective training signal for within-response credit allocation while retaining the simplicity and stability of GRPO.