Paper 2605.21467
DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards
- Published
- May 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- 17 stars
01 In brief
Summary
The paper introduces DelTA, a method for improving reinforcement learning from verifiable rewards (RLVR) in large language models.
The authors show that sequence-level RLVR updates act as an implicit linear discriminator over token-gradient vectors, determining which token probabilities increase or decrease.
Standard RLVR forms this discriminator using advantage-weighted centroids of token gradients, which can be dominated by shared high-frequency patterns (e.g., formatting tokens), diluting discriminative directions.
DelTA reweights token-gradient terms by estimating coefficients that amplify side-specific directions and downweight shared ones, reshaping the update direction.
Experiments on seven mathematical benchmarks show DelTA outperforms the strongest same-scale baselines by 3.26 points on Qwen3-8B-Base and 2.62 points on Qwen3-14B-Base.
Additional results on code generation, a different backbone (Olmo3-7B), and out-of-domain evaluations demonstrate generalization.
Ablations confirm the necessity of opposite-side comparison, the usefulness of token coefficients, and the contribution of each design component.
The method adds modest computational overhead (about 10% of first-step time).
02 From the paper
Abstract
Reinforcement learning from verifiable rewards (RLVR) has emerged as a central technique for improving the reasoning capabilities of large language models. Despite its effectiveness, how response-level rewards translate into token-level probability changes remains poorly understood. We introduce a discriminator view of RLVR updates, showing that the policy-gradient update direction implicitly acts as a linear discriminator over token-gradient vectors and thereby determines which token probabilities are increased or decreased during learning. Under standard sequence-level RLVR, this discriminator is constructed from positive- and negative-side centroids formed by advantage-weighted averaging of token-gradient vectors. However, such centroid construction can be dominated by shared high-frequency patterns, such as formatting tokens, diluting sparse yet discriminative directions that better distinguish high-reward responses from low-reward ones. To address this limitation, we propose $\textbf{DelTA}$, a discriminative token credit assignment method that estimates token coefficients to amplify side-specific token-gradient directions and downweight shared or weakly discriminative ones. These coefficients reweight a self-normalized RLVR surrogate, making the effective side-wise centroids more contrastive and thereby reshaping the RLVR update direction. On seven mathematical benchmarks, DelTA outperforms the strongest same-scale baselines by 3.26 and 2.62 average points on Qwen3-8B-Base and Qwen3-14B-Base, respectively. Additional results on code generation, a different backbone, and out-of-domain evaluations further demonstrate the generalization ability of DelTA.