CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization
Bo-Wen Zhang, Junwei He, Wen Wang, Song-Lin Lv et al. · arXiv · 2026
Abstract
In rubric-based reinforcement learning a detailed multi-criterion judgment collapses into a single number, which is then spread evenly across every token the model produced — so a formatting choice and the decisive reasoning step receive identical credit. This method recovers some of that lost resolution by re-scoring the same response with and without the rubric in the prompt and using the difference in token likelihoods to redistribute credit, adding no scorer model to train.
Why it matters
Neat engineering: it extracts a token-level signal already latent in the setup instead of bolting on more machinery. Read it for how credit assignment actually works inside GRPO-style training.
https://arxiv.org/abs/2607.25659