Paper 2509.22638
Language Models Can Learn from Verbal Feedback Without Scalar Rewards
- Published
- Sep 2025
- Research lab
- Independent
- Citations
- 15
- GitHub
- 65 stars
01 In brief
Summary
The paper proposes Feedback-Conditional Policy (FCP), a method for training LLMs directly from verbal feedback without converting it into scalar rewards.
FCP treats feedback as a conditioning signal, learning a policy πθ(o|x,c) that approximates the posterior P_off(o|x,c) ∝ π_ref(o|x)·p_env(c|x,o) via maximum likelihood on offline data.
An online bootstrapping stage then conditions on positive feedback to generate rollouts, which are re-annotated with fresh feedback to iteratively refine the policy.
Experiments on math (Big-Math) and general reasoning (WebInstruct) with Qwen2.5-7B-base show that FCP matches or surpasses scalar-based baselines (RFT, GRPO) in accuracy, while avoiding reward hacking and preserving feedback richness.
Ablations show that user-style feedback is competitive, length-related conditions can destabilize training, and partial online updates improve performance.
The authors argue that verbal feedback can be a first-class training signal, reframing RL as conditional generation rather than reward optimization.
02 From the paper
Abstract
LLMs are often trained with RL from human or AI feedback, yet such methods typically compress nuanced feedback into scalar rewards, discarding much of their richness and inducing scale imbalance. We propose treating verbal feedback as a conditioning signal. Inspired by language priors in text-to-image generation, which enable novel outputs from unseen prompts, we introduce the feedback-conditional policy (FCP). FCP learns directly from response-feedback pairs, approximating the feedback-conditional posterior through maximum likelihood training on offline data. We further develop an online bootstrapping stage where the policy generates under positive conditions and receives fresh feedback to refine itself. This reframes feedback-driven learning as conditional generation rather than reward optimization, offering a more expressive way for LLMs to directly learn from verbal feedback. Our code is available at https://github.com/sail-sg/feedback-conditional-policy.