The Year in AI Papers
The year/Independent research

Paper 2510.00938

Large Reasoning Models Learn Better Alignment from Flawed Thinking

ShengYun Peng, Eric Smith, Ivan Evtimov, Song Jiang, Pin-Yu Chen, Hongyuan Zhan, Haozhu Wang, Duen Horng Chau, Mahesh Pasupuleti, Jianfeng Chi

Published
Oct 2025
Research lab
Independent
Citations
10
GitHub
Not linked

01 In brief

Summary

Large reasoning models (LRMs) generate chain-of-thought (CoT) before answering but are easily biased by flawed reasoning prefills, leading to unsafe or overrefused outputs.

The paper introduces RECAP (Robust Safety Alignment via Counter-Aligned Prefilling), a reinforcement learning (RL) post-training method that trains models to override flawed reasoning trajectories.

RECAP prefills CoT with counter-aligned traces (unsafe for harmful prompts, over-conservative for benign ones) during training, forcing models to recover safe and helpful responses.

It requires no extra cost beyond vanilla RLHF and improves safety and jailbreak robustness while reducing overrefusal and preserving math reasoning.

Experiments on DeepSeek-distilled Llama-8B and Qwen-14B show average gains of +12.3% on direct harmful benchmarks, +21.0% on jailbreaking, and +7.8% on helpfulness, with maintained inference token budget.

RECAP-trained models show increased self-reflection and remain robust under adaptive attacks like full CoT hijacking and iterative prefill reset.

Ablations show counter-aligned prefills are essential, and the ratio and length of prefills control the safety-overrefusal trade-off.

02 From the paper

Abstract

Large reasoning models (LRMs) "think" by generating structured chain-of-thought (CoT) before producing a final answer, yet they still lack the ability to reason critically about safety alignment and are easily biased when a flawed premise is injected into their thought process. We propose RECAP (Robust Safety Alignment via Counter-Aligned Prefilling), a principled reinforcement learning (RL) method for post-training that explicitly teaches models to override flawed reasoning trajectories and reroute to safe and helpful responses. RECAP trains on a mixture of synthetically generated counter-aligned CoT prefills and standard prompts, requires no additional training cost or modifications beyond vanilla reinforcement learning from human feedback (RLHF), and substantially improves safety and jailbreak robustness, reduces overrefusal, and preserves core reasoning capability -- all while maintaining inference token budget. Extensive analysis shows that RECAP-trained models engage in self-reflection more frequently and remain robust under adaptive attacks, preserving safety even after repeated attempts to override their reasoning.