Research collection

Safety and alignment

Making models behave: alignment and preference-tuning methods, jailbreak and adversarial robustness, guardrails and refusal behaviour, and misuse or societal-risk evaluation.

Papers: 12
Research labs: 2
Official code: 4

1–12 of 12 papers in this collection

OpenAI

Predicting LLM Safety Before Release by Simulating Deployment

This paper introduces deployment simulation, a method for predicting LLM safety before release by resampling the next assistant response from de-identified production conversation prefixes using a candidate model. The authors evaluate this approach across GPT-5-series deployments, finding that it produces informative estimates of post-deployment…

Marcus Williams, Hannah Sheahan, Cameron Raymond, Tomek Korbak, et al.

Published: Jul 2026
Citations: 1
Code: Not linked

Read summary Original

bioRxiv

Measuring Epistemic Resilience of LLMs Under Misleading Medical Context

MedMisBench is a benchmark introduced to measure the epistemic resilience of large language models (LLMs) in medical settings, defined as the ability to maintain correct medical judgment when misleading context is present. It contains 10,932 medical question items and 48,889 misleading context-option pairs, built from five source datasets covering medical…

Hongjian Zhou, Xinyu Zou, Jinge Wu, Sean Wu, et al.

Published: Jun 2026
Citations: 1
Code: 9 stars

Read summary Original

Independent research

AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Security

AgentDoG 1.5, developed by Shanghai Artificial Intelligence Laboratory, is a lightweight and scalable framework for AI agent safety and security. It updates the three-dimensional safety taxonomy (risk source, failure mode, real-world harm) to cover new risks from Codex and OpenClaw execution scenarios, extending the ATBench benchmark family with…

Dongrui Liu, Yu Li, Zhonghao Yang, Peng Wang, et al.

Published: May 2026
Citations: 2
Code: Not linked

Read summary Original

arXiv.org

A Survey of Large Audio Language Models: Generalization, Trustworthiness, and Outlook

This survey provides a comprehensive analysis of Large Audio Language Models (LALMs), focusing on their generalization, trustworthiness, and future outlook. It examines the endogenous mechanisms of LALMs, including architectural foundations, representational paradigms, training and alignment strategies, and emergent reasoning mechanisms. The survey…

Kaiwen Luo, Zhenhong Zhou, Leyan Wang, Liang Lin, et al.

Published: May 2026
Citations: 4
Code: 258 stars

Read summary Original

OpenAI

IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMs

The paper introduces IH-Challenge, a reinforcement learning (RL) training dataset designed to improve instruction hierarchy (IH) robustness in large language models (LLMs). IH defines how models prioritize system, developer, user, and tool instructions under conflict, which is key for defending against jailbreaks, system prompt extractions, and prompt…

Chuan Guo, Juan Felipe Ceron Uribe, Sicheng Zhu, Christopher A. Choquette-Choo, et al.

Published: Mar 2026
Citations: 12
Code: Not linked

Read summary Original

arXiv.org

The Devil Behind Moltbook: Anthropic Safety is Always Vanishing in Self-Evolving AI Societies

The paper argues that a multi-agent system built from large language models cannot simultaneously achieve continuous self-evolution, complete isolation from external feedback, and safety invariance—a combination termed the self-evolution trilemma. Using an information-theoretic framework, safety is formalized as the KL divergence from an anthropic value…

Chenxu Wang, Chaozhuo Li, Songyang Liu, Zejian Chen, et al.

Published: Feb 2026
Citations: 3
Code: Not linked

Read summary Original

Independent research

AgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and Security

AgentDoG is a diagnostic guardrail framework for AI agent safety and security, developed by the Shanghai Artificial Intelligence Laboratory. It addresses limitations in existing guardrails by introducing a unified three-dimensional safety taxonomy that categorizes agentic risks by source (where), failure mode (how), and real-world harm (what). Guided by…

Dongrui Liu, Qihan Ren, Chen Qian, Shuai Shao, et al.

Published: Jan 2026
Citations: 24
Code: 680 stars

Read summary Original

OpenAI

Training LLMs for Honesty via Confessions

The paper proposes a method to train LLMs to produce 'confessions'—self-reports of compliance with instructions and policies—to improve honesty. Confession training adds a system message after the model's answer, requesting a structured report enumerating objectives, compliance analysis, and uncertainties. The confession reward is based solely on honesty…

Manas Joglekar, Jeremy Chen, Gabriel Wu, Jason Yosinski, et al.

Published: Dec 2025
Citations: 17
Code: Not linked

Read summary Original

Annual Meeting of the Association for Computational Linguistics

Too Good to be Bad: On the Failure of LLMs to Role-Play Villains

This paper investigates the ability of large language models (LLMs) to role-play morally ambiguous or villainous characters, hypothesizing that safety alignment conflicts with authentic antagonistic portrayal. The authors introduce the Moral RolePlay benchmark, a dataset with a four-level moral alignment scale (Moral Paragons, Flawed-but-Good, Egoists,…

Zihao Yi, Qingxuan Jiang, Ruotian Ma, Xingyu Chen, et al.

Published: Nov 2025
Citations: 7
Code: Not linked

Read summary Original

Anthropic

Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples

This paper investigates whether poisoning attacks on large language models (LLMs) require a constant number of poisoned samples regardless of dataset size, rather than a fixed percentage. The authors conducted the largest pretraining poisoning experiments to date, training models from 600M to 13B parameters on Chinchilla-optimal datasets (6B to 260B…

Alexandra Souly, Javier Rando, Ed Chapman, Xander Davies, et al.

Published: Oct 2025
Citations: 76
Code: Not linked

Read summary Original

arXiv.org

Large Reasoning Models Learn Better Alignment from Flawed Thinking

Large reasoning models (LRMs) generate chain-of-thought (CoT) before answering but are easily biased by flawed reasoning prefills, leading to unsafe or overrefused outputs. The paper introduces RECAP (Robust Safety Alignment via Counter-Aligned Prefilling), a reinforcement learning (RL) post-training method that trains models to override flawed reasoning…

ShengYun Peng, Eric Smith, Ivan Evtimov, Song Jiang, et al.

Published: Oct 2025
Citations: 10
Code: Not linked

Read summary Original

arXiv.org

Reasoning over Boundaries: Enhancing Specification Alignment via Test-time Deliberation

The paper introduces specification alignment, a challenge for LLMs to follow dynamic, scenario-specific behavioral and safety specifications. To evaluate this, the authors present SPECBENCH, a benchmark covering 5 scenarios, 103 specifications, and 1,500 prompts. Experiments on 33 models reveal significant alignment gaps and a safety-behavior trade-off.…

Haoran Zhang, Yafu Li, Xuyang Hu, Dongrui Liu, et al.

Published: Sep 2025
Citations: 3
Code: 24 stars

Read summary Original