Paper 2603.10521
IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMs
- Published
- Mar 2026
- Research lab
- OpenAI
- Citations
- 12
- GitHub
- Not linked
01 In brief
Summary
The paper introduces IH-Challenge, a reinforcement learning (RL) training dataset designed to improve instruction hierarchy (IH) robustness in large language models (LLMs).
IH defines how models prioritize system, developer, user, and tool instructions under conflict, which is key for defending against jailbreaks, system prompt extractions, and prompt injections.
The dataset is constructed with three principles: tasks are IF-simple (difficulty stems from IH conflicts, not problem-solving), programmatically gradable via Python graders, and diverse to avoid shortcut learning (e.g., overrefusal).
Fine-tuning GPT-5-Mini on IH-Challenge with online adversarial example generation (using an attacker LLM) improved IH robustness by +10.0% on average across 16 benchmarks (84.1% → 94.1%), reduced unsafe behavior from 6.6% to 0.7% while maintaining helpfulness, and saturated an internal agentic prompt injection evaluation, with minimal capability regression.
The model also generalized to out-of-distribution tasks and adaptive human red-teaming (success rate dropped from 36.2% to 11.7%).
The dataset is released on Hugging Face.
02 From the paper
Abstract
Instruction hierarchy (IH) defines how LLMs prioritize system, developer, user, and tool instructions under conflict, providing a concrete, trust-ordered policy for resolving instruction conflicts. IH is key to defending against jailbreaks, system prompt extractions, and agentic prompt injections. However, robust IH behavior is difficult to train: IH failures can be confounded with instruction-following failures, conflicts can be nuanced, and models can learn shortcuts such as overrefusing. We introduce IH-Challenge, a reinforcement learning training dataset, to address these difficulties. Fine-tuning GPT-5-Mini on IH-Challenge with online adversarial example generation improves IH robustness by +10.0% on average across 16 in-distribution, out-of-distribution, and human red-teaming benchmarks (84.1% to 94.1%), reduces unsafe behavior from 6.6% to 0.7% while improving helpfulness on general safety evaluations, and saturates an internal static agentic prompt injection evaluation, with minimal capability regression. We release the IH-Challenge dataset (https://huggingface.co/datasets/openai/ih-challenge) to support future research on robust instruction hierarchy.