Paper 2509.04664
Why Language Models Hallucinate
- Published
- Sep 2025
- Research lab
- OpenAI
- Citations
- 282
- GitHub
- Not linked
01 In brief
Summary
The paper argues that language model hallucinations arise from statistical pressures during pretraining and persist due to misaligned evaluation metrics.
The authors formalize hallucinations as errors in binary classification, showing that even with error-free training data, the cross-entropy objective leads to errors.
They introduce the Is-It-Valid (IIV) problem, connecting generative error rates to misclassification rates, and prove that hallucination rates are at least the fraction of training facts appearing once (singleton rate).
Post-training, they argue that binary grading (e.g., accuracy) rewards guessing over abstention, as abstaining yields zero credit while guessing can yield correct answers.
They propose modifying existing benchmarks to include explicit confidence thresholds and penalties for incorrect guesses, rather than adding new hallucination evaluations, to realign incentives and reduce hallucinations.
The paper includes examples of models hallucinating on birthdays and letter counting, and analyzes factors like poor models, computational hardness, and distribution shift.
02 From the paper
Abstract
Like students facing hard exam questions, large language models sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty. Such "hallucinations" persist even in state-of-the-art systems and undermine trust. We argue that language models hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty, and we analyze the statistical causes of hallucinations in the modern training pipeline. Hallucinations need not be mysterious -- they originate simply as errors in binary classification. If incorrect statements cannot be distinguished from facts, then hallucinations in pretrained language models will arise through natural statistical pressures. We then argue that hallucinations persist due to the way most evaluations are graded -- language models are optimized to be good test-takers, and guessing when uncertain improves test performance. This "epidemic" of penalizing uncertain responses can only be addressed through a socio-technical mitigation: modifying the scoring of existing benchmarks that are misaligned but dominate leaderboards, rather than introducing additional hallucination evaluations. This change may steer the field toward more trustworthy AI systems.