The Year in AI Papers
The year/Independent research

Paper 2508.09848

PRELUDE: A Benchmark Designed to Require Global Comprehension and Reasoning over Long Contexts

Mo Yu, Tsz Ting Chung, Chulun Zhou, Tong Li, Rui Lu, Jiangnan Li, Liyan Xu, Haoshu Lu, Ning Zhang, Jing Li, Jie Zhou

Published
Aug 2025
Research lab
Independent
Citations
6
GitHub
Not linked

01 In brief

Summary

PRELUDE is a benchmark for evaluating long-context understanding in LLMs by asking whether a character's prequel story is consistent with the canonical narrative of a book.

The task requires global comprehension and deep reasoning, as 88% of instances need evidence from multiple parts of the narrative.

The dataset includes 795 instances across 13 books, with labels for consistent and contradict cases, annotated by human experts.

Experiments with state-of-the-art LLMs, including ICL, RAG, in-domain training, and DeepResearch, show that the best systems lag human performance by over 15%, and models often produce correct answers with flawed reasoning, leading to a reasoning accuracy gap of over 30%.

The task is designed to mitigate shortcuts like memorization and summarization, and results indicate that current LLMs lack the intrinsic capability for this type of long-context reasoning.

02 From the paper

Abstract

We introduce PRELUDE, a benchmark for evaluating long-context understanding through the task of determining whether a character's prequel story is consistent with the canonical narrative of the original book. Our task poses a stronger demand for global comprehension and deep reasoning than existing benchmarks -- as the prequels are not part of the original story, assessing their plausibility typically requires searching and integrating information that is only indirectly related. Empirically, 88% of instances require evidence from multiple parts of the narrative. Experimental results highlight the challenge of our task: in-context learning, RAG and in-domain training with state-of-the-art LLMs, and commercial DeepResearch services, lag behind humans by >15%. A further human study reveals that models often produce correct answers with flawed reasoning, leading to an over 30% gap in reasoning accuracy compared to humans. These findings underscore the substantial room for improvement in long-context understanding and reasoning.