Paper 2508.07999
WideSearch: Benchmarking Agentic Broad Info-Seeking
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 53
- GitHub
- 149 stars
01 In brief
Summary
The paper introduces WideSearch, a benchmark for evaluating LLM-based search agents on large-scale, broad information-seeking tasks.
It comprises 200 manually curated questions (100 English, 100 Chinese) across 18 domains, requiring agents to collect and structure extensive atomic information into tables.
A five-stage quality control pipeline ensures task difficulty, verifiability, and alignment with human judgment.
Benchmarking over 10 state-of-the-art systems (single-agent, multi-agent, and end-to-end) reveals extremely low success rates, with the best performer achieving only 5.1% (OpenAI o3 in multi-agent mode), while most systems score near 0%.
In contrast, human testers with cross-validation can achieve near 100% success.
Error analysis identifies key deficiencies: incomplete query decomposition, lack of reflection, failure in evidence utilization, and knowledge hallucination.
Test-time scaling shows item-level F1 can reach ~80% with 128 attempts, but table-level success remains below 20%, indicating the challenge lies in ensuring absolute completeness and accuracy.
The authors suggest multi-agent architectures with parallel search and cross-validation as a promising direction for improvement.
02 From the paper
Abstract
From professional research to everyday planning, many tasks are bottlenecked by wide-scale information seeking, which is more repetitive than cognitively complex. With the rapid development of Large Language Models (LLMs), automated search agents powered by LLMs offer a promising solution to liberate humans from this tedious work. However, the capability of these agents to perform such "wide-context" collection reliably and completely remains largely unevaluated due to a lack of suitable benchmarks. To bridge this gap, we introduce WideSearch, a new benchmark engineered to evaluate agent reliability on these large-scale collection tasks. The benchmark features 200 manually curated questions (100 in English, 100 in Chinese) from over 15 diverse domains, grounded in real user queries. Each task requires agents to collect large-scale atomic information, which could be verified one by one objectively, and arrange it into a well-organized output. A rigorous five-stage quality control pipeline ensures the difficulty, completeness, and verifiability of the dataset. We benchmark over 10 state-of-the-art agentic search systems, including single-agent, multi-agent frameworks, and end-to-end commercial systems. Most systems achieve overall success rates near 0\%, with the best performer reaching just 5\%. However, given sufficient time, cross-validation by multiple human testers can achieve a near 100\% success rate. These results demonstrate that present search agents have critical deficiencies in large-scale information seeking, underscoring urgent areas for future research and development in agentic search. Our dataset, evaluation pipeline, and benchmark results have been publicly released at https://widesearch-seed.github.io/