Paper 2508.04026
VeriWeb: Verifiable Long-Chain Web Benchmark for Agentic Information-Seeking
- Published
- Aug 2025
- Research lab
- Independent
- Citations
- 9
- GitHub
- 88 stars
01 In brief
Summary
VeriWeb is a new benchmark for evaluating web agents on long-horizon, information-rich tasks.
It addresses limitations of prior benchmarks that focus on single-fact retrieval and outcome-only verification.
VeriWeb emphasizes long-chain complexity (breadth- and depth-oriented search) and subtask-level verifiability, decomposing tasks into interdependent subtasks with verifiable answers.
The dataset includes 302 human-annotated tasks across five domains (scientific, finance, technology, arts, social), with an average of 4.3 subtasks and 272.5 steps per task.
Experiments with various agents (deep research, search engine, browser-use, multi-agent) show no configuration exceeds 15% success rate or 40% completion rate, highlighting significant performance gaps.
Dominant failure modes are misinformation and incomplete results.
LLM-as-a-judge evaluation shows high agreement with human scores (Pearson correlation 0.9195).
The benchmark supports fine-grained analysis of agent capabilities and failure modes, underscoring the need for more powerful agentic information-seeking capabilities.
02 From the paper
Abstract
Recent advances have showcased the extraordinary capabilities of Large Language Model (LLM) agents in tackling web-based information-seeking tasks. However, existing efforts mainly focus on single-fact retrieval and rely on outcome-only verification, thereby limiting their scalability in realistic knowledge-intensive scenarios that involve long-horizon web tasks requiring large-scale retrieval and synthesis of information from diverse sources. In this work, we introduce VeriWeb, a novel verifiable long-chain web benchmark designed to facilitate the evaluation and development of web agents within realistic web environments. Our benchmark emphasizes two critical dimensions: (1) long-chain complexity, encompassing both breadth- and depth-oriented search tasks to assess how effectively web agents ensure comprehensive information coverage and consistent context tracking in multi-hop reasoning; and (2) subtask-level verifiability, where tasks are decomposed into a sequence of interdependent verifiable subtasks. This structure enables diverse exploration strategies within each subtask, while ensuring that each subtask-level answer remains unchanged and verifiable. The benchmark consists of 302 tasks across five real-world domains, each with a complete trajectory demonstration, annotated by human experts. Extensive experiments on VeriWeb using various agents powered by different foundation models reveal significant performance gaps in handling long-horizon web tasks, highlighting the need for more powerful agentic information-seeking capabilities.