Paper 2601.09688
DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluation
- Published
- Jan 2026
- Research lab
- Independent
- Citations
- 12
- GitHub
- 142 stars
01 In brief
Summary
DeepResearchEval is an automated framework for constructing deep research tasks and evaluating deep research systems.
It addresses limitations in existing benchmarks: annotation-intensive task construction, static evaluation dimensions, and incomplete fact verification.
The framework uses a persona-driven pipeline to generate realistic, complex tasks across 10 domains, filtered by Task Qualification and Search Necessity filters, yielding 100 high-quality tasks.
For evaluation, it proposes an Adaptive Point-wise Quality Evaluation that combines fixed dimensions (Coverage, Insight, Instruction-following, Clarity) with task-specific dimensions and weights, and an Active Fact-Checking module that verifies both cited and uncited statements via web search, labeling them Right, Wrong, or Unknown.
The framework was applied to 9 deep research systems, evaluating 900 reports.
Results show Gemini-2.5-Pro Deep Research achieved the highest quality score (8.51/10), while Manus achieved the highest factual correctness ratio (82.3%).
The evaluation also revealed a consistent gap between general and task-specific scores, indicating current systems often fail to meet task-specific criteria.
Validation showed high cross-judge consistency, stochastic stability, and 73% agreement with human annotations for fact-checking.
Limitations include English-centricity and high computational costs.
02 From the paper
Abstract
Deep research systems are widely used for multi-step web research, analysis, and cross-source synthesis, yet their evaluation remains challenging. Existing benchmarks often require annotation-intensive task construction, rely on static evaluation dimensions, or fail to reliably verify facts when citations are missing. To bridge these gaps, we introduce DeepResearchEval, an automated framework for deep research task construction and agentic evaluation. For task construction, we propose a persona-driven pipeline generating realistic, complex research tasks anchored in diverse user profiles, applying a two-stage filter Task Qualification and Search Necessity to retain only tasks requiring multi-source evidence integration and external retrieval. For evaluation, we propose an agentic pipeline with two components: an Adaptive Point-wise Quality Evaluation that dynamically derives task-specific evaluation dimensions, criteria, and weights conditioned on each generated task, and an Active Fact-Checking that autonomously extracts and verifies report statements via web search, even when citations are missing.