Paper 2512.01948
How Far Are We from Genuinely Useful Deep Research Agents?
- Published
- Dec 2025
- Research lab
- Independent
- Citations
- 9
- GitHub
- 66 stars
01 In brief
Summary
This paper introduces FINDER, a benchmark for evaluating Deep Research Agents (DRAs) on report generation, and DEFT, a failure taxonomy for diagnosing DRA errors.
FINDER consists of 100 human-curated research tasks with 419 structured checklist items, refining the DeepResearch Bench by adding detailed prompts and checklists.
DEFT, built via grounded theory with human-LLM co-annotation, categorizes 14 failure modes across three core dimensions: reasoning, retrieval, and generation.
Experiments on proprietary APIs, open-source models, and agent frameworks reveal that current DRAs struggle more with evidence integration, verification, and reasoning resilience than with task comprehension.
Over 39% of failures occur in generation, notably strategic content fabrication, while retrieval failures account for over 32%.
The authors propose a positive taxonomy metric and demonstrate DEFT's reliability through inter-coder reliability assessments.
The findings suggest that improving generative constraints and verification mechanisms is key to enhancing DRA output quality.
02 From the paper
Abstract
Deep Research Agents (DRAs) aim to automatically produce analyst-level reports through iterative information retrieval and synthesis. However, most existing DRAs were validated on question-answering benchmarks, while research on generating comprehensive reports remains overlooked. Worse, current benchmarks for report synthesis suffer from task complexity and subjective metrics -- this fails to reflect user demands and limits the practical utility of generated reports. To address these gaps, we present Fine-grained DEepResearch bench (FINDER), an enhanced benchmark consisting of 100 human-curated research tasks with 419 structured checklist items that standardize report structure, analytical depth, and factual grounding. Based on approximately 1,000 reports produced by mainstream DRAs, we further propose Deep rEsearch Failure Taxonomy (DEFT), the first failure taxonomy for deep research agents. DEFT contains 14 fine-grained failure modes across reasoning, retrieval, and generation, and is built upon grounded theory with human-LLM co-annotating and inter-annotator reliability validation. Our experimental findings reveal that current DRAs struggle not with task comprehension but with evidence integration, verification, and reasoning-resilient planning.