The Year in AI Papers
The year/Independent research

Paper 2509.01396

DeepResearch Arena: The First Exam of LLMs' Research Abilities via Seminar-Grounded Tasks

Haiyuan Wan, Chen Yang, Junchi Yu, Meiqi Tu, Jiaxuan Lu, Di Yu, Jianbao Cao, Ben Gao, Jiaqing Xie, Aoran Wang, Wenlong Zhang, Philip Torr, et al.

Published
Sep 2025
Research lab
Independent
Citations
26
GitHub
Not linked

01 In brief

Summary

The paper introduces DeepResearch Arena, a benchmark for evaluating deep research agents using tasks derived from academic seminar transcripts.

It addresses limitations of existing benchmarks, which either risk data leakage from static corpora or lack scalability and realism in expert-curated sets.

The authors propose a Multi-Agent Hierarchical Task Generation (MAHTG) system that extracts research-worthy inspirations from seminar transcripts, categorizes them into Limitation, Methodology, Transdisciplinarity, and Hypothesis, and transforms them into over 10,000 open-ended tasks across 12 disciplines.

The benchmark uses a hybrid evaluation framework combining Keypoint-Aligned Evaluation (KAE) for factual grounding and Adaptively-generated Checklist Evaluation (ACE) for subjective quality.

Experiments on eight models show performance gaps, with o4-mini-deepresearch and gemini-2.5-flash leading, while grok-4 shows strong English but weak Chinese performance.

Data leakage tests confirm low similarity scores, and automated metrics align well with human judgments.

02 From the paper

Abstract

Deep research agents have attracted growing attention for their potential to orchestrate multi-stage research workflows, spanning literature synthesis, methodological design, and empirical verification. Despite these strides, evaluating their research capability faithfully is rather challenging due to the difficulty of collecting frontier research questions that genuinely capture researchers' attention and intellectual curiosity. To address this gap, we introduce DeepResearch Arena, a benchmark grounded in academic seminars that capture rich expert discourse and interaction, better reflecting real-world research environments and reducing the risk of data leakage. To automatically construct DeepResearch Arena, we propose a Multi-Agent Hierarchical Task Generation (MAHTG) system that extracts research-worthy inspirations from seminar transcripts. The MAHTG system further translates research-worthy inspirations into high-quality research tasks, ensuring the traceability of research task formulation while filtering noise. With the MAHTG system, we curate DeepResearch Arena with over 10,000 high-quality research tasks from over 200 academic seminars, spanning 12 disciplines, such as literature, history, and science. Our extensive evaluation shows that DeepResearch Arena presents substantial challenges for current state-of-the-art agents, with clear performance gaps observed across different models.