Paper 2603.12180
Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections
- Published
- Mar 2026
- Research lab
- Independent
- Citations
- 1
- GitHub
- 39 stars
01 In brief
Summary
The paper introduces MADQA, a benchmark of 2,250 human-authored questions over 800 heterogeneous PDF documents, designed to evaluate multimodal agentic systems.
It formalizes Agentic Document Collection VQA with six properties: extractive answers, multi-hop reasoning, closed-world assumption, grounded attribution, agentic retrieval, and visual comprehension.
The benchmark uses Classical Test Theory for principled splits, reserving a Sentinel Pool of hard items.
Evaluation includes LLM-based accuracy, Page/Doc F1 for attribution, and a novel Kuiper statistic for effort calibration.
Results show the best agent (Gemini 3 Pro BM25 Agent) matches human accuracy (82.2%) but with low item agreement (κ=0.24), relying on brute-force search.
Agents fail to close the 18% gap to oracle performance (99.4%).
Humans calibrate effort better (Kuiper 14.6 vs.
22.9+ for agents).
Error analysis shows retrieval failures (35.7%) dominate, and query reformulation magnitude correlates with success.
The authors release the dataset and evaluation harness to promote calibrated, efficient reasoning.
02 From the paper
Abstract
Multimodal agents offer a promising path to automating complex document-intensive workflows. Yet, a critical question remains: do these agents demonstrate genuine strategic reasoning, or merely stochastic trial-and-error search? To address this, we introduce MADQA, a benchmark of 2,250 human-authored questions grounded in 800 heterogeneous PDF documents. Guided by Classical Test Theory, we design it to maximize discriminative power across varying levels of agentic abilities. To evaluate agentic behaviour, we introduce a novel evaluation protocol measuring the accuracy-effort trade-off. Using this framework, we show that while the best agents can match human searchers in raw accuracy, they succeed on largely different questions and rely on brute-force search to compensate for weak strategic planning. They fail to close the nearly 20% gap to oracle performance, persisting in unproductive loops. We release the dataset and evaluation harness to help facilitate the transition from brute-force retrieval to calibrated, efficient reasoning.