Paper 2510.24668
InteractComp: Evaluating Search Agents With Ambiguous Queries
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 11
- GitHub
- 22 stars
01 In brief
Summary
The paper introduces INTERACTCOMP, a benchmark for evaluating whether search agents can recognize and resolve ambiguous queries through interaction.
It contains 210 expert-curated questions across 9 domains, built using a target-distractor methodology where questions use only shared attributes of a lesser-known target and a popular alternative, making them ambiguous without clarification.
Evaluation of 17 models shows that the best model achieves only 13.73% accuracy in the full interaction setting, despite reaching 71.50% with complete context, revealing systematic overconfidence rather than reasoning deficits.
Forced interaction doubles accuracy, demonstrating latent capability, while longitudinal analysis shows interaction capabilities stagnated over 15 months despite seven-fold improvement in search performance.
The benchmark provides clean reward signals suitable for training approaches like reinforcement learning from verifiable rewards.
02 From the paper
Abstract
Language agents have demonstrated remarkable potential in web search and information retrieval. However, many search-agent benchmarks assume that user queries are complete and unambiguous. This assumption leaves under-tested a practical failure mode: agents may face ambiguous requests where the intended target cannot be identified without clarification. Yet most agents lack interactive mechanisms during the search process, and existing benchmarks cannot assess this capability. To address this gap, we introduce InteractComp, a benchmark designed to evaluate whether search agents can recognize query ambiguity and actively interact to resolve it during search. Following the principle of easy to verify, interact to disambiguate, we construct 210 expert-curated questions across 9 domains through a target-distractor methodology that creates controlled ambiguity resolvable only through interaction. Evaluation of 17 models reveals striking failure: the best model achieves only 13.73% accuracy despite 71.50% with complete context, exposing systematic overconfidence rather than reasoning deficits. Forced interaction produces dramatic gains, demonstrating latent capability current strategies fail to engage. Longitudinal analysis shows interaction capabilities stagnated over 15 months while search performance improved seven-fold, revealing a critical blind spot. This stagnation, coupled with the immediate feedback inherent to search tasks, makes InteractComp a valuable resource for both evaluating and training interaction capabilities in search agents. The code is available at https://github.com/FoundationAgents/InteractComp.