Research collection

Agent benchmarks and computer use

Evaluating agents on realistic tasks: computer-use and GUI agents, tool-use benchmarks, long-horizon task suites, and agent safety evaluation.

Papers: 66
Research labs: 4
Official code: 57

51–66 of 66 papers in this collection

51

arXiv.org

DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle

DAComp is a benchmark of 210 tasks evaluating LLM-based agents on enterprise data intelligence, covering repository-level data engineering (DE) and open-ended data analysis (DA). DE tasks include architecture, implementation, and evolution, requiring multi-stage SQL pipelines on schemas averaging 412 columns and over 2,000 lines of code. DA tasks involve…

Fangyu Lei, Jinxiang Meng, Yiming Huang, Junjie Zhao, et al.

Published: Dec 2025
Citations: 11
Code: 434 stars

Read summary Original

52

arXiv.org

AutoEnv: Automated Environments for Measuring Cross-Environment Agent Learning

The paper introduces AUTOENV, an automated framework for generating heterogeneous agent environments by treating them as factorizable distributions over transitions, observations, and rewards, with three abstraction layers (BaseEnv, ObsEnv, SkinEnv) and a three-stage verification pipeline. Using AUTOENV, the authors construct AUTOENV-36, a dataset of 36…

Jiayi Zhang, Yiran Peng, Fanqi Kong, Cheng Yang, et al.

Published: Nov 2025
Citations: 13
Code: 67 stars

Read summary Original

53

arXiv.org

Computer-Use Agents as Judges for Generative User Interface

The paper introduces AUI-Gym, a benchmark for automatic GUI development and testing, and a Coder-CUA collaboration framework. AUI-Gym includes 52 applications across six domains (app, landing, game, interactive, tool, utility) with 1,560 tasks synthesized by GPT-5 and human-validated. Each task is paired with a rule-based verifier that programmatically…

Kevin Qinghong Lin, Siyuan Hu, Linjie Li, Zhengyuan Yang, et al.

Published: Nov 2025
Citations: 6
Code: 45 stars

Read summary Original

54

Conference of the European Chapter of the Association for Computational Linguistics

Multimodal Evaluation of Russian-language Architectures

The paper introduces MERA Multi, the first open multimodal evaluation benchmark for Russian-language architectures, addressing the lack of such benchmarks for Slavic languages. It comprises 18 instruction-based tasks across text, image, audio, and video modalities, built on a unified taxonomy of multimodal abilities. The benchmark includes 11 private…

Artem Chervyakov, Ulyana Isaeva, Anton Emelyanov, Artem Safin, et al.

Published: Nov 2025
Citations: 3
Code: Not linked

Read summary Original

55

arXiv.org

Grounding Computer Use Agents on Human Demonstrations

The paper introduces GROUNDCUA, a large-scale desktop grounding dataset built from expert human demonstrations, covering 87 applications across 12 categories with 56K screenshots and over 3.56M human-verified element annotations. The dataset features high-resolution images (0.39–7.0 megapixels), dense annotations (average 64 per screenshot), and small…

Aarash Feizi, Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, et al.

Published: Nov 2025
Citations: 9
Code: 134 stars

Read summary Original

56

arXiv.org

InteractComp: Evaluating Search Agents With Ambiguous Queries

The paper introduces INTERACTCOMP, a benchmark for evaluating whether search agents can recognize and resolve ambiguous queries through interaction. It contains 210 expert-curated questions across 9 domains, built using a target-distractor methodology where questions use only shared attributes of a lesser-known target and a popular alternative, making them…

Mingyi Deng, Lijun Huang, Yani Fan, Fanqi Kong, et al.

Published: Oct 2025
Citations: 11
Code: 22 stars

Read summary Original

57

Annual Meeting of the Association for Computational Linguistics

OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflows

This paper introduces OS-Sentinel, a hybrid framework for detecting safety risks in mobile GUI agents powered by Vision-Language Models (VLMs). The authors first construct MobileRisk-Live, a dynamic Android emulator sandbox that records GUI observations, agent actions, and a System State Trace (T_sys) capturing underlying system metadata. From this, they…

Qiushi Sun, Mukai Li, Zhoumianze Liu, Zhihui Xie, et al.

Published: Oct 2025
Citations: 15
Code: 49 stars

Read summary Original

58

arXiv.org

DITING: A Multi-Agent Evaluation Framework for Benchmarking Web Novel Translation

The paper introduces DITING, the first comprehensive evaluation framework for web novel translation, assessing narrative and cultural fidelity across six dimensions: idiom translation, lexical ambiguity, terminology localization, tense consistency, zero-pronoun resolution, and cultural safety, supported by over 18K expert-annotated Chinese–English sentence…

Enze Zhang, Jiaying Wang, Mengxi Xiao, Jifei Liu, et al.

Published: Oct 2025
Citations: 4
Code: 12 stars

Read summary Original

59

OpenAI

GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks

This paper introduces GDPval, a benchmark for evaluating AI models on real-world, economically valuable tasks. It covers 44 occupations across the top 9 U.S. GDP sectors, with tasks created by industry experts averaging 14 years of experience. The benchmark includes 1,320 tasks in the full set and a 220-task gold subset, graded via human expert pairwise…

Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, et al.

Published: Oct 2025
Citations: 108
Code: Not linked

Read summary Original

60

arXiv.org

StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets?

STOCKBENCH is a new benchmark for evaluating LLM agents in realistic, multi-month stock trading environments, using data from March to June 2025 to avoid contamination. Agents receive daily prices, fundamentals, and news, and make sequential buy, sell, or hold decisions. Performance is measured by cumulative return, maximum drawdown, and Sortino ratio.…

Yanxu Chen, Zijun Yao, Yantao Liu, Amy Xin, et al.

Published: Oct 2025
Citations: 26
Code: 177 stars

Read summary Original

61

arXiv.org

MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use

MCPMark is a benchmark designed to evaluate LLM agents' use of the Model Context Protocol (MCP) in realistic, comprehensive workflows. It comprises 127 tasks across five MCP servers (Filesystem, Notion, Playwright, GitHub, PostgreSQL), each with curated initial states and programmatic verification scripts. Tasks require diverse CRUD operations and average…

Zijian Wu, Xiangyan Liu, Xinyuan Zhang, Lingjun Chen, et al.

Published: Sep 2025
Citations: 25
Code: 456 stars

Read summary Original

62

arXiv.org

ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data

ScaleCUA introduces a large-scale, cross-platform dataset and model family for computer use agents (CUAs), addressing data scarcity via a dual-loop pipeline combining automated agents and human experts across six platforms (Windows, macOS, Linux, Android, iOS, Web). The dataset includes 471K GUI understanding examples, 17.1M grounding annotations, and 19K…

Zhaoyang Liu, Jingjing Xie, Zichen Ding, Zehao Li, et al.

Published: Sep 2025
Citations: 46
Code: 1.1K stars

Read summary Original

63

arXiv.org

Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?

The paper introduces Inverse IFEval, a benchmark to evaluate LLMs' ability to follow counterintuitive instructions that conflict with training conventions. It identifies 'cognitive inertia' from SFT, where models struggle with instructions deviating from standard patterns. The benchmark includes eight instruction types (e.g., Question Correction,…

Qinyan Zhang, Xinping Lei, Ruijie Miao, Yu Fu, et al.

Published: Sep 2025
Citations: 6
Code: Not linked

Read summary Original

64

AAAI Conference on Artificial Intelligence

DeepResearch Arena: The First Exam of LLMs' Research Abilities via Seminar-Grounded Tasks

The paper introduces DeepResearch Arena, a benchmark for evaluating deep research agents using tasks derived from academic seminar transcripts. It addresses limitations of existing benchmarks, which either risk data leakage from static corpora or lack scalability and realism in expert-curated sets. The authors propose a Multi-Agent Hierarchical Task…

Haiyuan Wan, Chen Yang, Junchi Yu, Meiqi Tu, et al.

Published: Sep 2025
Citations: 26
Code: Not linked

Read summary Original

65

arXiv.org

MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers

MCP-Bench is a benchmark for evaluating LLM agents on realistic, multi-step tool-use tasks via the Model Context Protocol (MCP). It connects agents to 28 live MCP servers with 250 tools across domains like finance, travel, and science, enabling complex intra- and cross-server workflows. Tasks are generated via an LLM-based synthesis pipeline that discovers…

Zhenting Wang, Qi Chang, Hemani Patel, Shashank Biju, et al.

Published: Aug 2025
Citations: 90
Code: 496 stars

Read summary Original

66

arXiv.org

WideSearch: Benchmarking Agentic Broad Info-Seeking

The paper introduces WideSearch, a benchmark for evaluating LLM-based search agents on large-scale, broad information-seeking tasks. It comprises 200 manually curated questions (100 English, 100 Chinese) across 18 domains, requiring agents to collect and structure extensive atomic information into tables. A five-stage quality control pipeline ensures task…

Ryan Wong, Jiawei Wang, Junjie Zhao, Li Chen, et al.

Published: Aug 2025
Citations: 53
Code: 149 stars

Read summary Original