The Year in AI Papers
The year/Topics/Agent benchmarks and computer use

Research collection

Agent benchmarks and computer use

Evaluating agents on realistic tasks: computer-use and GUI agents, tool-use benchmarks, long-horizon task suites, and agent safety evaluation.

Papers
66
Research labs
4
Official code
57

1–50 of 66 papers in this collection

01

Independent research

StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents

StateAct is a code-first, multi-agent harness for long-horizon computer-use tasks that grounds the main agent in program state (files, DOM, backends) rather than screenshots, which are lossy and non-injective. The main agent acts via code, while a dedicated GUI subagent handles visual interaction on only 28 of 108 tasks and 1.1% of main-agent steps. An…

Yan Yang, Xiangru Jian, Ziyang Luo, Zirui Zhao, et al.
Published
Jul 2026
Citations
0
Code
Not linked
Read summary Original
02

Independent research

EvolvingWorld: An Open-Schema Framework for Co-Evolving Role-Play Agents and World Model in Interactive Literary World

EvolvingWorld is a framework and benchmark for simulating interactive literary worlds where characters and the world co-evolve over long horizons. It addresses limitations of existing systems that treat role-play as static persona imitation or isolated scene generation. The framework uses an open-schema design, coupling a Character Agent for…

Qing Zong, Yue Guo, Mengxin Yang, Yiwen Guo, et al.
Published
Jul 2026
Citations
0
Code
18 stars
Read summary Original
03

Independent research

Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading

Long-Horizon-Terminal-Bench is a new benchmark of 46 containerized terminal tasks across nine categories, designed to test AI agents on long-horizon workflows. Unlike prior benchmarks that use binary pass/fail grading, it decomposes each task into graded subtasks, providing dense partial-credit rewards. Tasks require an average of 239 episodes, 9.8M…

Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, et al.
Published
Jul 2026
Citations
3
Code
467 stars
Read summary Original
04

Independent research

AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents

The paper introduces AgenticSTS, a bounded-memory testbed for long-horizon LLM agents, instantiated in the game Slay the Spire 2. Instead of appending raw transcripts, the agent composes each decision prompt from five typed layers (L1-L5): fixed protocol, state schemas, game rules, episodic summaries, and triggered strategic skills. This contract keeps…

Xiangchen Cheng, Yunwei Jiang, Jianwen Sun, Zizhen Li, et al.
Published
Jul 2026
Citations
0
Code
123 stars
Read summary Original
05

Independent research

Agentic Abstention: Do Agents Know When to Stop Instead of Act?

The paper introduces Agentic Abstention, the problem of deciding when an LLM agent should stop acting and abstain rather than continue interacting with an environment. Unlike single-turn LLM abstention, this is a sequential decision problem where agents can answer, abstain, or act at each turn. The authors construct a benchmark of over 28,000 tasks across…

Han Luo, Bingbing Wen, Lucy Lu Wang
Published
Jun 2026
Citations
0
Code
42 stars
Read summary Original
06

Independent research

NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?

NatureBench is a benchmark of 90 tasks distilled from peer-reviewed Nature-family publications (2022–2025) to evaluate whether AI coding agents can move beyond reproduction toward scientific discovery. It is built on NatureGym, an automated pipeline that converts papers into containerized task packages with an information firewall, hidden ground truth, and…

Yuru Wang, Lejun Cheng, Yuxin Zuo, Sihang Zeng, et al.
Published
Jun 2026
Citations
1
Code
89 stars
Read summary Original
07

Independent research

EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions

EnterpriseClawBench is an enterprise agent benchmark built from 5,291 real workplace sessions at an AI startup, yielding 852 reproducible tasks (120 in a manually audited Lite subset). The pipeline applies mechanical gates (length, fixture, redaction, network) and rewrites prompts into single-turn tasks with role/skill taxonomies, hard rules, and semantic…

Jincheng Zhong, Weizhi Wang, Che Jiang, Kai Tian, et al.
Published
Jun 2026
Citations
0
Code
46 stars
Read summary Original
08

Independent research

PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems

PlanBench-XL is a new interactive benchmark for evaluating long-horizon planning of LLM tool-use agents in large-scale tool ecosystems. It consists of 327 retail tasks over 1,665 tools, requiring agents to iteratively retrieve tools, infer implicit sub-goals, and adapt to dynamic environments. The benchmark features a retrieval-mediated environment with…

Jiayu Liu, Qihan Lin, Cheng Qian, Rui Wang, et al.
Published
Jun 2026
Citations
4
Code
38 stars
Read summary Original
09

Independent research

EvoArena: Tracking Memory Evolution for Robust LLM Agents in Dynamic Environments

The paper introduces EvoArena, a benchmark suite for evaluating LLM agents under persistent environment evolution, and EvoMem, a patch-based memory paradigm that records memory updates as structured histories. EvoArena comprises three subsets: Terminal-Bench-Evo (evolving terminal workflows), SWE-Chain-Evo (evolving codebases), and PersonaMem-Evo (evolving…

Jundong Xu, Qingchuan Li, Jiaying Wu, Yihuai Lan, et al.
Published
Jun 2026
Citations
0
Code
21 stars
Read summary Original
10

Independent research

Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks

Claw-SWE-Bench is a multilingual SWE-bench-style benchmark and adapter protocol for evaluating general-purpose agent harnesses (claws) on coding tasks. It comprises 350 GitHub issue-resolution instances across 8 languages and 43 repositories, drawn from SWE-bench-Multilingual and SWE-bench-Verified-Mini after future-commit cleanup. The benchmark fixes…

Mengyu Zheng, Kai Han, Boxun Li, Haiyang Xu, et al.
Published
Jun 2026
Citations
2
Code
96 stars
Read summary Original
11

Independent research

WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces

WeaveBench is a long-horizon, hybrid-interface benchmark for computer-use agents, comprising 114 tasks across 8 real-world work domains. Each task requires agents to combine GUI observations/actions with CLI/code operations within a single trajectory, satisfying three admission criteria: channel non-substitutability, long-horizon execution, and…

Wanli Li, Bowen Zhou, Yunyao Yu, Zhou Xu, et al.
Published
Jun 2026
Citations
2
Code
160 stars
Read summary Original
12

Independent research

SWE-Explore: Benchmarking How Coding Agents Explore Repositories

SWE-Explore is a benchmark that isolates and evaluates the repository exploration capability of coding agents, independent of patch generation. It formalizes exploration as a ranked, line-level context selection task: given an issue and repository, an explorer returns a ranked list of code regions under a fixed line budget. The benchmark includes 848…

Shaoqiu Zhang, Yuhang Wang, Jialiang Liang, Yuling Shi, et al.
Published
Jun 2026
Citations
10
Code
39 stars
Read summary Original
13

Independent research

SoCRATES: Towards Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variations

SoCRATES is a benchmark for evaluating proactive LLM mediators in realistic, multi-domain conflict scenarios. It uses an agentic pipeline to curate scenarios from real conflicts across eight domains, probes mediators along five socio-cognitive axes (strategic posture, party composition, history length, emotional reactivity, cultural identity), and scores…

Taewon Yun, Hyeonseong Park, Jeonghwan Choi, Hayoon Park, et al.
Published
Jun 2026
Citations
0
Code
0 stars
Read summary Original
14

Independent research

Agents' Last Exam

Agents' Last Exam (ALE) is a benchmark introduced by UC Berkeley and collaborators to evaluate AI agents on long-horizon, economically valuable, real-world professional tasks with verifiable outcomes. Developed with 250+ industry experts, ALE covers 55 subfields across 13 industry clusters, grounded in the O*NET/SOC 2018 occupational taxonomy, and includes…

Yiyou Sun, Xinyang Han, Weichen Zhang, Yuanbo Pang, et al.
Published
Jun 2026
Citations
5
Code
936 stars
Read summary Original
15

Independent research

ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research

ResearchClawBench (RCBench) is a benchmark for evaluating end-to-end autonomous scientific research, comprising 40 tasks across 10 scientific domains (Astronomy, Chemistry, Earth Science, Energy, Information, Life, Material, Math, Neuroscience, Physics). Each task is derived from a real published paper, provides related literature and raw data, and hides…

Wanghan Xu, Shuo Li, Tianlin Ye, Qinglong Cao, et al.
Published
May 2026
Citations
3
Code
232 stars
Read summary Original
16

arXiv.org

A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks

The paper introduces TASTE (Task Synthesis from Tool Sequence Evolution), an automatic method for generating challenging agent benchmarks with broader tool-use coverage. TASTE reverses the traditional task construction process by first sampling diverse tool sequences using an Adaptive Contrastive n-gram model trained on LLM-judged validity signals, then…

Tomer Keren, Nitay Calderon, Asaf Yehudai, Yotam Perlitz, et al.
Published
May 2026
Citations
0
Code
4 stars
Read summary Original
17

arXiv.org

MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research

MOBILEGYM is a browser-hosted, lightweight simulation platform for mobile GUI agent research, targeting interaction fidelity without replicating proprietary backends. It enables verifiable outcome signals via deterministic state-based judging over structured JSON state, and scalable online RL through low-cost parallel rollouts. The environment state is…

Dingbang Wu, Rui Hao, Haiyang Wang, Shuzhe Wu, et al.
Published
May 2026
Citations
2
Code
750 stars
Read summary Original
18

arXiv.org

OpenComputer: Verifiable Software Worlds for Computer-Use Agents

OpenComputer is a verifier-grounded framework for constructing verifiable software worlds for computer-use agents. It integrates four components: app-specific state verifiers, a self-evolving verification layer, a task-generation pipeline, and an evaluation harness. The framework covers 33 desktop applications and 1,000 finalized tasks. Experiments show…

Jinbiao Wei, Qianran Ma, Yilun Zhao, Xiao Zhou, et al.
Published
May 2026
Citations
6
Code
31 stars
Read summary Original
19

arXiv.org

CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?

χ-Bench is a benchmark for evaluating AI agents on long-horizon, policy-rich healthcare workflows across three domains: provider prior authorization, payer utilization management, and care management. It uses a high-fidelity simulator of 20 healthcare apps exposed via 87 MCP tools, guided by a 1,279-document managed-care operations handbook. Across 30…

Haolin Chen, Deon Metelski, Leon Qi, Tao Xia, et al.
Published
May 2026
Citations
4
Code
54 stars
Read summary Original
20

NVIDIA

MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models

MEMLENS is a new benchmark for evaluating multimodal long-term memory in large vision-language models (LVLMs) and memory-augmented agents. It comprises 789 questions across five memory abilities (information extraction, multi-session reasoning, temporal reasoning, knowledge update, and answer refusal) at four context lengths (32K–256K tokens). An…

Xiyu Ren, Zhaowei Wang, Yiming Du, Zhongwei Xie, et al.
Published
May 2026
Citations
0
Code
26 stars
Read summary Original
21

arXiv.org

$π$-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows

The paper introduces π-BENCH, a benchmark for evaluating proactive assistance in long-horizon personal assistant workflows. It comprises 100 multi-turn tasks across 5 domain-specific user personas (researcher, marketer, law trainee, pharmacist, financier), organized into 20-session episodes with cross-session dependencies. Each task begins with an…

Haoran Zhang, Luxin Xu, Zhilin Wang, Runquan Gui, et al.
Published
May 2026
Citations
1
Code
58 stars
Read summary Original
22

arXiv.org

Auditing Agent Harness Safety

This paper introduces HarnessAudit, a framework for auditing LLM agent execution harnesses, and HarnessAudit-Bench, a benchmark of 210 tasks across 8 domains. The framework evaluates full execution trajectories on three layers: boundary compliance (tool, resource, and information-flow violations), execution fidelity (action validity and task completion),…

Chengzhi Liu, Yichen Guo, Yepeng Liu, Yuzhe Yang, et al.
Published
May 2026
Citations
1
Code
51 stars
Read summary Original
23

arXiv.org

EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents

EVA-Bench is an end-to-end evaluation framework for voice agents that addresses two challenges: generating realistic simulated conversations and measuring quality across voice-specific failure modes. It uses bot-to-bot audio conversations with automatic simulation validation, and introduces two composite metrics: EVA-A (Accuracy) for task completion,…

Tara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz, Oluwanifemi Bamgbose, et al.
Published
May 2026
Citations
3
Code
192 stars
Read summary Original
24

arXiv.org

ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents

ClawGUI is an open-source framework that integrates online reinforcement learning (RL) training, standardized evaluation, and real-device deployment for GUI agents. It addresses three gaps: closed training pipelines, misaligned evaluation protocols, and a broken deployment loop. ClawGUI-RL provides the first open-source RL infrastructure supporting…

Fei Tang, Zhiqiong Lu, Boxuan Zhang, Weiming Lu, et al.
Published
Apr 2026
Citations
10
Code
Not linked
Read summary Original
25

Qwen

OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation

OCCUBENCH is a benchmark for evaluating AI agents on real-world professional tasks across 100 scenarios, 65 domains, and 10 industry categories. It uses Language Environment Simulators (LESs), where an LLM simulates domain-specific tool responses, enabling evaluation in domains without public environments. A multi-agent pipeline synthesizes 382 solvable,…

Xiaomeng Hu, Yinger Zhang, Fei Huang, Jianhong Tu, et al.
Published
Apr 2026
Citations
1
Code
21 stars
Read summary Original
26

arXiv.org

ClawBench: Can AI Agents Complete Everyday Online Tasks?

ClawBench is a benchmark of 153 everyday online tasks across 144 live websites and 15 categories, focusing on write-heavy, state-changing workflows like purchases, reservations, and applications. Unlike sandboxed benchmarks, it evaluates agents on production sites with dynamic content and anti-bot defenses, using final-request interception to block…

Yuxuan Zhang, Yubo Wang, Yipeng Zhu, Penghui Du, et al.
Published
Apr 2026
Citations
19
Code
550 stars
Read summary Original
27

arXiv.org

GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents

GameWorld is a benchmark for evaluating multimodal large language models (MLLMs) as game agents in browser environments. It includes 34 diverse games across five genres (Runner, Arcade, Platformer, Puzzle, Simulation) and 170 tasks, each with outcome-based, state-verifiable metrics computed from serialized gameAPI state. The benchmark supports two agent…

Mingyu Ouyang, Siyuan Hu, Kevin Qinghong Lin, Hwee Tou Ng, et al.
Published
Apr 2026
Citations
10
Code
216 stars
Read summary Original
28

Independent research

Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents

Claw-Eval is an end-to-end evaluation suite for LLM-based autonomous agents, addressing gaps in trajectory-opaque grading, underspecified safety/robustness, and narrow task coverage. It comprises 300 human-verified tasks across 9 categories (General, Multimodal, Multi-turn Dialogue) with 2,159 rubric items. Each run is audited via three evidence channels:…

Bowen Ye, Rang Li, Qibin Yang, Yuanxin Liu, et al.
Published
Apr 2026
Citations
38
Code
741 stars
Read summary Original
29

arXiv.org

Terminal Agents Suffice for Enterprise Automation

The paper argues that minimal terminal-based coding agents, which interact directly with platform APIs via a terminal and filesystem, can match or outperform more complex agent architectures for enterprise automation. The authors introduce StarShell, a terminal agent, and compare it against GUI-driven web agents and MCP-based tool-augmented agents across…

Patrice Bechard, Orlando Marquez Ayala, Emily Chen, Jordan Skelton, et al.
Published
Mar 2026
Citations
5
Code
Not linked
Read summary Original
30

arXiv.org

CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agents

CUA-SUITE is a large-scale ecosystem of human-annotated video demonstrations for training and evaluating desktop computer-use agents (CUAs). It addresses the scarcity of continuous, high-quality human demonstration videos, which recent work identifies as critical for scaling agents. The core resource, VIDEOCUA, provides approximately 10,000…

Xiangru Jian, Shravan Nayak, Kevin Qinghong Lin, Aarash Feizi, et al.
Published
Mar 2026
Citations
3
Code
Not linked
Read summary Original
31

arXiv.org

ClawKeeper: Comprehensive Safety Protection for OpenClaw Agents Through Skills, Plugins, and Watchers

ClawKeeper is a comprehensive security framework for OpenClaw, an open-source autonomous agent runtime, addressing critical vulnerabilities such as sensitive data leakage, privilege escalation, and malicious skill execution. It integrates three complementary protection layers: skill-based protection at the instruction level, plugin-based runtime…

Songyang Liu, Chaozhuo Li, Chenxu Wang, Jinyu Hou, et al.
Published
Mar 2026
Citations
18
Code
1K stars
Read summary Original
32

arXiv.org

SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models

SocialOmni is a new benchmark for evaluating social interactivity in omni-modal large language models (OLMs), addressing the gap left by static, accuracy-centric benchmarks. It evaluates three dimensions: who is speaking (speaker identification), when to interject (turn-taking timing), and how to phrase interruptions (response generation). The benchmark…

Tianyu Xie, Jinfa Huang, Yuexiao Ma, Rongfang Luo, et al.
Published
Mar 2026
Citations
5
Code
46 stars
Read summary Original
33

arXiv.org

EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings

ENTERPRISEOPS-GYM is a benchmark for evaluating LLM agents in realistic enterprise settings, featuring a containerized sandbox with 164 database tables and 512 tools across eight domains (CSM, HR, ITSM, Email, Calendar, Teams, Drive, Hybrid). It includes 1,150 expert-curated tasks, 30 of which are infeasible to test refusal behavior, with SQL-based…

Shiva Krishna Reddy Malay, Shravan Nayak, Jishnu Sethumadhavan Nair, Sagar Davasam, et al.
Published
Mar 2026
Citations
6
Code
115 stars
Read summary Original
34

arXiv.org

LMEB: Long-horizon Memory Embedding Benchmark

The paper introduces LMEB, a benchmark for evaluating embedding models on long-horizon memory retrieval tasks, which are underexplored in existing benchmarks like MTEB. LMEB comprises 22 datasets and 193 zero-shot retrieval tasks across four memory types: episodic, dialogue, semantic, and procedural. The authors evaluate 15 embedding models, ranging from…

Xinping Zhao, Xinshuo Hu, Jiaxin Xu, Danyu Tang, et al.
Published
Mar 2026
Citations
7
Code
14 stars
Read summary Original
35

arXiv.org

Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections

The paper introduces MADQA, a benchmark of 2,250 human-authored questions over 800 heterogeneous PDF documents, designed to evaluate multimodal agentic systems. It formalizes Agentic Document Collection VQA with six properties: extractive answers, multi-hop reasoning, closed-world assumption, grounded attribution, agentic retrieval, and visual…

Łukasz Borchmann, Jordy Van Landeghem, Michał Turski, Shreyansh Padarha, et al.
Published
Mar 2026
Citations
1
Code
39 stars
Read summary Original
36

Annual Meeting of the Association for Computational Linguistics

Lost in Stories: Consistency Bugs in Long Story Generation by LLMs

The paper introduces ConStory-Bench, a benchmark for evaluating narrative consistency in long-form story generation by LLMs, and CONSTORY-CHECKER, an automated pipeline that detects contradictions with textual evidence. The benchmark includes 2,000 prompts across four task scenarios and a taxonomy of five error categories with 19 subtypes. Evaluation of…

Junjie Li, Xinrui Guo, Yuhao Wu, Roy Ka-Wei Lee, et al.
Published
Mar 2026
Citations
3
Code
192 stars
Read summary Original
37

Annual Meeting of the Association for Computational Linguistics

RubricBench: Aligning Model-Generated Rubrics with Human Standards

RubricBench is a new benchmark with 1,147 pairwise comparisons designed to assess rubric-guided evaluation in reward models. It uses a multi-dimensional filtration pipeline to select hard samples with input complexity, output surface bias, and process failures, each annotated with human-derived atomic rubrics. Experiments show that while rubric-aware…

Qiyuan Zhang, Junyi Zhou, Yufei Wang, Fuyuan Lyu, et al.
Published
Mar 2026
Citations
10
Code
31 stars
Read summary Original
38

arXiv.org

MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios

MobilityBench is a scalable benchmark for evaluating LLM-based route-planning agents in real-world mobility scenarios, built from 100,000 anonymized queries from Amap across 22 countries and over 350 cities. It covers 11 task scenarios in four families: Basic Information Retrieval, Route-Dependent Information Retrieval, Basic Route Planning, and…

Zhiheng Song, Jingshuai Zhang, Chuan Qin, Chao Wang, et al.
Published
Feb 2026
Citations
6
Code
157 stars
Read summary Original
39

Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval

SQuTR: A Robustness Benchmark for Spoken Query to Text Retrieval under Acoustic Noise

SQuTR is a benchmark for evaluating spoken query to text retrieval under controlled acoustic noise. It aggregates 37,317 unique queries from six English and Chinese text retrieval datasets (FiQA, HotpotQA, Natural Questions, MedicalRetrieval, DuRetrieval, T2Retrieval), synthesizing speech with CosyVoice-3 using 200 speakers. Four acoustic conditions…

Yuejie Li, Ke Yang, Yueying Hua, Berlin Chen, et al.
Published
Feb 2026
Citations
1
Code
10 stars
Read summary Original
40

arXiv.org

Code2World: A GUI World Model via Renderable Code Generation

Code2World is a vision-language coder that predicts the next GUI state by generating renderable HTML code, combining high visual fidelity with structural controllability. To address data scarcity, the authors constructed AndroidCode, a corpus of over 80K screen-action pairs, by translating GUI trajectories from AndroidControl into HTML using GPT-5 and…

Yuhao Zheng, Li'an Zhong, Yi Wang, Rui Dai, et al.
Published
Feb 2026
Citations
14
Code
323 stars
Read summary Original
41

arXiv.org

OdysseyArena: Benchmarking Large Language Models For Long-Horizon, Active and Inductive Interactions

ODYSSEYARENA is a benchmark suite for evaluating large language models (LLMs) on long-horizon, active, and inductive interactions, where agents must infer latent transition rules from experience rather than follow explicit instructions. The paper formalizes four structural primitives—discrete symbolic rules, continuous stochastic dynamics, periodic…

Hang Yan, Fangzhi Xu, Qiushi Sun, Jinyang Wu, et al.
Published
Feb 2026
Citations
6
Code
33 stars
Read summary Original
42

arXiv.org

Spider-Sense: Intrinsic Risk Sensing for Efficient Agent Defense with Hierarchical Adaptive Screening

The paper introduces SPIDER-SENSE, a framework for defending LLM-based autonomous agents against security threats. It argues that existing mandatory, stage-wise security checks are inefficient and proposes Intrinsic Risk Sensing (IRS), which embeds risk awareness into the agent's execution flow, triggering defenses only when a risk is perceived. Once…

Zhenxiong Yu, Zhi Yang, Zhiheng Jin, Shuhe Wang, et al.
Published
Feb 2026
Citations
1
Code
21 stars
Read summary Original
43

Volume 1

CAR-bench: Evaluating the Consistency and Limit-Awareness of LLM Agents under Real-World Uncertainty

CAR-bench is a benchmark for evaluating LLM agents in real-world, user-facing applications, specifically in-car voice assistants. It addresses limitations of existing benchmarks that overlook reliability under uncertainty. The environment includes an LLM-simulated user, 19 domain policies, 58 interconnected tools, and dynamic states/databases. Beyond…

Johannes Kirmayr, Lukas Stappen, Elisabeth André
Published
Jan 2026
Citations
6
Code
33 stars
Read summary Original
44

Together AI

DSGym: A Holistic Framework for Evaluating and Training Data Science Agents

DSGym is a standardized framework for evaluating and training data science agents in isolated, stateful execution environments. It addresses limitations in existing benchmarks, which often allow tasks to be solved without accessing data files (shortcut solvability), lack cross-benchmark standardization, and have narrow domain coverage. DSGym provides a…

Fan Nie, Junlin Wang, Harper Hua, Federico Bianchi, et al.
Published
Jan 2026
Citations
12
Code
58 stars
Read summary Original
45

Annual Meeting of the Association for Computational Linguistics

ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development

ABC-Bench is a benchmark for evaluating LLM-based agents on full-lifecycle backend development tasks, covering repository exploration, code editing, environment configuration, deployment, and end-to-end API testing. It includes 224 tasks from 127 MIT-licensed GitHub repositories, spanning 8 languages and 19 frameworks. The ABC-Pipeline automates task…

Jie Yang, Honglin Guo, Li Ji, Jiazheng Zhou, et al.
Published
Jan 2026
Citations
3
Code
33 stars
Read summary Original
46

arXiv.org

$A^3$-Bench: Benchmarking Memory-Driven Scientific Reasoning via Anchor and Attractor Activation

The paper introduces A3-Bench, a benchmark for evaluating memory-driven scientific reasoning in large language models (LLMs). It is grounded in the concepts of anchors (foundational knowledge units) and attractors (experience-based templates), which are activated during reasoning. The authors annotate 2,198 science problems across math, physics, and…

Jian Zhang, Yu He, Zhiyuan Wang, Zhangqi Wang, et al.
Published
Jan 2026
Citations
1
Code
Not linked
Read summary Original
47

arXiv.org

Watching, Reasoning, and Searching: A Video Deep Research Benchmark on Open Web for Agentic Video Reasoning

The paper introduces VideoDR, the first benchmark for video deep research, which requires models to extract multi-frame visual anchors from videos, perform interactive web searches, and conduct multi-hop reasoning over combined video-web evidence to answer open-domain factoid questions. The benchmark comprises 500 samples across six semantic domains, with…

Chengwen Liu, Xiaomin Yu, Zhuoyue Chang, Zhe Huang, et al.
Published
Jan 2026
Citations
6
Code
160 stars
Read summary Original
48

Annual Meeting of the Association for Computational Linguistics

Thinking with Map: Reinforced Parallel Map-Augmented Agent for Geolocalization

The paper introduces Thinking with Map, a map-augmented agent for image geolocalization that equips a large vision-language model (LVLM) with map tools (POI search, static/satellite map queries, image zoom) to iteratively propose and verify location hypotheses. The method uses a two-stage optimization: agentic reinforcement learning (GRPO) to improve…

Yuxiang Ji, Yong Wang, Ziyu Ma, Yiming Hu, et al.
Published
Jan 2026
Citations
8
Code
177 stars
Read summary Original
49

Annual Meeting of the Association for Computational Linguistics

KnowMe-Bench: Benchmarking Person Understanding for Lifelong Digital Companions

KnowMe-Bench is a benchmark for evaluating person understanding in lifelong digital companions, built from long-form autobiographical narratives rather than sparse chat logs. It addresses two gaps in existing benchmarks: evaluation misalignment (retrieval proxies vs. true person understanding) and data substrate misalignment (low-density, decontextualized…

Tingyu Wu, Zhisheng Chen, Ziyan Weng, Shuhe Wang, et al.
Published
Jan 2026
Citations
8
Code
133 stars
Read summary Original
50

Annual Meeting of the Association for Computational Linguistics

Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflows

FINCH (FinWorkBench) is a benchmark for evaluating AI agents on real-world, enterprise-grade finance and accounting workflows. It is built from authentic enterprise data, including Enron emails and spreadsheets, EUSES, and financial reports from institutions like the World Bank, covering 2000–2025. The dataset comprises 172 composite workflows with 384…

Haoyu Dong, Pengkun Zhang, Yan Gao, Xuanyu Dong, et al.
Published
Dec 2025
Citations
5
Code
13 stars
Read summary Original