The Year in AI Papers
The year/Independent research

Paper 2511.07332

Grounding Computer Use Agents on Human Demonstrations

Aarash Feizi, Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Kaixin Li, Rabiul Awal, Xing Han Lù, Johan Obando-Ceron, Juan A. Rodriguez, Nicolas Chapados, David Vazquez, Adriana Romero-Soriano, et al.

Published
Nov 2025
Research lab
Independent
Citations
9
GitHub
134 stars

01 In brief

Summary

The paper introduces GROUNDCUA, a large-scale desktop grounding dataset built from expert human demonstrations, covering 87 applications across 12 categories with 56K screenshots and over 3.56M human-verified element annotations.

The dataset features high-resolution images (0.39–7.0 megapixels), dense annotations (average 64 per screenshot), and small element sizes, addressing the lack of high-quality desktop resources.

Using GROUNDCUA, the authors develop the GROUNDNEXT family of vision-language models (3B and 7B) trained with supervised fine-tuning (SFT) on 700K samples and reinforcement learning (RL) on 10K samples.

GROUNDNEXT achieves state-of-the-art results on five benchmarks (ScreenSpotPro, OSWorld-G, MMBench-GUI, ScreenSpot-v2, UI-Vision) using less than one-tenth the training data of prior work like JEDI (9M samples).

In an agentic setting on OSWorld-Verified with o3 as planner, GROUNDNEXT-3B achieves 50.6 overall, comparable to JEDI-7B (51.0) and surpassing larger models like OpenCUA-72B (46.1).

The paper demonstrates that high-quality, expert-driven data is more effective than sheer data volume, and shows cross-platform generalization to mobile and web despite desktop-only training.

The authors plan to release the dataset and models for open research.

02 From the paper

Abstract

Building reliable computer-use agents requires grounding: accurately connecting natural language instructions to the correct on-screen elements. While large datasets exist for web and mobile interactions, high-quality resources for desktop environments are limited. To address this gap, we introduce GroundCUA, a large-scale desktop grounding dataset built from expert human demonstrations. It covers 87 applications across 12 categories and includes 56K screenshots, with every on-screen element carefully annotated for a total of over 3.56M human-verified annotations. From these demonstrations, we generate diverse instructions that capture a wide range of real-world tasks, providing high-quality data for model training. Using GroundCUA, we develop the GroundNext family of models that map instructions to their target UI elements. At both 3B and 7B scales, GroundNext achieves state-of-the-art results across five benchmarks using supervised fine-tuning, while requiring less than one-tenth the training data of prior work. Reinforcement learning post-training further improves performance, and when evaluated in an agentic setting on the OSWorld benchmark using o3 as planner, GroundNext attains comparable or superior results to models trained with substantially more data,. These results demonstrate the critical role of high-quality, expert-driven datasets in advancing general-purpose computer-use agents.