Paper 2511.15567
Computer-Use Agents as Judges for Generative User Interface
- Published
- Nov 2025
- Research lab
- Independent
- Citations
- 6
- GitHub
- 45 stars
01 In brief
Summary
The paper introduces AUI-Gym, a benchmark for automatic GUI development and testing, and a Coder-CUA collaboration framework.
AUI-Gym includes 52 applications across six domains (app, landing, game, interactive, tool, utility) with 1,560 tasks synthesized by GPT-5 and human-validated.
Each task is paired with a rule-based verifier that programmatically checks task solvability.
The framework positions the Coder as Designer, generating and revising UIs, and the CUA as Judge, providing feedback through two signals: task solvability (identifying unsupported tasks) and CUA navigation (success/failure of task execution).
A CUA Dashboard compresses multi-step navigation trajectories into a single 1920x1080 image, reducing visual tokens by 76.2% on average.
Experiments with GPT-5, GPT-4o, and Qwen3-Coder-30B show that integrated feedback improves function completeness (e.g., GPT-5 from 67.9% to 81.5%) and CUA success rate (e.g., GPT-5 from 24.5% to 26.0%, with larger gains for weaker coders).
Ablations show the Dashboard outperforms text-only or screenshot-only feedback.
Qualitative analysis shows revisions add functionality and improve agent accessibility (e.g., adding buttons, increasing contrast).
The work demonstrates that designing UIs for agents, rather than humans, improves task execution and robustness.
02 From the paper
Abstract
Computer-Use Agents (CUA) are becoming increasingly capable of autonomously operating digital environments through Graphical User Interfaces (GUI). Yet, most GUI remain designed primarily for humans--prioritizing aesthetics and usability--forcing agents to adopt human-oriented behaviors that are unnecessary for efficient task execution. At the same time, rapid advances in coding-oriented language models (Coder) have transformed automatic GUI design. This raises a fundamental question: Can CUA as judges to assist Coder for automatic GUI design? To investigate, we introduce AUI-Gym, a benchmark for Automatic GUI development spanning 52 applications across diverse domains. Using language models, we synthesize 1560 tasks that simulate real-world scenarios. To ensure task reliability, we further develop a verifier that programmatically checks whether each task is executable within its environment. Building on this, we propose a Coder-CUA in Collaboration framework: the Coder acts as Designer, generating and revising websites, while the CUA serves as Judge, evaluating functionality and refining designs. Success is measured not by visual appearance, but by task solvability and CUA navigation success rate. To turn CUA feedback into usable guidance, we design a CUA Dashboard that compresses multi-step navigation histories into concise visual summaries, offering interpretable guidance for iterative redesign. By positioning agents as both designers and judges, our framework shifts interface design toward agent-native efficiency and reliability. Our work takes a step toward shifting agents from passive use toward active participation in digital environments. Our code and dataset are available at https://github.com/showlab/AUI.