The Year in AI Papers
The year/Independent research

Paper 2512.15431

Step-GUI Technical Report

Haolong Yan, Jia Wang, Xin Huang, Yeqing Shen, Ziyang Meng, Zhimin Fan, Kaijun Tan, Jin Gao, Lieyu Shi, Mi Yang, Shiliang Yang, Zhirui Wang, et al.

Published
Dec 2025
Research lab
Independent
Citations
27
GitHub
Not linked

01 In brief

Summary

This technical report introduces Step-GUI, a family of multimodal GUI agent models (4B/8B) built on Qwen3-VL, achieving state-of-the-art performance across benchmarks (8B: 80.2% AndroidWorld, 48.5% OSWorld, 62.6% ScreenShot-Pro).

The core innovation is the Calibrated Step Reward System (CSRS), a self-evolving training pipeline that converts model-generated trajectories into reliable training data via trajectory-level calibration, achieving >90% annotation accuracy with 10-100× lower cost.

Training uses a three-stage paradigm: Mid-Training, Cold-Start Fine-Tuning, and Reinforcement Learning with Verifiable Rewards (RLVR), incorporating techniques like gradient preservation and semi-online exploration.

The report also proposes GUI-MCP, the first Model Context Protocol for GUI automation, featuring a hierarchical dual-layer architecture (Low-level MCP for atomic operations, High-level MCP for task delegation to local models) enabling high-privacy execution.

Additionally, AndroidDaily is introduced as a benchmark grounded in real-world mobile usage, with 3146 static actions and 235 end-to-end tasks across daily scenarios.

Results show Step-GUI-8B achieves 89.91% static and 52.50% end-to-end on AndroidDaily, while maintaining robust general multimodal capabilities.

The self-evolving pipeline demonstrates progressive improvement across six training rounds, with notable phase transitions on AndroidWorld and steady gains on OSWorld, validating the closed-loop data refinement approach for building practical GUI agents.

02 From the paper

Abstract

Recent advances in multimodal large language models unlock unprecedented opportunities for GUI automation. However, a fundamental challenge remains: how to efficiently acquire high-quality training data while maintaining annotation reliability? We introduce a self-evolving training pipeline powered by the Calibrated Step Reward System, which converts model-generated trajectories into reliable training signals through trajectory-level calibration, achieving >90% annotation accuracy with 10-100x lower cost. Leveraging this pipeline, we introduce Step-GUI, a family of models (4B/8B) that achieves state-of-the-art GUI performance (8B: 80.2% AndroidWorld, 48.5% OSWorld, 62.6% ScreenShot-Pro) while maintaining robust general capabilities. As GUI agent capabilities improve, practical deployment demands standardized interfaces across heterogeneous devices while protecting user privacy. To this end, we propose GUI-MCP, the first Model Context Protocol for GUI automation with hierarchical architecture that combines low-level atomic operations and high-level task delegation to local specialist models, enabling high-privacy execution where sensitive data stays on-device. Finally, to assess whether agents can handle authentic everyday usage, we introduce AndroidDaily, a benchmark grounded in real-world mobile usage patterns with 3146 static actions and 235 end-to-end tasks across high-frequency daily scenarios (8B: static 89.91%, end-to-end 52.50%). Our work advances the development of practical GUI agents and demonstrates strong potential for real-world deployment in everyday digital interactions.